├── .gitignore
├── README.md
├── train.ipynb
├── linear_transformer.py
├── rotation demonstration-Adam-p0.ipynb
├── variable_L_exp.ipynb
├── variable_N_exp.ipynb
├── plot_stochastic_noise.ipynb
└── plot_loss.ipynb


/.gitignore:
--------------------------------------------------------------------------------
1 | "*.pdf"
2 | "*.pth"
3 | "*.png"


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # LinearTransformer
 2 | Pytorch code for reproducing experiments for the following papers:
 3 | 
 4 | [1] [Transformers learn to implement preconditioned gradient descent for in-context learning](https://arxiv.org/abs/2306.00297).  *Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, Suvrit Sra*  
 5 | [2] [Linear attention is (maybe) all you need (to understand Transformer optimization)](https://arxiv.org/abs/2310.01082).  *Kwangjun Ahn, Xiang Cheng, Minhak Song, Chulhee Yun, Suvrit Sra, Ali Jadbabaie*
 6 | 
 7 | 
 8 | <h2>Experiments for <a href=https://arxiv.org/abs/2306.00297>Transformers learn to implement preconditioned gradient descent for in-context learning</a></h2>
 9 | 
10 | **'simple demonstration.ipynb'**:
11 | - Training a 3 layer Linear Transformer with SGD/Adam, **covariates have identity covariance**
12 | - Plotting test loss
13 | - Displaying matrices at end of training + distance to identity (similar to Figure 4 of [1])
14 | 
15 | **'rotation demonstration-Adam.ipynb'**:
16 | - Training a 3 layer Linear Transformer with Adam, **covariates have non-identity covariance** (Adam requires about 100x more steps to converge compared to the identity covariance case)
17 | - Plotting test loss
18 | - Displaying matrices at end of training + distance to identity (similar to Figure 4 of [1])
19 | 'rotation demonstration-Adam-p0.ipynb' is similar to 'rotation demonstration-Adam.ipynb', but enforces that the P matrix has top left block = 0
20 | 
21 | **'variable_L_exp.ipynb'**:
22 | - Compares n-layer linear Transformer against n-step Gradient Descent/ Preconditioned Gradient Descent, for n = 1,2,3,4, for fixed context length N=20
23 | 
24 | **'variable_N_exp.ipynb':**
25 | - Compares 3-layer linear Transformer against 3-step Gradient Descent/ Preconditioned Gradient Descent, for context length N={2,4,6...20}
26 | 
27 | 
28 | **'linear_transformer.py'** contains definition of the Linear Transformer model, along with some other handy functions.
29 | 
30 | <h2>Experiments for <a href=https://arxiv.org/abs/2310.01082>Linear attention is (maybe) all you need (to understand Transformer optimization)</a></h2>
31 | 
32 | ### Quck Start
33 | Setting: training 3 layer linear transformer with Adam/SGD (with clipping), covariates have normal convariance
34 | 
35 | 1. Run `train.ipynb` - training linear transformer
36 | 2. Run `plot_loss.ipynb` - generates loss vs iteration plot
37 | 3. Run `plot_stochastic_noise.ipynb` - generates stochastic gradient noise histogram
38 | 4. Run `plot_condition_number.ipynb` - generates robust 5ondition number plot
39 | 5. Run `plot_smoothness_vs_gradnorm.ipynb` - generates smoothness vs gradienr norm plot
40 | 
41 | ### Hyperparameters
42 | 
43 | `mode`: method of generating samples (`['normal', 'sphere', 'gamma']`)
44 | 
45 | `alg`: Optimization algorithm (`['sgd', 'adam']`)
46 | 
47 | `toclip`: `True` if use clipping algorithm. Otherwise, `False`.
48 | 
49 | `lr`: learning rate
50 | 
51 | `sd`: random seed
52 | 
53 | `max_iters`: maximum number of iterations
54 | 
55 | `n_layer`: number of layers of linear transformer
56 | 
57 | `N`: number of in-context samples
58 | 
59 | ### Learning Rates
60 | 
61 | |Setting (n_layer=3)|SGDM (with clipping)|Adam (with clipping)|
62 | |-----|-----:|-----:|
63 | |`mode='normal', N=5`|0.01|0.005|
64 | |`mode='normal', N=20`|0.02|0.02|
65 | |`mode='sphere', N=20`|5|0.1|
66 | |`mode='gamma', N=20`|0.02|0.02|


--------------------------------------------------------------------------------
/train.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "code",
  5 |    "execution_count": 19,
  6 |    "metadata": {},
  7 |    "outputs": [],
  8 |    "source": [
  9 |     "import torch\n",
 10 |     "%matplotlib inline\n",
 11 |     "from matplotlib import pyplot as plt\n",
 12 |     "import math\n",
 13 |     "import torch.nn.functional as F\n",
 14 |     "from torch.nn.functional import relu\n",
 15 |     "from torch import nn\n",
 16 |     "import torch.optim as optim\n",
 17 |     "import torch.optim.lr_scheduler as lr_scheduler\n",
 18 |     "import random\n",
 19 |     "import numpy as np\n",
 20 |     "import gc\n",
 21 |     "from pylab import *\n",
 22 |     "import os\n",
 23 |     "import random\n",
 24 |     "import json\n",
 25 |     "import pandas as pd\n",
 26 |     "from scipy.stats import norm\n",
 27 |     "pd.set_option('display.float_format', lambda x: '%.5f' % x)\n",
 28 |     "import sys\n",
 29 |     "import matplotlib.pyplot as plt\n",
 30 |     "import time\n",
 31 |     "\n",
 32 |     "from linear_transformer import Transformer_F, attention, generate_data, in_context_loss, generate_data_inplace\n",
 33 |     "\n",
 34 |     "np.set_printoptions(precision = 4, suppress = True)\n",
 35 |     "torch.set_printoptions(precision=2)\n",
 36 |     "device = torch.device(\"cuda\")\n",
 37 |     "torch.cuda.set_device(0)"
 38 |    ]
 39 |   },
 40 |   {
 41 |    "cell_type": "code",
 42 |    "execution_count": 20,
 43 |    "metadata": {},
 44 |    "outputs": [],
 45 |    "source": [
 46 |     "# Set Hyperparameters\n",
 47 |     "\n",
 48 |     "# Fixed\n",
 49 |     "n_head = 1\n",
 50 |     "d = 5\n",
 51 |     "B = 1000\n",
 52 |     "var = 0.05\n",
 53 |     "shape_k = 0.1\n",
 54 |     "\n",
 55 |     "# Number of Iterations to run\n",
 56 |     "max_iters = 10000\n",
 57 |     "hist_stride = 1\n",
 58 |     "\n",
 59 |     "# We vary the following parameters\n",
 60 |     "n_layer = 3\n",
 61 |     "mode = 'normal'\n",
 62 |     "N = 20\n",
 63 |     "seeds = [0,1,2,3,4,5]\n",
 64 |     "algos = ['sgd','adam']\n",
 65 |     "lrs = [0.02]\n"
 66 |    ]
 67 |   },
 68 |   {
 69 |    "cell_type": "code",
 70 |    "execution_count": 21,
 71 |    "metadata": {},
 72 |    "outputs": [],
 73 |    "source": [
 74 |     "# pipe output to log file\n",
 75 |     "log_dir = 'log' \n",
 76 |     "os.makedirs(log_dir, exist_ok=True)\n",
 77 |     "f = open(log_dir + '/train.log', \"a\", 1)\n",
 78 |     "sys.stdout = f\n",
 79 |     "filename_format = log_dir + '/train_layer{}_N{}_{}_{}_{}_lr{}_sd{}.pth'"
 80 |    ]
 81 |   },
 82 |   {
 83 |    "cell_type": "code",
 84 |    "execution_count": 22,
 85 |    "metadata": {},
 86 |    "outputs": [],
 87 |    "source": [
 88 |     "# one-step update of (non-)clipping algotirthm\n",
 89 |     "def clip_and_step(allparam, optimizer, toclip, clip_threshold = 1.):\n",
 90 |     "    grad_all = allparam.grad\n",
 91 |     "    grad_p = grad_all\n",
 92 |     "    norm_p = grad_p.norm()\n",
 93 |     "    if toclip and norm_p > clip_threshold:\n",
 94 |     "            grad_all.mul_(clip_threshold/norm_p)\n",
 95 |     "    optimizer.step()\n",
 96 |     "    return norm_p.item()"
 97 |    ]
 98 |   },
 99 |   {
100 |    "cell_type": "code",
101 |    "execution_count": 23,
102 |    "metadata": {},
103 |    "outputs": [],
104 |    "source": [
105 |     "\n",
106 |     "## Train linear transformer\n",
107 |     "\n",
108 |     "for alg in algos:\n",
109 |     "    for toclip in [True]: # True means with clipping, False means without clipping\n",
110 |     "        for lr in lrs:\n",
111 |     "            for sd in seeds:\n",
112 |     "                filename = filename_format.format(n_layer, N, mode, alg, toclip, lr, sd)\n",
113 |     "                print(filename)\n",
114 |     "                np.random.seed(sd)\n",
115 |     "                torch.manual_seed(sd)\n",
116 |     "                hist_list = list()\n",
117 |     "\n",
118 |     "                # initialize model paramter\n",
119 |     "                model = Transformer_F(n_layer, n_head, d, var)\n",
120 |     "                model.to(device)\n",
121 |     "\n",
122 |     "                # create optimizer\n",
123 |     "                if alg == 'sgd':\n",
124 |     "                    optimizer = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9, weight_decay=0)\n",
125 |     "                elif alg == 'adam':\n",
126 |     "                    optimizer = torch.optim.AdamW(model.parameters(), lr=lr, betas=(0.9, 0.9), weight_decay=0)\n",
127 |     "                else: assert False\n",
128 |     "\n",
129 |     "                for t in range(max_iters):\n",
130 |     "                    start = time.time()\n",
131 |     "                    # save model parameters\n",
132 |     "                    if t%hist_stride ==0:\n",
133 |     "                        hist_list.append(model.allparam.clone().detach())\n",
134 |     "\n",
135 |     "                    #  generate a new batch of training set\n",
136 |     "                    Z, y = generate_data(mode,N,d,B,shape_k)\n",
137 |     "                    Z = Z.to(device)\n",
138 |     "                    y = y.to(device)\n",
139 |     "\n",
140 |     "                    loss = in_context_loss(model, Z, y)\n",
141 |     "                    loss_value = loss.item()\n",
142 |     "                    loss.backward()\n",
143 |     "\n",
144 |     "                    if mode == 'sphere':\n",
145 |     "                        clip_threshold = 0.1\n",
146 |     "                    else:\n",
147 |     "                        clip_threshold = 1.0\n",
148 |     "\n",
149 |     "                    # take optimizer step\n",
150 |     "                    norms = clip_and_step(model.allparam, optimizer,toclip,clip_threshold)\n",
151 |     "                    optimizer.zero_grad()\n",
152 |     "                    \n",
153 |     "                    end=time.time()\n",
154 |     "                    if t%100 ==0 or t<5:\n",
155 |     "                        print('iter {} | Loss: {}  time: {}  gradnorm: {}'.format(t,loss_value, end-start, norms))\n",
156 |     "                \n",
157 |     "                torch.save({'hist_list':hist_list}, filename)"
158 |    ]
159 |   },
160 |   {
161 |    "cell_type": "code",
162 |    "execution_count": 24,
163 |    "metadata": {},
164 |    "outputs": [],
165 |    "source": [
166 |     "sys.stdout = sys.__stdout__\n",
167 |     "f.close()"
168 |    ]
169 |   }
170 |  ],
171 |  "metadata": {
172 |   "kernelspec": {
173 |    "display_name": "pytorch",
174 |    "language": "python",
175 |    "name": "python3"
176 |   },
177 |   "language_info": {
178 |    "codemirror_mode": {
179 |     "name": "ipython",
180 |     "version": 3
181 |    },
182 |    "file_extension": ".py",
183 |    "mimetype": "text/x-python",
184 |    "name": "python",
185 |    "nbconvert_exporter": "python",
186 |    "pygments_lexer": "ipython3",
187 |    "version": "3.11.3"
188 |   },
189 |   "orig_nbformat": 4
190 |  },
191 |  "nbformat": 4,
192 |  "nbformat_minor": 2
193 | }
194 | 


--------------------------------------------------------------------------------
/linear_transformer.py:
--------------------------------------------------------------------------------
  1 | ###########################################
  2 | # This file contains the following:
  3 | # 1. Linear Transformer Model
  4 | # 2. Function for clipping gradient
  5 | # 3. Function for generating random data
  6 | #
  7 | # The notation for linear attention follows
  8 | # the paper at https://arxiv.org/pdf/2306.00297.pdf
  9 | ###########################################
 10 | 
 11 | 
 12 | import torch
 13 | from torch import nn
 14 | import numpy as np
 15 | 
 16 | device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
 17 | 
 18 | # Definition of a single linear attention unit for linear-regression data
 19 | # P is the value matrix
 20 | # Q is the product of key,query matrices
 21 | # the dimensions of the input are
 22 | # B: batch-size of prompts
 23 | # N: context length (excluding query)
 24 | # d: covariate dimension
 25 | # P,Q are d x d matrices
 26 | # Z is a B x (N+1) + (d+1) matrix
 27 | # Output is also B x (N+1) + (d+1)
 28 | 
 29 | # For linear attention, activation = None
 30 | # For standard attention, activation(x) = torch.nn.functional.softmax(x, dim = 2)
 31 | # For ReLU attention, activation(x) = torch.nn.relu(x)
 32 | def attention(P,Q,Z, activation = None):
 33 |     B= Z.shape[0]
 34 |     N = Z.shape[1]-1
 35 |     d = Z.shape[2]-1
 36 |     P_full =  torch.cat([P,torch.zeros(1,d).to(device)],dim=0)
 37 |     P_full =  torch.cat([P_full,torch.zeros(d+1,1).to(device)],dim=1)
 38 |     P_full[d,d] = 1
 39 |     Q_full = torch.cat([Q, torch.zeros(1,d).to(device)],dim=0)
 40 |     Q_full = torch.cat([Q_full, torch.zeros(d+1,1).to(device)],dim=1)
 41 |     A = torch.eye(N+1).to(device)
 42 |     A[N,N] = 0
 43 |     Attn = torch.einsum('BNi, ij, BMj -> BNM', (Z,Q_full,Z))
 44 |     if activation is not None:
 45 |         Attn = activation(Attn)
 46 |     key = torch.einsum('ij, BNj -> BNi', (P_full,Z))
 47 |     Output = torch.einsum('BNM,ML, BLi -> BNi', (Attn,A,key))
 48 |     return Output /N
 49 | 
 50 | 
 51 | # The Linear Transformer module
 52 | # n_layer denotes the number of layers
 53 | # n_head denotes the number of heads. In most of our experiments, n_head = 1
 54 | # d denotes the dimension of covariates
 55 | # var denotes the variance of initialization. It needs to be sufficiently small, but exact value is not important
 56 | # allparam: contains all the parameters, has dimension n_layer x n_head x 2 x d x d
 57 | # For example
 58 | # - P matrix at layer i, head j is allparam[i,j,0,:,:]
 59 | # - Q matrix at layer i, head j is allparam[i,j,1,:,:]
 60 | class Transformer_F(nn.Module):
 61 |     def __init__(self, n_layer, n_head, d, var):
 62 |         super(Transformer_F, self).__init__()
 63 |         self.register_parameter('allparam', torch.nn.Parameter(torch.zeros(n_layer, n_head, 2, d, d)))
 64 |         with torch.no_grad():
 65 |             self.allparam.normal_(0,var)
 66 |         self.n_layer = n_layer
 67 |         self.n_head = n_head
 68 | 
 69 |     def forward(self, Z):
 70 |         for i in range(self.n_layer):
 71 |             Zi = Z
 72 |             residues = 0
 73 |             # the forwarad map of each layer is given by F(Z) = Z + attention(Z)
 74 |             for j in range(self.n_head):
 75 |                 Pij = self.allparam[i,j,0,:,:]
 76 |                 Qij = self.allparam[i,j,1,:,:]
 77 |                 residues = residues + attention(Pij,Qij,Zi)
 78 |             Z = Zi + residues
 79 |         return Z
 80 |     
 81 |     #enforces top-left-dxd-block sparsity on p
 82 |     def zero_p(self):
 83 |         for i in range(self.n_layer):
 84 |             for j in range(self.n_head):
 85 |                 with torch.no_grad():
 86 |                     self.allparam[i,j,0,:,:].zero_()
 87 | 
 88 | # evaluate the loss of model, given data (Z,y)
 89 | def in_context_loss(model, Z, y):
 90 |     N = Z.shape[1]-1
 91 |     d = Z.shape[2]-1
 92 |     output = model(Z)
 93 |     diff = output[:,N,d]+y
 94 |     loss = ((diff)**2).mean() 
 95 |     return loss
 96 |         
 97 | # generate random data for linear regression
 98 | # mode: distribution of samples to generate. Currently supports 'normal', 'gamma', 'sphere'
 99 | # N: number of context examples
100 | # d: dimension of covariates
101 | # For gamma distribution:
102 | # - shape_k: shape parameter of gamma distribution (unused otherwise)
103 | # - scale parameter: hard coded so that when shape_k = 5/2 and d=5, the generated data is standard normal
104 | def generate_data(mode='normal',N=20,d=1,B=1000,shape_k=0.1, U=None, D=None):
105 |     W= torch.FloatTensor(B, d).normal_(0,1).to(device)
106 |     X = torch.FloatTensor(B, N, d).normal_(0, 1).to(device)
107 |     X_test = torch.FloatTensor(B,1,d).normal_(0, 1).to(device)
108 |     
109 |     if U is not None:
110 |         U = U.to(device)
111 |         D = D.to(device)
112 |         W= torch.FloatTensor(B, d).normal_(0,1).to(device)
113 |         W = torch.mm(W,torch.inverse(D))
114 |         W = torch.mm(W,U.t())
115 |     
116 |     if mode =='sphere':
117 |         X.div_(X.norm(p=2,dim=2)[:,:,None])
118 |         X_test.div_(X_test.norm(p=2,dim=2)[:,:,None])
119 |     elif mode == 'gamma':
120 |         # random gamma scaling for X
121 |         gamma_scales = np.random.gamma(shape=shape_k, scale=(10/shape_k)**(0.5), size=[B,N])
122 |         gamma_scales = torch.Tensor(gamma_scales).to(device)
123 |         gamma_scales = gamma_scales.sqrt()
124 |         # random gamma scaling for X_test
125 |         gamma_test_scales = np.random.gamma(shape=shape_k, scale=(10/shape_k)**(0.5), size=[B,1])
126 |         gamma_test_scales = torch.Tensor(gamma_test_scales).to(device)
127 |         gamma_test_scales = gamma_test_scales.sqrt()
128 |         # normalize to unit norm
129 |         X.div_(X.norm(p=2,dim=2)[:,:,None])
130 |         X_test.div_(X_test.norm(p=2,dim=2)[:,:,None])
131 |         # scale by gamma
132 |         X.mul_(gamma_scales[:,:,None])
133 |         X_test.mul_(gamma_test_scales[:,:,None])
134 |     elif mode =='normal':
135 |         assert True
136 |     elif mode == 'relu':
137 |         return generate_data_relu(N=N, d=d, B=B, hidden_dim=d)
138 |     elif mode == 'mlp':
139 |         generate_data_mlp(N=N, d=d, B=B, hidden_dim=d)
140 |     else:
141 |         assert False
142 |         
143 |     if U is not None:
144 |         X = torch.einsum('ij, jk, BNk -> BNi', (U,D,X))
145 |         X_test = torch.einsum('ij, jk, BNk -> BNi', (U,D,X_test))
146 |         
147 |     y = torch.einsum('bi,bni->bn', (W, X)).unsqueeze(2)
148 |     y_zero = torch.zeros(B,1,1).to(device)
149 |     y_test = torch.einsum('bi,bni->bn', (W, X_test)).squeeze(1)
150 |     X_comb= torch.cat([X,X_test],dim=1)
151 |     y_comb= torch.cat([y,y_zero],dim=1)
152 |     Z= torch.cat([X_comb,y_comb],dim=2)
153 |     return Z.to(device),y_test.to(device)
154 | 
155 | def generate_data_inplace(Z, U=None, D=None):
156 |     
157 |     
158 |     B = Z.shape[0]
159 |     N = Z.shape[1]-1
160 |     d = Z.shape[2]-1
161 |     X = Z[:,:,0:-1]
162 |     X.normal_(0, 1).to(device)
163 |     W= torch.FloatTensor(B, d).normal_(0,1).to(device)
164 |     if U is not None:
165 |         U = U.to(device)
166 |         D = D.to(device)
167 |         W = torch.mm(W,torch.inverse(D))
168 |         W = torch.mm(W,U.t())
169 |         Z[:,:,0:-1] = torch.einsum('ij, jk, BNk -> BNi', (U,D,X))
170 |         
171 |     Z[:,:,-1] = torch.einsum('bi,bni->bn', (W, Z[:,:,0:-1])) #y update
172 |     y_test = Z[:,-1,-1].detach().clone()
173 |     Z[:,-1,-1].zero_()
174 |     return Z.to(device),y_test.to(device)
175 | 
176 | def generate_data_sine(N=10, B=1000):
177 |     # Sample amplitude a and phase p for each task
178 |     a = torch.FloatTensor(B).uniform_(0.1, 5).to(device)
179 |     p = torch.FloatTensor(B).uniform_(0, math.pi).to(device)
180 |  
181 |     X = torch.FloatTensor(B, N).uniform_(-5, 5).to(device)
182 |  
183 |     Y = a.unsqueeze(1) * torch.sin(p.unsqueeze(1) + X)
184 |  
185 |     X = X.unsqueeze(-1)
186 |     Y = Y.unsqueeze(-1)
187 | 
188 |     return X, Y
189 | 
190 | def generate_data_relu(mode='normal', N=20, d=1, B=1000, shape_k=0.1, U=None, D=None, hidden_dim=100):
191 |     # Generate random input data
192 |     X = torch.FloatTensor(B, N, d).normal_(0, 1).to(device)
193 |     X_test = torch.FloatTensor(B, 1, d).normal_(0, 1).to(device)
194 | 
195 |     # Additional transformations if mode is 'sphere' or 'gamma' [Similar to the existing generate_data function]
196 | 
197 |     # Define a 1-hidden layer ReLU network
198 |     model = nn.Sequential(
199 |         nn.Linear(d, hidden_dim),
200 |         nn.ReLU(),
201 |         nn.Linear(hidden_dim, 1)
202 |     ).to(device)
203 |     model[0].weight.data.normal_(0, 0.1)
204 |     model[2].weight.data.normal_(0, 0.1)
205 | 
206 |     # Generate y values using the ReLU network
207 |     y = model(X.view(-1, d)).view(B, N, 1)
208 |     y_test = model(X_test.view(-1, d)).view(B, 1).squeeze(1)
209 |  
210 |     y_zero = torch.zeros(B, 1, 1).to(device)
211 |     X_comb = torch.cat([X, X_test], dim=1)
212 |     y_comb = torch.cat([y, y_zero], dim=1)
213 |     Z = torch.cat([X_comb, y_comb], dim=2)
214 | 
215 |     return Z, y_test
216 | 
217 | def generate_data_mlp(N=20, d=1, B=1000, hidden_dim=100):
218 |     # Generate random input data
219 |     X = torch.FloatTensor(B, N, d).normal_(0, 1).to(device)
220 |     X_test = torch.FloatTensor(B, 1, d).normal_(0, 1).to(device)
221 | 
222 |     # Additional transformations if mode is 'sphere' or 'gamma' [Similar to the existing generate_data function]
223 | 
224 |     # Define a 1-hidden layer ReLU network
225 |     model = nn.Sequential(
226 |         nn.Linear(d, hidden_dim),
227 |         nn.ReLU(),
228 |         nn.Linear(hidden_dim, d)
229 |     ).to(device)
230 |     model[0].weight.data.normal_(0, 1)
231 |     model[2].weight.data.normal_(0, 1)
232 | 
233 |     X_MLP = model(X.view(-1, d)).view(B, N, d)
234 |     X_test_MLP = model(X_test.view(-1, d)).view(B, 1, d)
235 | 
236 |     W = torch.FloatTensor(B, d).normal_(0,1).to(device)
237 |     y = torch.einsum('bi,bni->bn', (W, X_MLP)).unsqueeze(2)
238 |     y_zero = torch.zeros(B,1,1).to(device)
239 |     y_test = torch.einsum('bi,bni->bn', (W, X_test_MLP)).squeeze(1)
240 |     X_comb= torch.cat([X_MLP,X_test_MLP],dim=1)
241 |     y_comb= torch.cat([y,y_zero],dim=1)
242 |     Z= torch.cat([X_comb,y_comb],dim=2)
243 | 
244 |     return Z, y_test
245 | 


--------------------------------------------------------------------------------
/rotation demonstration-Adam-p0.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "code",
  5 |    "execution_count": 11,
  6 |    "id": "3fcfaf4d",
  7 |    "metadata": {},
  8 |    "outputs": [],
  9 |    "source": [
 10 |     "import torch\n",
 11 |     "from matplotlib import pyplot as plt\n",
 12 |     "import sys\n",
 13 |     "import time\n",
 14 |     "import os\n",
 15 |     "import numpy as np\n",
 16 |     "import math\n",
 17 |     "\n",
 18 |     "#####################################################\n",
 19 |     "# Same as rotation demonstration-Adam.ipynb, except\n",
 20 |     "# we additionally enforce that P=0 for each layer\n",
 21 |     "#####################################################\n",
 22 |     "\n",
 23 |     "#use cuda if available, else use cpu\n",
 24 |     "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
 25 |     "#torch.cuda.set_device(2)\n",
 26 |     "# import the model and some useful functions\n",
 27 |     "from linear_transformer import Transformer_F, attention, generate_data, in_context_loss, generate_data_inplace\n",
 28 |     "\n",
 29 |     "# set up some print options\n",
 30 |     "np.set_printoptions(precision = 2, suppress = True)\n",
 31 |     "torch.set_printoptions(precision=2)\n",
 32 |     "\n",
 33 |     "#begin logging\n",
 34 |     "cur_dir = 'log' \n",
 35 |     "os.makedirs(cur_dir, exist_ok=True)\n",
 36 |     "#f = open(cur_dir + '/rotation.log', \"a\", 1)\n",
 37 |     "#sys.stdout = f"
 38 |    ]
 39 |   },
 40 |   {
 41 |    "cell_type": "code",
 42 |    "execution_count": 18,
 43 |    "id": "9700bf1b",
 44 |    "metadata": {},
 45 |    "outputs": [],
 46 |    "source": [
 47 |     "# Set up problem parameters\n",
 48 |     "\n",
 49 |     "lr = 0.02\n",
 50 |     "clip_r = 0.01\n",
 51 |     "alg = 'adam'\n",
 52 |     "mode = 'normal'\n",
 53 |     "\n",
 54 |     "n_layer = 3  # number of layers of transformer\n",
 55 |     "N = 20     # context length\n",
 56 |     "d = 5        # dimension\n",
 57 |     "\n",
 58 |     "\n",
 59 |     "n_head = 1  # 1-headed attention\n",
 60 |     "B = 20000  # 1000 minibatch size\n",
 61 |     "var = 0.0001  # initializations scale of transformer parameter\n",
 62 |     "shape_k = 0.1  # shape_k: parameter for Gamma distributed covariates\n",
 63 |     "max_iters = 30000  # Number of Iterations to run\n",
 64 |     "hist_stride = 1  # stride for saved model paramters in `train.ipynb'\n",
 65 |     "stride = 100\n",
 66 |     "\n",
 67 |     "# a convenience function for taking a step and clipping\n",
 68 |     "def clip_and_step(allparam, optimizer, clip_r = None):\n",
 69 |     "    norm_p=None\n",
 70 |     "    grad_all = allparam.grad\n",
 71 |     "    if clip_r is not None:\n",
 72 |     "        for l in range(grad_all.shape[0]):\n",
 73 |     "            for h in range(grad_all.shape[1]):\n",
 74 |     "                for t in range(grad_all.shape[2]):\n",
 75 |     "                    norm_p = grad_all[l,h,t,:,:].norm().item()\n",
 76 |     "                    if norm_p > clip_r:\n",
 77 |     "                        grad_all[l,h,t,:,:].mul_(clip_r/norm_p)\n",
 78 |     "    optimizer.step()\n",
 79 |     "    return norm_p"
 80 |    ]
 81 |   },
 82 |   {
 83 |    "cell_type": "code",
 84 |    "execution_count": null,
 85 |    "id": "e69d0ea4",
 86 |    "metadata": {
 87 |     "scrolled": false
 88 |    },
 89 |    "outputs": [],
 90 |    "source": [
 91 |     "filename_format = '/rotation_hist_adam_pnull_{}_{}_{}.pth'\n",
 92 |     "filename = filename_format.format(n_layer, N, d)\n",
 93 |     "filename = (cur_dir + filename)\n",
 94 |     "hist_dict = {}\n",
 95 |     "U_dict = {}\n",
 96 |     "D_dict = {}\n",
 97 |     "\n",
 98 |     "seeds = [0,1,2,3,4]\n",
 99 |     "keys = [(s,) for s in seeds]\n",
100 |     "for key in keys:\n",
101 |     "    sd = key[0]\n",
102 |     "    \n",
103 |     "    prob_seed = sd\n",
104 |     "    opt_seed = sd\n",
105 |     "    \n",
106 |     "    hist_dict[key] = []\n",
107 |     "    \n",
108 |     "    #set seed and initialize model\n",
109 |     "    torch.manual_seed(opt_seed)\n",
110 |     "    \n",
111 |     "    model = Transformer_F(n_layer, 1, d, var)\n",
112 |     "    model.to(device)\n",
113 |     "    #initialize algorithm. Important: set beta = 0.9 for adam, 0.999 is very slow\n",
114 |     "    optimizer = torch.optim.AdamW(model.parameters(), lr=lr, betas=(0.99, 0.9), weight_decay=0)\n",
115 |     "    \n",
116 |     "    # set seed\n",
117 |     "    # sample random rotation matrix\n",
118 |     "    # initialize initial training batch\n",
119 |     "    np.random.seed(prob_seed)\n",
120 |     "    torch.manual_seed(prob_seed)\n",
121 |     "    gaus = torch.FloatTensor(5,5).uniform_(-1,1).cuda()\n",
122 |     "    U = torch.linalg.svd (gaus)[0].cuda()\n",
123 |     "    D = torch.diag(torch.FloatTensor([1,1,1/2,1/4,1])).cuda()\n",
124 |     "    U_dict[key]=U\n",
125 |     "    D_dict[key]=D\n",
126 |     "    Z, y = generate_data(mode,N,d,B,shape_k, U, D)\n",
127 |     "    Z = Z.to(device)\n",
128 |     "    y = y.to(device)\n",
129 |     "    for t in range(max_iters):\n",
130 |     "        if t%2000==0 and t>1:# and t < 200001:\n",
131 |     "            optimizer.param_groups[0]['lr'] = optimizer.param_groups[0]['lr'] *0.5\n",
132 |     "        if t%100==0:\n",
133 |     "            Z,y = generate_data_inplace(Z, U=U, D=D)\n",
134 |     "        start = time.time()\n",
135 |     "        # save model parameters\n",
136 |     "        if t%stride ==0:\n",
137 |     "            hist_dict[key].append(model.allparam.clone().detach())\n",
138 |     "        loss = in_context_loss(model, Z, y)\n",
139 |     "        # compute gradient, take step\n",
140 |     "        loss.backward()\n",
141 |     "        norms = clip_and_step(model.allparam, optimizer, clip_r=clip_r)\n",
142 |     "        optimizer.zero_grad()\n",
143 |     "        \n",
144 |     "        #IMPORTANT: zero out the p matrices after each update! This enforces the P=0 constraint.\n",
145 |     "        model.zero_p()\n",
146 |     "\n",
147 |     "        end=time.time()\n",
148 |     "        if t%100 ==0 or t<5:\n",
149 |     "            print('iter {} | Loss: {}  time: {}  gradnorm: {}'.format(t,loss.item(), end-start, norms))\n",
150 |     "#save to \n",
151 |     "torch.save({'hist_dict':hist_dict, 'U_dict':U_dict, 'D_dict':D_dict}, filename)"
152 |    ]
153 |   },
154 |   {
155 |    "cell_type": "code",
156 |    "execution_count": 54,
157 |    "id": "83154b0e",
158 |    "metadata": {},
159 |    "outputs": [],
160 |    "source": [
161 |     "####################################\n",
162 |     "# compute test loss\n",
163 |     "####################################\n",
164 |     "#hist_dict = torch.load(filename)['hist_dict']\n",
165 |     "loss_dict = {}\n",
166 |     "for key in hist_dict:\n",
167 |     "    sd = key[0]\n",
168 |     "    \n",
169 |     "    U = U_dict[key]\n",
170 |     "    D = D_dict[key]\n",
171 |     "    \n",
172 |     "    loss_dict[key] = torch.zeros(max_iters//stride)\n",
173 |     "    \n",
174 |     "    np.random.seed(99)\n",
175 |     "    torch.manual_seed(99)\n",
176 |     "    Z, y = generate_data(mode,N,d,B,shape_k,U,D)\n",
177 |     "    Z = Z.to(device)\n",
178 |     "    y = y.to(device)\n",
179 |     "    model = Transformer_F(n_layer, n_head, d, var).to(device)\n",
180 |     "    for t in range(0,max_iters,stride):\n",
181 |     "        with torch.no_grad():\n",
182 |     "            model.allparam.copy_(hist_dict[key][t//stride])\n",
183 |     "        loss_dict[key][t//stride] = in_context_loss(model, Z, y).item()"
184 |    ]
185 |   },
186 |   {
187 |    "cell_type": "code",
188 |    "execution_count": null,
189 |    "id": "b7c41abb",
190 |    "metadata": {},
191 |    "outputs": [],
192 |    "source": [
193 |     "####################################\n",
194 |     "# plot the test loss with error bars\n",
195 |     "####################################\n",
196 |     "\n",
197 |     "fig_dir = 'figures' \n",
198 |     "os.makedirs(fig_dir, exist_ok=True)\n",
199 |     "\n",
200 |     "fig, ax = plt.subplots(1, 1,figsize = (9, 7))\n",
201 |     "\n",
202 |     "losses = torch.zeros(len(seeds), max_iters//stride)\n",
203 |     "keys = loss_dict.keys()\n",
204 |     "for idx, key in enumerate(keys):\n",
205 |     "    losses[idx,:] = loss_dict[key].log()\n",
206 |     "losses_mean = torch.mean(losses, axis=0)\n",
207 |     "losses_std = torch.std(losses, axis=0)\n",
208 |     "ax.plot(range(0,max_iters,stride), losses_mean, color = 'blue', lw = 3)#, label='Adam')\n",
209 |     "ax.fill_between(range(0,max_iters,stride), losses_mean-losses_std, losses_mean+losses_std, color = 'black', alpha = 0.2)\n",
210 |     "ax.set_xlabel('Iteration',fontsize=30)\n",
211 |     "ax.set_ylabel('log(Loss)',fontsize=30)\n",
212 |     "ax.tick_params(axis='both', which='major', labelsize=20, width = 3, length = 10)\n",
213 |     "ax.tick_params(axis='both', which='minor', labelsize=20, width = 3, length = 5)\n",
214 |     "\n",
215 |     "\n",
216 |     "plt.tight_layout()\n",
217 |     "plt.savefig(fig_dir + '/rotation_demonstration_adam_pnull_loss_plot.pdf', dpi=600)"
218 |    ]
219 |   },
220 |   {
221 |    "cell_type": "code",
222 |    "execution_count": null,
223 |    "id": "88adc8d9",
224 |    "metadata": {},
225 |    "outputs": [],
226 |    "source": [
227 |     "####################################\n",
228 |     "# display the parameter matrices\n",
229 |     "# image/font setting assumes d=5\n",
230 |     "####################################\n",
231 |     "\n",
232 |     "key = (0,)\n",
233 |     "\n",
234 |     "U = U_dict[(0,)]\n",
235 |     "D = D_dict[(0,)]\n",
236 |     "UD = torch.mm(U,D)        \n",
237 |     "for l in range(n_layer):\n",
238 |     "    for h in range(n_head):\n",
239 |     "        fig, ax = plt.subplots(1, 1,figsize = (6, 6))\n",
240 |     "        matrix = hist_dict[key][-1][l,h,1,:,:]\n",
241 |     "        #rotate matrix by inverse of UD\n",
242 |     "        matrix = torch.mm(torch.mm(UD.t(), matrix), UD)\n",
243 |     "        # Create a heatmap using imshow\n",
244 |     "        im = ax.imshow(matrix.cpu(), cmap='gray_r')\n",
245 |     "        # Add the matrix values as text\n",
246 |     "        for i in range(matrix.shape[0]):\n",
247 |     "            for j in range(matrix.shape[1]):\n",
248 |     "                ax.text(j, i, format(matrix[i, j], '.2f'), ha='center', va='center', color='r')\n",
249 |     "        # Add a colorbar for reference\n",
250 |     "        fig.colorbar(im)\n",
251 |     "        #ax.set_title('$A_{}$'.format(l),fontsize=20)\n",
252 |     "        plt.savefig(fig_dir + '/rotation_demonstration_pnull_A{}.pdf'.format(l), dpi=600)\n",
253 |     "    "
254 |    ]
255 |   },
256 |   {
257 |    "cell_type": "code",
258 |    "execution_count": 56,
259 |    "id": "1a10c824",
260 |    "metadata": {},
261 |    "outputs": [],
262 |    "source": [
263 |     "########################################################\n",
264 |     "# plot the distance-to-identity of each matrix with time\n",
265 |     "########################################################\n",
266 |     "\n",
267 |     "# function for computing distance to identity\n",
268 |     "def compute_dist_identity(M):\n",
269 |     "    scale = torch.sum(torch.diagonal(M))/M.shape[0]\n",
270 |     "    ideal_identity = scale* torch.eye(M.shape[0]).to(device)\n",
271 |     "    difference = M - ideal_identity\n",
272 |     "    err = (torch.norm(difference,p='fro')/torch.norm(M,p='fro'))\n",
273 |     "    return err\n",
274 |     "\n",
275 |     "########################################\n",
276 |     "# compute distances (assume n_head = 1)\n",
277 |     "########################################\n",
278 |     "dist_dict = {}\n",
279 |     "\n",
280 |     "id_dist_dict={}\n",
281 |     "            \n",
282 |     "for key in hist_dict:\n",
283 |     "    (sd,) = key\n",
284 |     "    dist_dict[key] = torch.zeros(n_layer, 2, max_iters//stride)\n",
285 |     "    id_dist_dict[key] = torch.zeros(n_layer, 2, max_iters//stride)\n",
286 |     "    U = U_dict[key]\n",
287 |     "    D = D_dict[key]\n",
288 |     "    UD = torch.mm(U,D)        \n",
289 |     "    for t in range(0,max_iters,stride):\n",
290 |     "        with torch.no_grad():\n",
291 |     "            allparam = hist_dict[key][t//stride]\n",
292 |     "        for i in range(n_layer):\n",
293 |     "            for j in range(2):\n",
294 |     "                matrix = allparam[i,0,j,:,:]\n",
295 |     "                if j ==1:\n",
296 |     "                    id_dist_dict[key][i,j,t//stride] = compute_dist_identity(matrix).item()\n",
297 |     "                    matrix = torch.mm(torch.mm(UD.t(), matrix), UD)\n",
298 |     "                dist_dict[key][i,j,t//stride] = compute_dist_identity(matrix).item()\n",
299 |     "####################################\n",
300 |     "# plot distances\n",
301 |     "####################################\n",
302 |     "\n",
303 |     "fig_dir = 'figures' \n",
304 |     "os.makedirs(fig_dir, exist_ok=True)\n",
305 |     "\n",
306 |     "labels = ['$B_0$', '$B_1$', None, \n",
307 |     "          '$\\Sigma^{1/2} A_0 \\Sigma^{1/2}$', \n",
308 |     "          '$\\Sigma^{1/2} A_1 \\Sigma^{1/2}$', \n",
309 |     "          '$\\Sigma^{1/2} A_2 \\Sigma^{1/2}$']\n",
310 |     "names = ['B0', 'B1', None, \n",
311 |     "          'A0', \n",
312 |     "          'A1', \n",
313 |     "          'A2']\n",
314 |     "colors = ['blue','blue',None, 'blue','blue','blue']\n",
315 |     "\n",
316 |     "for l in range(n_layer):\n",
317 |     "    for pq in range(2):\n",
318 |     "        if l==n_layer-1 and pq==0:\n",
319 |     "            continue\n",
320 |     "        if pq ==0:\n",
321 |     "            continue\n",
322 |     "        fig, ax = plt.subplots(1, 1,figsize = (9, 7))\n",
323 |     "        if pq==1:\n",
324 |     "            id_dist_p = torch.zeros(len(seeds), max_iters//stride)\n",
325 |     "            for idx, sd in enumerate(seeds):\n",
326 |     "                losses[idx,:] = id_dist_dict[(sd,)][l,pq,:]\n",
327 |     "            dist_mean = torch.mean(losses, axis=0)\n",
328 |     "            dist_std = torch.std(losses, axis=0)\n",
329 |     "            ax.plot(range(0,max_iters,stride), dist_mean, color = 'red', lw = 3, label='$A_{}$'.format(l))\n",
330 |     "            ax.fill_between(range(0,max_iters,stride), dist_mean-dist_std, dist_mean+dist_std, color = 'red', alpha = 0.2)\n",
331 |     "        \n",
332 |     "        dist_p = torch.zeros(len(seeds), max_iters//stride)\n",
333 |     "        for idx, sd in enumerate(seeds):\n",
334 |     "            losses[idx,:] = dist_dict[(sd,)][l,pq,:]\n",
335 |     "        dist_mean = torch.mean(losses, axis=0)\n",
336 |     "        dist_std = torch.std(losses, axis=0)\n",
337 |     "        \n",
338 |     "        style_id = l + 3*pq\n",
339 |     "        \n",
340 |     "        ax.plot(range(0,max_iters,stride), dist_mean, color = colors[style_id], lw = 3, label=labels[style_id])\n",
341 |     "        ax.fill_between(range(0,max_iters,stride), dist_mean-dist_std, dist_mean+dist_std, color = colors[style_id], alpha = 0.2)\n",
342 |     "        ax.tick_params(axis='both', which='major', labelsize=20, width = 3, length = 10)\n",
343 |     "        ax.tick_params(axis='both', which='minor', labelsize=20, width = 3, length = 5)\n",
344 |     "        \n",
345 |     "        ax.set_ylim([0,1])\n",
346 |     "        ax.set_xlabel('Iteration',fontsize=30)\n",
347 |     "        ax.set_ylabel('Distance to Id',fontsize=30)\n",
348 |     "        ax.legend(fontsize=30)\n",
349 |     "        \n",
350 |     "        plt.savefig(fig_dir + '/rotation_demonstration_dist_to_id_adam_pnull_{}.pdf'.format(names[style_id]), dpi=600)"
351 |    ]
352 |   }
353 |  ],
354 |  "metadata": {
355 |   "kernelspec": {
356 |    "display_name": "Python 3 (ipykernel)",
357 |    "language": "python",
358 |    "name": "python3"
359 |   },
360 |   "language_info": {
361 |    "codemirror_mode": {
362 |     "name": "ipython",
363 |     "version": 3
364 |    },
365 |    "file_extension": ".py",
366 |    "mimetype": "text/x-python",
367 |    "name": "python",
368 |    "nbconvert_exporter": "python",
369 |    "pygments_lexer": "ipython3",
370 |    "version": "3.9.12"
371 |   }
372 |  },
373 |  "nbformat": 4,
374 |  "nbformat_minor": 5
375 | }
376 | 


--------------------------------------------------------------------------------
/variable_L_exp.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "code",
  5 |    "execution_count": 94,
  6 |    "id": "3fcfaf4d",
  7 |    "metadata": {},
  8 |    "outputs": [],
  9 |    "source": [
 10 |     "import torch\n",
 11 |     "from matplotlib import pyplot as plt\n",
 12 |     "import sys\n",
 13 |     "import time\n",
 14 |     "import os\n",
 15 |     "import numpy as np\n",
 16 |     "import math\n",
 17 |     "\n",
 18 |     "##############################################################################################################\n",
 19 |     "# Trains a linear Transformer with 1,2,3,4 layers\n",
 20 |     "# Plots the test loss of trained Transformer against 1,2,3,4 steps of gradient descent (with and without preconditioning)\n",
 21 |     "##############################################################################################################\n",
 22 |     "\n",
 23 |     "#use cuda if available, else use cpu\n",
 24 |     "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
 25 |     "#torch.cuda.set_device(1)\n",
 26 |     "# import the model and some useful functions\n",
 27 |     "from linear_transformer import Transformer_F, attention, generate_data, in_context_loss, generate_data_inplace\n",
 28 |     "\n",
 29 |     "# set up some print options\n",
 30 |     "np.set_printoptions(precision = 2, suppress = True)\n",
 31 |     "torch.set_printoptions(precision=2)\n",
 32 |     "\n",
 33 |     "#begin logging\n",
 34 |     "cur_dir = 'log' \n",
 35 |     "os.makedirs(cur_dir, exist_ok=True)\n",
 36 |     "#f = open(cur_dir + '/rotation.log', \"a\", 1)\n",
 37 |     "#sys.stdout = f"
 38 |    ]
 39 |   },
 40 |   {
 41 |    "cell_type": "code",
 42 |    "execution_count": 95,
 43 |    "id": "9700bf1b",
 44 |    "metadata": {},
 45 |    "outputs": [],
 46 |    "source": [
 47 |     "# Set up problem parameters\n",
 48 |     "\n",
 49 |     "lr = 0.01\n",
 50 |     "clip_r = 0.01\n",
 51 |     "alg = 'adam'\n",
 52 |     "mode = 'normal'\n",
 53 |     "\n",
 54 |     "n_layer = 4  # number of layers of transformer\n",
 55 |     "N = 20     # context length\n",
 56 |     "d = 5        # dimension\n",
 57 |     "\n",
 58 |     "\n",
 59 |     "n_head = 1  # 1-headed attention\n",
 60 |     "B = 20000  # 1000 minibatch size\n",
 61 |     "var = 0.0001  # initializations scale of transformer parameter\n",
 62 |     "shape_k = 0.1  # shape_k: parameter for Gamma distributed covariates\n",
 63 |     "max_iters = 20000  # Number of Iterations to run\n",
 64 |     "hist_stride = 1  # stride for saved model paramters in `train.ipynb'\n",
 65 |     "stride = 100\n",
 66 |     "\n",
 67 |     "# a convenience function for taking a step and clipping\n",
 68 |     "def clip_and_step(allparam, optimizer, clip_r = None):\n",
 69 |     "    norm_p=None\n",
 70 |     "    grad_all = allparam.grad\n",
 71 |     "    if clip_r is not None:\n",
 72 |     "        for l in range(grad_all.shape[0]):\n",
 73 |     "            for h in range(grad_all.shape[1]):\n",
 74 |     "                for t in range(grad_all.shape[2]):\n",
 75 |     "                    norm_p = grad_all[l,h,t,:,:].norm().item()\n",
 76 |     "                    if norm_p > clip_r:\n",
 77 |     "                        grad_all[l,h,t,:,:].mul_(clip_r/norm_p)\n",
 78 |     "    optimizer.step()\n",
 79 |     "    return norm_p\n",
 80 |     "\n",
 81 |     "#format for saving run data\n",
 82 |     "filename_format = '/variable_L_hist_{}_{}_{}.pth'\n",
 83 |     "n_layers = [1,2,3,4]  # number of layers of transformer\n",
 84 |     "seeds=[0,1,2,3,4]\n",
 85 |     "keys = []\n",
 86 |     "for s in seeds:\n",
 87 |     "    for n_layer in n_layers:\n",
 88 |     "        keys.append((s,n_layer,))"
 89 |    ]
 90 |   },
 91 |   {
 92 |    "cell_type": "code",
 93 |    "execution_count": null,
 94 |    "id": "e69d0ea4",
 95 |    "metadata": {
 96 |     "scrolled": false
 97 |    },
 98 |    "outputs": [],
 99 |    "source": [
100 |     "for key in keys:\n",
101 |     "    sd = key[0]\n",
102 |     "    n_layer = key[1]\n",
103 |     "    filename = cur_dir + filename_format.format(n_layer, N, sd)\n",
104 |     "    print(key)\n",
105 |     "    \n",
106 |     "    prob_seed = sd\n",
107 |     "    opt_seed = sd\n",
108 |     "    \n",
109 |     "    hist = []\n",
110 |     "    \n",
111 |     "    #set seed and initialize model\n",
112 |     "    torch.manual_seed(opt_seed)\n",
113 |     "    model = Transformer_F(n_layer, 1, d, var)\n",
114 |     "    model.to(device)\n",
115 |     "    #initialize algorithm. Important: set beta = 0.9 for adam, 0.999 is very slow\n",
116 |     "    optimizer = torch.optim.AdamW(model.parameters(), lr=lr, betas=(0.99, 0.9), weight_decay=0)\n",
117 |     "    \n",
118 |     "    # set seed\n",
119 |     "    # sample random rotation matrix\n",
120 |     "    # initialize initial training batch\n",
121 |     "    np.random.seed(prob_seed)\n",
122 |     "    torch.manual_seed(prob_seed)\n",
123 |     "    gaus = torch.FloatTensor(5,5).uniform_(-1,1).cuda()\n",
124 |     "    U = torch.linalg.svd (gaus)[0].cuda()\n",
125 |     "    D = torch.diag(torch.FloatTensor([1,1,1/2,1/4,1])).cuda()\n",
126 |     "    Z, y = generate_data(mode,N,d,B,shape_k, U, D)\n",
127 |     "    Z = Z.to(device)\n",
128 |     "    y = y.to(device)\n",
129 |     "    for t in range(max_iters):\n",
130 |     "        if t%4000==0 and t>1:\n",
131 |     "            optimizer.param_groups[0]['lr'] = optimizer.param_groups[0]['lr'] *0.5\n",
132 |     "        if t%100==0:\n",
133 |     "            Z,y = generate_data_inplace(Z, U=U, D=D)\n",
134 |     "        start = time.time()\n",
135 |     "        # save model parameters\n",
136 |     "        if t%stride ==0:\n",
137 |     "            hist.append(model.allparam.clone().detach())\n",
138 |     "        loss = in_context_loss(model, Z, y)\n",
139 |     "        # compute gradient, take step\n",
140 |     "        loss.backward()\n",
141 |     "        norms = clip_and_step(model.allparam, optimizer, clip_r=clip_r)\n",
142 |     "        optimizer.zero_grad()\n",
143 |     "        end=time.time()\n",
144 |     "        if t%100 ==0 or t<5:\n",
145 |     "            print('iter {} | Loss: {}  time: {}  gradnorm: {}'.format(t,loss.item(), end-start, norms))\n",
146 |     "    torch.save({'hist':hist, 'U':U, 'D':D}, filename)"
147 |    ]
148 |   },
149 |   {
150 |    "cell_type": "code",
151 |    "execution_count": 96,
152 |    "id": "83154b0e",
153 |    "metadata": {},
154 |    "outputs": [],
155 |    "source": [
156 |     "########################################################\n",
157 |     "# compute test loss for trained linear Transformers\n",
158 |     "########################################################\n",
159 |     "loss_dict = {}\n",
160 |     "for sd in seeds:\n",
161 |     "    key = (sd,)\n",
162 |     "    loss_dict[key] = torch.zeros(4)\n",
163 |     "    for n_layer in n_layers:\n",
164 |     "        # load parameters for given n_layer and seed\n",
165 |     "        filename = cur_dir + filename_format.format(n_layer, N, sd)\n",
166 |     "        hist = torch.load(filename)['hist']\n",
167 |     "        U = torch.load(filename)['U']\n",
168 |     "        D = torch.load(filename)['D']\n",
169 |     "        \n",
170 |     "        # given short(er) training steps, may have some unstable solutions\n",
171 |     "        # on a validation set of (seed=999), find the solution with best validation\n",
172 |     "        # loss from the last 20 runs\n",
173 |     "        np.random.seed(999)\n",
174 |     "        torch.manual_seed(999)\n",
175 |     "        Z, y = generate_data(mode,N,d,B,shape_k,U,D)\n",
176 |     "        Z = Z.to(device)\n",
177 |     "        y = y.to(device)\n",
178 |     "        model = Transformer_F(n_layer, n_head, d, var).to(device)\n",
179 |     "        loss = 100\n",
180 |     "        bestmodel = None\n",
181 |     "        for t in range(len(hist)-20, len(hist)):\n",
182 |     "            with torch.no_grad():\n",
183 |     "                model.allparam.copy_(hist[t])\n",
184 |     "            newloss = in_context_loss(model, Z, y).item()\n",
185 |     "            if (newloss < loss):\n",
186 |     "                loss=newloss\n",
187 |     "                bestmodel = hist[t]\n",
188 |     "        with torch.no_grad():\n",
189 |     "            model.allparam.copy_(bestmodel)\n",
190 |     "        np.random.seed(99)\n",
191 |     "        torch.manual_seed(99)\n",
192 |     "        Z, y = generate_data(mode,N,d,B,shape_k,U,D)\n",
193 |     "        Z = Z.to(device)\n",
194 |     "        y = y.to(device)    \n",
195 |     "        #compute loss\n",
196 |     "        loss_dict[key][n_layer-1] = in_context_loss(model, Z, y).log().item()"
197 |    ]
198 |   },
199 |   {
200 |    "cell_type": "code",
201 |    "execution_count": null,
202 |    "id": "e9ca8fff",
203 |    "metadata": {},
204 |    "outputs": [],
205 |    "source": [
206 |     "# evaluate the performance of x steps of Gradient Descent\n",
207 |     "def do_gd(Z,eta,numstep):\n",
208 |     "    N = Z.shape[0]-1\n",
209 |     "    X = Z[0:N-1,0:5]\n",
210 |     "    Y = Z[0:N-1,5]\n",
211 |     "    w = torch.zeros(X.shape[1]).to(device)\n",
212 |     "    for k in range(numstep):\n",
213 |     "        XTXw = torch.einsum('ik,ij,j->k',X,X,w)\n",
214 |     "        XTY = torch.einsum('ik,i->k',X,Y)\n",
215 |     "        grad = XTXw - XTY\n",
216 |     "        w = w - eta * grad\n",
217 |     "    return w\n",
218 |     "\n",
219 |     "def eval_w_instance(Z, Ytest, w):\n",
220 |     "    N = Z.shape[0]-1\n",
221 |     "    Xtest = Z[N,0:5]\n",
222 |     "    prediction = torch.einsum('i,i->',w,Xtest)\n",
223 |     "    return (Ytest - prediction)**2, prediction\n",
224 |     "\n",
225 |     "\n",
226 |     "gd_loss_matrix = torch.zeros(len(seeds),4)\n",
227 |     "\n",
228 |     "\n",
229 |     "for n_layer in n_layers:\n",
230 |     "    #first find best eta\n",
231 |     "    #load seed 1 for U,D matrices\n",
232 |     "    sd = 1\n",
233 |     "    best_loss = 10000\n",
234 |     "    best_eta = 0\n",
235 |     "    numstep = n_layer\n",
236 |     "    # load UD matrices\n",
237 |     "    filename = cur_dir + filename_format.format(n_layer, N, sd)\n",
238 |     "    U = torch.load(filename)['U']\n",
239 |     "    D = torch.load(filename)['D']\n",
240 |     "    #generate test data using seed 999\n",
241 |     "    np.random.seed(999)\n",
242 |     "    torch.manual_seed(999)\n",
243 |     "    Z, y = generate_data(mode,N,d,B,shape_k,U,D)\n",
244 |     "    Z = Z.to(device)\n",
245 |     "    y = y.to(device)\n",
246 |     "    #done generating data \n",
247 |     "    \n",
248 |     "    for eta in [0.008, 0.01, 0.02, 0.04, 0.08, 0.16]:\n",
249 |     "        ### start of evaluate mean loss ###\n",
250 |     "        total_loss = 0\n",
251 |     "        for i in range(5000):\n",
252 |     "            Zi = Z[i,:,:]\n",
253 |     "            Ytesti = y[i]\n",
254 |     "            w = do_gd(Zi,eta,numstep)\n",
255 |     "            gd_loss, gd_pred = eval_w_instance(Zi, Ytesti, w)\n",
256 |     "            total_loss = total_loss + gd_loss\n",
257 |     "        mean_loss = total_loss / 5000\n",
258 |     "        ### end of evaluate mean loss ###\n",
259 |     "        print('eta: {}, loss: {}'.format(eta, mean_loss))\n",
260 |     "        if (mean_loss < best_loss):\n",
261 |     "            best_eta = eta\n",
262 |     "            best_loss = mean_loss\n",
263 |     "    print('best eta: {} for n_layer={}'.format(best_eta, n_layer))\n",
264 |     "    \n",
265 |     "    #now do actual evaluation\n",
266 |     "    for sd in seeds:\n",
267 |     "        opt_seed = sd\n",
268 |     "        \n",
269 |     "        filename = cur_dir + filename_format.format(n_layer, N, sd)\n",
270 |     "        U = torch.load(filename)['U']\n",
271 |     "        D = torch.load(filename)['D']\n",
272 |     "        #generate test data\n",
273 |     "        torch.manual_seed(sd)\n",
274 |     "        Z, y = generate_data(mode,N,d,B,shape_k,U,D)\n",
275 |     "        Z = Z.to(device)\n",
276 |     "        y = y.to(device)\n",
277 |     "        #done generating data \n",
278 |     "        eta = best_eta\n",
279 |     "        ### start of evaluate mean loss ###\n",
280 |     "        total_loss = 0\n",
281 |     "        for i in range(Z.shape[0]):\n",
282 |     "            Zi = Z[i,:,:]\n",
283 |     "            Ytesti = y[i]\n",
284 |     "            w = do_gd(Zi,eta,numstep)\n",
285 |     "            gd_loss, gd_pred = eval_w_instance(Zi, Ytesti, w)\n",
286 |     "            total_loss = total_loss + gd_loss\n",
287 |     "        mean_loss = total_loss / Z.shape[0]\n",
288 |     "        gd_loss_matrix[sd,n_layer-1] = mean_loss\n",
289 |     "        \n",
290 |     "#compute mean and std of log test loss for plotting\n",
291 |     "gd_loss_mean = gd_loss_matrix.log().mean(dim=0)\n",
292 |     "gd_loss_std = gd_loss_matrix.log().var(dim=0)**0.5"
293 |    ]
294 |   },
295 |   {
296 |    "cell_type": "code",
297 |    "execution_count": null,
298 |    "id": "98d80e86",
299 |    "metadata": {},
300 |    "outputs": [],
301 |    "source": [
302 |     "def do_preconditioned_gd(Z,eta,numstep,U,D):\n",
303 |     "    N = Z.shape[0]-1\n",
304 |     "    X = Z[0:N-1,0:5]\n",
305 |     "    Y = Z[0:N-1,5]\n",
306 |     "    w = torch.zeros(X.shape[1]).to(device)\n",
307 |     "    X = torch.einsum('ij, jk, Nk -> Ni', (torch.inverse(D),U.t(),X))\n",
308 |     "    for k in range(numstep):\n",
309 |     "        XTXw = torch.einsum('ik,ij,j->k',X,X,w)\n",
310 |     "        XTY = torch.einsum('ik,i->k',X,Y)\n",
311 |     "        grad = XTXw - XTY\n",
312 |     "        w = w - eta * grad\n",
313 |     "    return w\n",
314 |     "\n",
315 |     "def eval_w_instance_precon(Z, Ytest, w, U, D):\n",
316 |     "    N = Z.shape[0]-1\n",
317 |     "    Xtest = Z[N,0:5]\n",
318 |     "    Xtest = torch.einsum('ij, jk, k -> i', (torch.inverse(D),U.t(),Xtest))\n",
319 |     "    prediction = torch.einsum('i,i->',w,Xtest)\n",
320 |     "    return (Ytest - prediction)**2, prediction\n",
321 |     "\n",
322 |     "\n",
323 |     "\n",
324 |     "pgd_loss_matrix = torch.zeros(len(seeds),4)\n",
325 |     "\n",
326 |     "for n_layer in n_layers:\n",
327 |     "    #first find best eta\n",
328 |     "    #load seed 1 for U,D matrices\n",
329 |     "    sd = 1\n",
330 |     "    best_loss = 10000\n",
331 |     "    best_eta = 0\n",
332 |     "    numstep = n_layer\n",
333 |     "    # load UD matrices\n",
334 |     "    filename = cur_dir + filename_format.format(n_layer, N, sd)\n",
335 |     "    U = torch.load(filename)['U'].to(device)\n",
336 |     "    D = torch.load(filename)['D'].to(device)\n",
337 |     "    #generate test data using seed 999\n",
338 |     "    np.random.seed(999)\n",
339 |     "    torch.manual_seed(999)\n",
340 |     "    Z, y = generate_data(mode,N,d,B,shape_k,U,D)\n",
341 |     "    Z = Z.to(device)\n",
342 |     "    y = y.to(device)\n",
343 |     "    #done generating data \n",
344 |     "    \n",
345 |     "    for eta in [0.001, 0.002, 0.004, 0.008, 0.01, 0.02, 0.04, 0.08, 0.16]:\n",
346 |     "        ### start of evaluate mean loss ###\n",
347 |     "        total_loss = 0\n",
348 |     "        for i in range(5000):\n",
349 |     "            Zi = Z[i,:,:]\n",
350 |     "            Ytesti = y[i]\n",
351 |     "            w = do_preconditioned_gd(Zi,eta,numstep,U,D)\n",
352 |     "            pgd_loss, pgd_pred = eval_w_instance_precon(Zi, Ytesti, w, U, D)\n",
353 |     "            total_loss = total_loss + pgd_loss\n",
354 |     "        mean_loss = total_loss / 5000\n",
355 |     "        ### end of evaluate mean loss ###\n",
356 |     "        print('eta: {}, loss: {}'.format(eta, mean_loss))\n",
357 |     "        if (mean_loss < best_loss):\n",
358 |     "            best_eta = eta\n",
359 |     "            best_loss = mean_loss\n",
360 |     "    print('best eta: {} for n_layer={}'.format(best_eta, n_layer))\n",
361 |     "    \n",
362 |     "    #now do actual evaluation\n",
363 |     "    for sd in seeds:\n",
364 |     "        opt_seed = sd\n",
365 |     "        \n",
366 |     "        filename = cur_dir + filename_format.format(n_layer, N, sd)\n",
367 |     "        U = torch.load(filename)['U'].to(device)\n",
368 |     "        D = torch.load(filename)['D'].to(device)\n",
369 |     "        #generate test data\n",
370 |     "        torch.manual_seed(sd)\n",
371 |     "        Z, y = generate_data(mode,N,d,B,shape_k,U,D)\n",
372 |     "        Z = Z.to(device)\n",
373 |     "        y = y.to(device)\n",
374 |     "        #done generating data \n",
375 |     "        eta = best_eta\n",
376 |     "        ### start of evaluate mean loss ###\n",
377 |     "        total_loss = 0\n",
378 |     "        for i in range(5000):\n",
379 |     "            Zi = Z[i,:,:]\n",
380 |     "            Ytesti = y[i]\n",
381 |     "            w = do_preconditioned_gd(Zi,eta,numstep,U,D)\n",
382 |     "            pgd_loss, pgd_pred = eval_w_instance_precon(Zi, Ytesti, w, U, D)\n",
383 |     "            total_loss = total_loss + pgd_loss\n",
384 |     "        mean_loss = total_loss / 5000\n",
385 |     "        pgd_loss_matrix[sd,n_layer-1] = mean_loss\n",
386 |     "\n",
387 |     "#compute mean and std of log test loss for plotting\n",
388 |     "pgd_loss_mean = pgd_loss_matrix.log().mean(dim=0)\n",
389 |     "pgd_loss_std = pgd_loss_matrix.log().var(dim=0)**0.5\n",
390 |     "        "
391 |    ]
392 |   },
393 |   {
394 |    "cell_type": "code",
395 |    "execution_count": null,
396 |    "id": "5af8555c",
397 |    "metadata": {},
398 |    "outputs": [],
399 |    "source": [
400 |     "####################################\n",
401 |     "# plot final test loss against N\n",
402 |     "####################################\n",
403 |     "\n",
404 |     "fig_dir = 'figures' \n",
405 |     "os.makedirs(fig_dir, exist_ok=True)\n",
406 |     "\n",
407 |     "fig, ax = plt.subplots(1, 1,figsize = (9, 9))\n",
408 |     "\n",
409 |     "losses = torch.zeros(len(seeds), len(n_layers))\n",
410 |     "keys = loss_dict.keys()\n",
411 |     "for idx, key in enumerate(keys):\n",
412 |     "    losses[idx,:] = loss_dict[key]\n",
413 |     "losses_mean = torch.mean(losses, axis=0)\n",
414 |     "losses_std = torch.std(losses, axis=0)\n",
415 |     "\n",
416 |     "plt.plot(n_layers, gd_loss_mean, color='blue', label='GD')\n",
417 |     "plt.fill_between(n_layers, gd_loss_mean - gd_loss_std, gd_loss_mean + gd_loss_std, color='blue', alpha=0.2)\n",
418 |     "plt.plot(n_layers, pgd_loss_mean, color='green', label='Preconditioned GD')\n",
419 |     "plt.fill_between(n_layers, pgd_loss_mean - pgd_loss_std, pgd_loss_mean + pgd_loss_std, color='green', alpha=0.2)\n",
420 |     "ax.plot(n_layers, losses_mean, color = 'red', lw = 3, label='Linear Transformer')\n",
421 |     "ax.fill_between(n_layers, losses_mean-losses_std, losses_mean+losses_std, color = 'red', alpha = 0.2)\n",
422 |     "\n",
423 |     "plt.ylabel('log(Loss)',fontsize=30)\n",
424 |     "plt.xlabel('Number of Layers/Steps',fontsize=30)\n",
425 |     "ax.tick_params(axis='both', which='major', labelsize=30, width = 3, length = 10)\n",
426 |     "ax.tick_params(axis='both', which='minor', labelsize=20, width = 3, length = 5)\n",
427 |     "ax.legend(fontsize=24)\n",
428 |     "#ax.set_yscale('log')\n",
429 |     "\n",
430 |     "\n",
431 |     "plt.tight_layout()\n",
432 |     "plt.savefig(fig_dir + '/variable-L-plot.pdf', dpi=600)"
433 |    ]
434 |   },
435 |   {
436 |    "cell_type": "code",
437 |    "execution_count": null,
438 |    "id": "78b77440",
439 |    "metadata": {},
440 |    "outputs": [],
441 |    "source": []
442 |   }
443 |  ],
444 |  "metadata": {
445 |   "kernelspec": {
446 |    "display_name": "Python 3 (ipykernel)",
447 |    "language": "python",
448 |    "name": "python3"
449 |   },
450 |   "language_info": {
451 |    "codemirror_mode": {
452 |     "name": "ipython",
453 |     "version": 3
454 |    },
455 |    "file_extension": ".py",
456 |    "mimetype": "text/x-python",
457 |    "name": "python",
458 |    "nbconvert_exporter": "python",
459 |    "pygments_lexer": "ipython3",
460 |    "version": "3.9.12"
461 |   }
462 |  },
463 |  "nbformat": 4,
464 |  "nbformat_minor": 5
465 | }
466 | 


--------------------------------------------------------------------------------
/variable_N_exp.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "code",
  5 |    "execution_count": 2,
  6 |    "id": "3fcfaf4d",
  7 |    "metadata": {},
  8 |    "outputs": [],
  9 |    "source": [
 10 |     "import torch\n",
 11 |     "from matplotlib import pyplot as plt\n",
 12 |     "import sys\n",
 13 |     "import time\n",
 14 |     "import os\n",
 15 |     "import numpy as np\n",
 16 |     "import math\n",
 17 |     "\n",
 18 |     "#####################################################\n",
 19 |     "# This is almost identical to simple demonstration \n",
 20 |     "# -- except covariates have a skewed covariance matrix\n",
 21 |     "#\n",
 22 |     "# In this notebook, we train a 3-layer linear transformer with\n",
 23 |     "# - context-length 20\n",
 24 |     "# - covariate dimension 5, standard Gaussian distribution\n",
 25 |     "# We plot\n",
 26 |     "# - test loss against number of iterations\n",
 27 |     "# - imshow of each parameter matrix at end of training\n",
 28 |     "# - distance-to-identity of each parameter matrix\n",
 29 |     "#####################################################\n",
 30 |     "\n",
 31 |     "#use cuda if available, else use cpu\n",
 32 |     "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
 33 |     "torch.cuda.set_device(0)\n",
 34 |     "# import the model and some useful functions\n",
 35 |     "from linear_transformer import Transformer_F, attention, generate_data, in_context_loss, generate_data_inplace\n",
 36 |     "\n",
 37 |     "# set up some print options\n",
 38 |     "np.set_printoptions(precision = 2, suppress = True)\n",
 39 |     "torch.set_printoptions(precision=2)\n",
 40 |     "\n",
 41 |     "#begin logging\n",
 42 |     "cur_dir = 'log' \n",
 43 |     "os.makedirs(cur_dir, exist_ok=True)\n",
 44 |     "#f = open(cur_dir + '/rotation.log', \"a\", 1)\n",
 45 |     "#sys.stdout = f"
 46 |    ]
 47 |   },
 48 |   {
 49 |    "cell_type": "code",
 50 |    "execution_count": 3,
 51 |    "id": "9700bf1b",
 52 |    "metadata": {},
 53 |    "outputs": [],
 54 |    "source": [
 55 |     "# Set up problem parameters\n",
 56 |     "\n",
 57 |     "lr = 0\n",
 58 |     "clip_r = 0.001\n",
 59 |     "alg = 'adam'\n",
 60 |     "mode = 'normal'\n",
 61 |     "\n",
 62 |     "n_layer = 3  # number of layers of transformer\n",
 63 |     "d = 5        # dimension\n",
 64 |     "\n",
 65 |     "\n",
 66 |     "n_head = 1  # 1-headed attention\n",
 67 |     "B = 40000  # 1000 minibatch size\n",
 68 |     "var = 0.0001  # initializations scale of transformer parameter\n",
 69 |     "shape_k = 0.1  # shape_k: parameter for Gamma distributed covariates\n",
 70 |     "max_iters = 8000  # Number of Iterations to run\n",
 71 |     "hist_stride = 1  # stride for saved model paramters in `train.ipynb'\n",
 72 |     "stride = 100\n",
 73 |     "\n",
 74 |     "# a convenience function for taking a step and clipping\n",
 75 |     "def clip_and_step(allparam, optimizer, clip_r = None):\n",
 76 |     "    norm_p=None\n",
 77 |     "    grad_all = allparam.grad\n",
 78 |     "    if clip_r is not None:\n",
 79 |     "        for l in range(grad_all.shape[0]):\n",
 80 |     "            for h in range(grad_all.shape[1]):\n",
 81 |     "                for t in range(grad_all.shape[2]):\n",
 82 |     "                    norm_p = grad_all[l,h,t,:,:].norm().item()\n",
 83 |     "                    if norm_p > clip_r:\n",
 84 |     "                        grad_all[l,h,t,:,:].mul_(clip_r/norm_p)\n",
 85 |     "    optimizer.step()\n",
 86 |     "    return norm_p\n",
 87 |     "\n",
 88 |     "filename_format = '/variable_N_hist_{}_{}_{}.pth'\n",
 89 |     "Ns = range(20,21,2)     # context length\n",
 90 |     "seeds=[0,1,2,3,4]\n",
 91 |     "keys = []\n",
 92 |     "for s in seeds:\n",
 93 |     "    for N in Ns:\n",
 94 |     "        keys.append((s,N,))"
 95 |    ]
 96 |   },
 97 |   {
 98 |    "cell_type": "code",
 99 |    "execution_count": null,
100 |    "id": "e69d0ea4",
101 |    "metadata": {
102 |     "scrolled": false
103 |    },
104 |    "outputs": [],
105 |    "source": [
106 |     "for key in keys:\n",
107 |     "    sd = key[0]\n",
108 |     "    N = key[1]\n",
109 |     "    filename = cur_dir + filename_format.format(n_layer, N, sd)\n",
110 |     "    print(key)\n",
111 |     "    \n",
112 |     "    prob_seed = sd\n",
113 |     "    opt_seed = sd\n",
114 |     "    \n",
115 |     "    hist = []\n",
116 |     "    \n",
117 |     "    #set seed and initialize model\n",
118 |     "    torch.manual_seed(opt_seed)\n",
119 |     "    \n",
120 |     "    model = Transformer_F(n_layer, 1, d, var)\n",
121 |     "    model.to(device)\n",
122 |     "    #initialize algorithm. Important: set beta = 0.9 for adam, 0.999 is very slow\n",
123 |     "    \n",
124 |     "    \n",
125 |     "    if N < 5:\n",
126 |     "        lr = 0.001\n",
127 |     "    elif N < 15:\n",
128 |     "        lr = 0.01\n",
129 |     "    else:\n",
130 |     "        lr = 0.01\n",
131 |     "    \n",
132 |     "    optimizer = torch.optim.AdamW(model.parameters(), lr=lr, betas=(0.99, 0.99), weight_decay=0)\n",
133 |     "    \n",
134 |     "    # set seed\n",
135 |     "    # sample random rotation matrix\n",
136 |     "    # initialize initial training batch\n",
137 |     "    np.random.seed(prob_seed)\n",
138 |     "    torch.manual_seed(prob_seed)\n",
139 |     "    gaus = torch.FloatTensor(5,5).uniform_(-1,1).cuda()\n",
140 |     "    U = torch.linalg.svd (gaus)[0].cuda()\n",
141 |     "    D = torch.diag(torch.FloatTensor([1,1,1/2,1/4,1])).cuda()\n",
142 |     "    \n",
143 |     "    #  generate a SINGLE BATCH of training set USED FOREVER\n",
144 |     "    Z, y = generate_data(mode,N,d,B,shape_k, U, D)\n",
145 |     "    Z = Z.to(device)\n",
146 |     "    y = y.to(device)\n",
147 |     "    for t in range(max_iters):\n",
148 |     "        start = time.time()\n",
149 |     "        if t%2000==0 and t>1:\n",
150 |     "            optimizer.param_groups[0]['lr'] = optimizer.param_groups[0]['lr'] *0.5\n",
151 |     "        Z,y = generate_data_inplace(Z, U=U, D=D)\n",
152 |     "        start = time.time()\n",
153 |     "        # save model parameters\n",
154 |     "        if t%stride ==0:\n",
155 |     "            hist.append(model.allparam.clone().detach())\n",
156 |     "        loss = in_context_loss(model, Z, y)\n",
157 |     "        # compute gradient, take step\n",
158 |     "        loss.backward()\n",
159 |     "        norms = clip_and_step(model.allparam, optimizer, clip_r=clip_r)\n",
160 |     "        optimizer.zero_grad()\n",
161 |     "        end=time.time()\n",
162 |     "        if t%500 ==0 or t<5:\n",
163 |     "            print('iter {} | Loss: {}  time: {}  gradnorm: {}'.format(t,loss.item(), end-start, norms))\n",
164 |     "    torch.save({'hist':hist, 'U':U, 'D':D}, filename)"
165 |    ]
166 |   },
167 |   {
168 |    "cell_type": "code",
169 |    "execution_count": 8,
170 |    "id": "83154b0e",
171 |    "metadata": {},
172 |    "outputs": [],
173 |    "source": [
174 |     "####################################\n",
175 |     "# compute test loss\n",
176 |     "####################################\n",
177 |     "#hist_dict = torch.load(filename)['hist_dict']\n",
178 |     "keys = []\n",
179 |     "seeds = [0,1,2,3,4]\n",
180 |     "Ns = range(2,21,2)\n",
181 |     "for s in seeds:\n",
182 |     "    for N in Ns:\n",
183 |     "        keys.append((s,N,))\n",
184 |     "loss_dict = {}\n",
185 |     "for sd in seeds:\n",
186 |     "    key = (sd,)\n",
187 |     "    loss_dict[key] = torch.zeros(len(Ns))\n",
188 |     "    for N in Ns:\n",
189 |     "        filename = cur_dir + filename_format.format(n_layer, N, sd)\n",
190 |     "        hist = torch.load(filename)['hist']\n",
191 |     "        U = torch.load(filename)['U']\n",
192 |     "        D = torch.load(filename)['D']\n",
193 |     "        np.random.seed(999)\n",
194 |     "        torch.manual_seed(999)\n",
195 |     "        Z, y = generate_data(mode,N,d,B,shape_k,U,D)\n",
196 |     "        Z = Z.to(device)\n",
197 |     "        y = y.to(device)\n",
198 |     "        model = Transformer_F(n_layer, n_head, d, var).to(device)\n",
199 |     "        loss = 100\n",
200 |     "        bestmodel = None\n",
201 |     "        for t in range(len(hist)-10, len(hist)):\n",
202 |     "            with torch.no_grad():\n",
203 |     "                model.allparam.copy_(hist[t])\n",
204 |     "            newloss = in_context_loss(model, Z, y).item()\n",
205 |     "            if (newloss < loss):\n",
206 |     "                loss=newloss\n",
207 |     "                bestmodel = hist[t]\n",
208 |     "        with torch.no_grad():\n",
209 |     "            model.allparam.copy_(bestmodel)\n",
210 |     "        np.random.seed(99)\n",
211 |     "        torch.manual_seed(99)\n",
212 |     "        Z, y = generate_data(mode,N,d,B,shape_k,U,D)\n",
213 |     "        Z = Z.to(device)\n",
214 |     "        y = y.to(device)    \n",
215 |     "        loss_dict[key][N//2-1] = in_context_loss(model, Z, y).item()"
216 |    ]
217 |   },
218 |   {
219 |    "cell_type": "code",
220 |    "execution_count": null,
221 |    "id": "ff4227a3",
222 |    "metadata": {},
223 |    "outputs": [],
224 |    "source": [
225 |     "########################################################\n",
226 |     "# plot log final test loss against N, for sanity check\n",
227 |     "########################################################\n",
228 |     "\n",
229 |     "fig_dir = 'figures' \n",
230 |     "os.makedirs(fig_dir, exist_ok=True)\n",
231 |     "\n",
232 |     "fig, ax = plt.subplots(1, 1,figsize = (9, 9))\n",
233 |     "\n",
234 |     "losses = torch.zeros(len(seeds), len(Ns))\n",
235 |     "keys = loss_dict.keys()\n",
236 |     "for idx, key in enumerate(keys):\n",
237 |     "    losses[idx,:] = np.log(loss_dict[key])\n",
238 |     "losses_mean = torch.mean(losses, axis=0)\n",
239 |     "losses_std = torch.std(losses, axis=0)\n",
240 |     "ax.plot(Ns, losses_mean, color = 'red', lw = 3, label='3-Layer Linear Transformer')\n",
241 |     "ax.fill_between(Ns, losses_mean-losses_std, losses_mean+losses_std, color = 'red', alpha = 0.2)"
242 |    ]
243 |   },
244 |   {
245 |    "cell_type": "code",
246 |    "execution_count": null,
247 |    "id": "e9ca8fff",
248 |    "metadata": {},
249 |    "outputs": [],
250 |    "source": [
251 |     "def do_gd(Z,eta,numstep):\n",
252 |     "    N = Z.shape[0]-1\n",
253 |     "    X = Z[0:N-1,0:5]\n",
254 |     "    Y = Z[0:N-1,5]\n",
255 |     "    w = torch.zeros(X.shape[1]).to(device)\n",
256 |     "    for k in range(numstep):\n",
257 |     "        XTXw = torch.einsum('ik,ij,j->k',X,X,w)\n",
258 |     "        XTY = torch.einsum('ik,i->k',X,Y)\n",
259 |     "        grad = XTXw - XTY\n",
260 |     "        w = w - eta * grad\n",
261 |     "    return w\n",
262 |     "\n",
263 |     "def eval_w_instance(Z, Ytest, w):\n",
264 |     "    N = Z.shape[0]-1\n",
265 |     "    Xtest = Z[N,0:5]\n",
266 |     "    prediction = torch.einsum('i,i->',w,Xtest)\n",
267 |     "    return (Ytest - prediction)**2, prediction\n",
268 |     "\n",
269 |     "\n",
270 |     "## code for running 3-step GD loss\n",
271 |     "gd_loss_matrix = torch.zeros(len(seeds),10)\n",
272 |     "#for seed in seeds:\n",
273 |     "#    gd_loss_matrix.append([None]*10)\n",
274 |     "    \n",
275 |     "for N in Ns:\n",
276 |     "    #find best eta\n",
277 |     "    #load seed 1 just to find eta\n",
278 |     "    sd = 1\n",
279 |     "    best_loss = 10000\n",
280 |     "    best_eta = 0\n",
281 |     "    numstep = 3\n",
282 |     "    \n",
283 |     "    filename = cur_dir + filename_format.format(n_layer, N, sd)\n",
284 |     "    U = torch.load(filename)['U']\n",
285 |     "    D = torch.load(filename)['D']\n",
286 |     "    #generate test data\n",
287 |     "    np.random.seed(999)\n",
288 |     "    torch.manual_seed(999)\n",
289 |     "    Z, y = generate_data(mode,N,d,B,shape_k,U,D)\n",
290 |     "    Z = Z.to(device)\n",
291 |     "    y = y.to(device)\n",
292 |     "    #done generating data \n",
293 |     "    \n",
294 |     "    for eta in [0.001, 0.002, 0.004, 0.008, 0.01, 0.02, 0.04, 0.08, 0.16]:\n",
295 |     "        ### start of evaluate mean loss ###\n",
296 |     "        total_loss = 0\n",
297 |     "        for i in range(5000):\n",
298 |     "            Zi = Z[i,:,:]\n",
299 |     "            Ytesti = y[i]\n",
300 |     "            w = do_gd(Zi,eta,numstep)\n",
301 |     "            gd_loss, gd_pred = eval_w_instance(Zi, Ytesti, w)\n",
302 |     "            total_loss = total_loss + gd_loss\n",
303 |     "        mean_loss = total_loss / 5000\n",
304 |     "        ### end of evaluate mean loss ###\n",
305 |     "        print('eta: {}, loss: {}'.format(eta, mean_loss))\n",
306 |     "        if (mean_loss < best_loss):\n",
307 |     "            best_eta = eta\n",
308 |     "            best_loss = mean_loss\n",
309 |     "    print('best eta: {} for N={}'.format(best_eta, N))\n",
310 |     "    \n",
311 |     "    #now do actual evaluation\n",
312 |     "    for sd in seeds:\n",
313 |     "        opt_seed = sd\n",
314 |     "        \n",
315 |     "        filename = cur_dir + filename_format.format(n_layer, N, sd)\n",
316 |     "        U = torch.load(filename)['U']\n",
317 |     "        D = torch.load(filename)['D']\n",
318 |     "        #generate test data\n",
319 |     "        torch.manual_seed(sd)\n",
320 |     "        Z, y = generate_data(mode,N,d,B,shape_k,U,D)\n",
321 |     "        Z = Z.to(device)\n",
322 |     "        y = y.to(device)\n",
323 |     "        #done generating data \n",
324 |     "        eta = best_eta\n",
325 |     "        ### start of evaluate mean loss ###\n",
326 |     "        total_loss = 0\n",
327 |     "        for i in range(5000):\n",
328 |     "            Zi = Z[i,:,:]\n",
329 |     "            Ytesti = y[i]\n",
330 |     "            w = do_gd(Zi,eta,numstep)\n",
331 |     "            gd_loss, gd_pred = eval_w_instance(Zi, Ytesti, w)\n",
332 |     "            total_loss = total_loss + gd_loss\n",
333 |     "        mean_loss = total_loss / 5000\n",
334 |     "        gd_loss_matrix[sd,int(N/2-1)] = mean_loss\n",
335 |     "        \n",
336 |     "gd_loss_mean = gd_loss_matrix.mean(dim=0)\n",
337 |     "gd_loss_std = gd_loss_matrix.var(dim=0)**0.5        "
338 |    ]
339 |   },
340 |   {
341 |    "cell_type": "code",
342 |    "execution_count": null,
343 |    "id": "4a2b85e8",
344 |    "metadata": {},
345 |    "outputs": [],
346 |    "source": [
347 |     "def do_preconditioned_gd(Z,eta,numstep,U,D):\n",
348 |     "    N = Z.shape[0]-1\n",
349 |     "    X = Z[0:N-1,0:5]\n",
350 |     "    Y = Z[0:N-1,5]\n",
351 |     "    w = torch.zeros(X.shape[1]).to(device)\n",
352 |     "    X = torch.einsum('ij, jk, Nk -> Ni', (torch.inverse(D),U.t(),X))\n",
353 |     "    for k in range(numstep):\n",
354 |     "        XTXw = torch.einsum('ik,ij,j->k',X,X,w)\n",
355 |     "        XTY = torch.einsum('ik,i->k',X,Y)\n",
356 |     "        grad = XTXw - XTY\n",
357 |     "        w = w - eta * grad\n",
358 |     "    return w\n",
359 |     "\n",
360 |     "def eval_w_instance_precon(Z, Ytest, w, U, D):\n",
361 |     "    N = Z.shape[0]-1\n",
362 |     "    Xtest = Z[N,0:5]\n",
363 |     "    Xtest = torch.einsum('ij, jk, k -> i', (torch.inverse(D),U.t(),Xtest))\n",
364 |     "    prediction = torch.einsum('i,i->',w,Xtest)\n",
365 |     "    return (Ytest - prediction)**2, prediction\n",
366 |     "\n",
367 |     "\n",
368 |     "## code for running 3-step GD loss\n",
369 |     "pgd_loss_matrix = torch.zeros(len(seeds),10)\n",
370 |     "#for seed in seeds:\n",
371 |     "#    gd_loss_matrix.append([None]*10)\n",
372 |     "    \n",
373 |     "for N in Ns:\n",
374 |     "    #find best eta\n",
375 |     "    #load seed 1 just to find eta\n",
376 |     "    best_loss = 10000\n",
377 |     "    best_eta = 0\n",
378 |     "    numstep = 3\n",
379 |     "    \n",
380 |     "    filename = cur_dir + filename_format.format(n_layer, N, sd)\n",
381 |     "    U = torch.load(filename)['U'].to(device)\n",
382 |     "    D = torch.load(filename)['D'].to(device)\n",
383 |     "    #generate test data\n",
384 |     "    np.random.seed(999)\n",
385 |     "    torch.manual_seed(999)\n",
386 |     "    Z, y = generate_data(mode,N,d,B,shape_k,U,D)\n",
387 |     "    Z = Z.to(device)\n",
388 |     "    y = y.to(device)\n",
389 |     "    #done generating data \n",
390 |     "    \n",
391 |     "    for eta in [0.008, 0.01, 0.02, 0.04, 0.08, 0.16]:\n",
392 |     "        ### start of evaluate mean loss ###\n",
393 |     "        total_loss = 0\n",
394 |     "        for i in range(5000):\n",
395 |     "            Zi = Z[i,:,:]\n",
396 |     "            Ytesti = y[i]\n",
397 |     "            w = do_preconditioned_gd(Zi,eta,numstep,U,D)\n",
398 |     "            pgd_loss, pgd_pred = eval_w_instance_precon(Zi, Ytesti, w, U, D)\n",
399 |     "            total_loss = total_loss + pgd_loss\n",
400 |     "        mean_loss = total_loss / 5000\n",
401 |     "        ### end of evaluate mean loss ###\n",
402 |     "        print('eta: {}, loss: {}'.format(eta, mean_loss))\n",
403 |     "        if (mean_loss < best_loss):\n",
404 |     "            best_eta = eta\n",
405 |     "            best_loss = mean_loss\n",
406 |     "    print('best eta: {} for N={}'.format(best_eta, N))\n",
407 |     "    \n",
408 |     "    #now do actual evaluation\n",
409 |     "    for sd in seeds:\n",
410 |     "        opt_seed = sd\n",
411 |     "        \n",
412 |     "        filename = cur_dir + filename_format.format(n_layer, N, sd)\n",
413 |     "        U = torch.load(filename)['U'].to(device)\n",
414 |     "        D = torch.load(filename)['D'].to(device)\n",
415 |     "        #generate test data\n",
416 |     "        torch.manual_seed(sd)\n",
417 |     "        Z, y = generate_data(mode,N,d,B,shape_k,U,D)\n",
418 |     "        Z = Z.to(device)\n",
419 |     "        y = y.to(device)\n",
420 |     "        #done generating data \n",
421 |     "        eta = best_eta\n",
422 |     "        ### start of evaluate mean loss ###\n",
423 |     "        total_loss = 0\n",
424 |     "        for i in range(5000):\n",
425 |     "            Zi = Z[i,:,:]\n",
426 |     "            Ytesti = y[i]\n",
427 |     "            w = do_preconditioned_gd(Zi,eta,numstep,U,D)\n",
428 |     "            pgd_loss, pgd_pred = eval_w_instance_precon(Zi, Ytesti, w, U, D)\n",
429 |     "            total_loss = total_loss + pgd_loss\n",
430 |     "        mean_loss = total_loss / 5000\n",
431 |     "        pgd_loss_matrix[sd,int(N/2-1)] = mean_loss\n",
432 |     "        \n",
433 |     "pgd_loss_mean = pgd_loss_matrix.mean(dim=0)\n",
434 |     "pgd_loss_std = pgd_loss_matrix.var(dim=0)**0.5"
435 |    ]
436 |   },
437 |   {
438 |    "cell_type": "code",
439 |    "execution_count": null,
440 |    "id": "e2644ed4",
441 |    "metadata": {},
442 |    "outputs": [],
443 |    "source": [
444 |     "def do_OLS(Z,eta,numstep,U,D):\n",
445 |     "    N = Z.shape[0]-1\n",
446 |     "    X = Z[0:N-1,0:5]\n",
447 |     "    Y = Z[0:N-1,5]\n",
448 |     "    w = torch.zeros(X.shape[1])\n",
449 |     "    X = torch.einsum('ij, jk, Nk -> Ni', (torch.inverse(D),U.t(),X))\n",
450 |     "    for k in range(numstep):\n",
451 |     "        XTX = torch.einsum('ik,ij->kj',X,X)\n",
452 |     "        XTY = torch.einsum('ik,i->k',X,Y)\n",
453 |     "        w = torch.einsum('ik,k->i', torch.linalg.pinv(XTX) , XTY)\n",
454 |     "    return w\n",
455 |     "\n",
456 |     "def eval_w_instance_OLS(Z, Ytest, w,U,D):\n",
457 |     "    N = Z.shape[0]-1\n",
458 |     "    Xtest = Z[N,0:5]\n",
459 |     "    Xtest = torch.einsum('ij, jk, k -> i', (torch.inverse(D),U.t(),Xtest))\n",
460 |     "    prediction = torch.einsum('i,i->',w,Xtest)\n",
461 |     "    return (Ytest - prediction)**2, prediction\n",
462 |     "\n",
463 |     "\n",
464 |     "## code for running 3-step GD loss\n",
465 |     "OLS_loss_matrix = torch.zeros(len(seeds),10)\n",
466 |     "#for seed in seeds:\n",
467 |     "#    gd_loss_matrix.append([None]*10)\n",
468 |     "    \n",
469 |     "for N in Ns:\n",
470 |     "    #now do actual evaluation\n",
471 |     "    for sd in seeds:\n",
472 |     "        opt_seed = sd\n",
473 |     "        filename = cur_dir + filename_format.format(n_layer, N, sd)\n",
474 |     "        U = torch.load(filename)['U'].to(device)\n",
475 |     "        D = torch.load(filename)['D'].to(device)\n",
476 |     "        #generate test data\n",
477 |     "        torch.manual_seed(sd)\n",
478 |     "        Z, y = generate_data(mode,N,d,B,shape_k,U,D)\n",
479 |     "        Z = Z.to(device)\n",
480 |     "        y = y.to(device)\n",
481 |     "        #done generating data \n",
482 |     "        eta = best_eta\n",
483 |     "        ### start of evaluate mean loss ###\n",
484 |     "        total_loss = 0\n",
485 |     "        for i in range(5000):\n",
486 |     "            Zi = Z[i,:,:]\n",
487 |     "            Ytesti = y[i]\n",
488 |     "            w = do_OLS(Zi,eta,numstep,U,D)\n",
489 |     "            OLS_loss, OLS_pred = eval_w_instance_OLS(Zi, Ytesti, w, U, D)\n",
490 |     "            total_loss = total_loss + OLS_loss\n",
491 |     "        mean_loss = total_loss / 5000\n",
492 |     "        OLS_loss_matrix[sd,int(N/2-1)] = mean_loss\n",
493 |     "        print('N={}, loss={}'.format(N,mean_loss))\n",
494 |     "        \n",
495 |     "ols_loss_mean = OLS_loss_matrix.mean(dim=0)\n",
496 |     "ols_loss_std = OLS_loss_matrix.var(dim=0)**0.5"
497 |    ]
498 |   },
499 |   {
500 |    "cell_type": "code",
501 |    "execution_count": null,
502 |    "id": "5af8555c",
503 |    "metadata": {},
504 |    "outputs": [],
505 |    "source": [
506 |     "####################################\n",
507 |     "# plot final test loss against N\n",
508 |     "####################################\n",
509 |     "\n",
510 |     "fig_dir = 'figures' \n",
511 |     "os.makedirs(fig_dir, exist_ok=True)\n",
512 |     "\n",
513 |     "fig, ax = plt.subplots(1, 1,figsize = (9, 9))\n",
514 |     "\n",
515 |     "losses = torch.zeros(len(seeds), len(Ns))\n",
516 |     "keys = loss_dict.keys()\n",
517 |     "for idx, key in enumerate(keys):\n",
518 |     "    losses[idx,:] = loss_dict[key]\n",
519 |     "losses_mean = torch.mean(losses, axis=0)\n",
520 |     "losses_std = torch.std(losses, axis=0)\n",
521 |     "\n",
522 |     "plt.plot(Ns, gd_loss_mean, color='blue', label='3-Step GD')\n",
523 |     "plt.fill_between(Ns, gd_loss_mean - gd_loss_std, gd_loss_mean + gd_loss_std, color='blue', alpha=0.2)\n",
524 |     "plt.plot(Ns, pgd_loss_mean, color='green', label='3-Step Preconditioned GD')\n",
525 |     "plt.fill_between(Ns, pgd_loss_mean - pgd_loss_std, pgd_loss_mean + pgd_loss_std, color='green', alpha=0.2)\n",
526 |     "ax.plot(Ns, losses_mean, color = 'red', lw = 3, label='3-Layer Linear Transformer')\n",
527 |     "ax.fill_between(Ns, losses_mean-losses_std, losses_mean+losses_std, color = 'red', alpha = 0.2)\n",
528 |     "plt.plot(Ns, ols_loss_mean, color='purple', label='OLS')\n",
529 |     "plt.fill_between(Ns, ols_loss_mean - ols_loss_std, ols_loss_mean + ols_loss_std, color='purple', alpha=0.2)\n",
530 |     "\n",
531 |     "plt.ylabel('Loss',fontsize=30)\n",
532 |     "plt.xlabel('Number of ICL Examples',fontsize=30)\n",
533 |     "ax.tick_params(axis='both', which='major', labelsize=30, width = 3, length = 10)\n",
534 |     "ax.tick_params(axis='both', which='minor', labelsize=20, width = 3, length = 5)\n",
535 |     "plt.xticks(np.arange(2,24, 4))\n",
536 |     "ax.legend(fontsize=24)\n",
537 |     "#ax.set_yscale('log')\n",
538 |     "\n",
539 |     "\n",
540 |     "plt.tight_layout()\n",
541 |     "plt.savefig(fig_dir + '/3-step-variable-N-plot.pdf', dpi=600)"
542 |    ]
543 |   }
544 |  ],
545 |  "metadata": {
546 |   "kernelspec": {
547 |    "display_name": "Python 3 (ipykernel)",
548 |    "language": "python",
549 |    "name": "python3"
550 |   },
551 |   "language_info": {
552 |    "codemirror_mode": {
553 |     "name": "ipython",
554 |     "version": 3
555 |    },
556 |    "file_extension": ".py",
557 |    "mimetype": "text/x-python",
558 |    "name": "python",
559 |    "nbconvert_exporter": "python",
560 |    "pygments_lexer": "ipython3",
561 |    "version": "3.9.12"
562 |   }
563 |  },
564 |  "nbformat": 4,
565 |  "nbformat_minor": 5
566 | }
567 | 


--------------------------------------------------------------------------------
/plot_stochastic_noise.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "code",
  5 |    "execution_count": 1,
  6 |    "metadata": {},
  7 |    "outputs": [],
  8 |    "source": [
  9 |     "import torch\n",
 10 |     "%matplotlib inline\n",
 11 |     "from matplotlib import pyplot as plt\n",
 12 |     "import math\n",
 13 |     "import torch.nn.functional as F\n",
 14 |     "from torch.nn.functional import relu\n",
 15 |     "from torch import nn\n",
 16 |     "import torch.optim as optim\n",
 17 |     "import torch.optim.lr_scheduler as lr_scheduler\n",
 18 |     "import random\n",
 19 |     "import numpy as np\n",
 20 |     "import gc\n",
 21 |     "from pylab import *\n",
 22 |     "import os\n",
 23 |     "import random\n",
 24 |     "import json\n",
 25 |     "import pandas as pd\n",
 26 |     "from scipy.stats import norm\n",
 27 |     "pd.set_option('display.float_format', lambda x: '%.5f' % x)\n",
 28 |     "import sys\n",
 29 |     "import matplotlib.pyplot as plt\n",
 30 |     "import time\n",
 31 |     "\n",
 32 |     "from linear_transformer import Transformer_F, attention, generate_data, in_context_loss, generate_data_inplace\n",
 33 |     "\n",
 34 |     "np.set_printoptions(precision = 4, suppress = True)\n",
 35 |     "torch.set_printoptions(precision=2)\n",
 36 |     "device = torch.device(\"cuda\")\n",
 37 |     "torch.cuda.set_device(0)"
 38 |    ]
 39 |   },
 40 |   {
 41 |    "cell_type": "code",
 42 |    "execution_count": 2,
 43 |    "metadata": {},
 44 |    "outputs": [],
 45 |    "source": [
 46 |     "# Set Hyperparameters\n",
 47 |     "\n",
 48 |     "# Fixed\n",
 49 |     "n_head = 1\n",
 50 |     "d = 5\n",
 51 |     "B = 1000\n",
 52 |     "ma = 1\n",
 53 |     "var = 0.05\n",
 54 |     "shape_k = 0.1\n",
 55 |     "\n",
 56 |     "# We vary the following parameters\n",
 57 |     "n_layer = 3\n",
 58 |     "mode = 'normal'\n",
 59 |     "N = 20\n",
 60 |     "n_sample = 10000 # number of stochastic gradients to sample\n",
 61 |     "seed = 1\n",
 62 |     "\n",
 63 |     "log_dir = 'log' \n",
 64 |     "os.makedirs(log_dir, exist_ok=True)"
 65 |    ]
 66 |   },
 67 |   {
 68 |    "cell_type": "code",
 69 |    "execution_count": 3,
 70 |    "metadata": {},
 71 |    "outputs": [],
 72 |    "source": [
 73 |     "np.random.seed(seed)\n",
 74 |     "torch.manual_seed(seed)\n",
 75 |     "\n",
 76 |     "model = Transformer_F(n_layer, n_head, d, var)\n",
 77 |     "model.to(device)\n",
 78 |     "\n",
 79 |     "# compute estimated true gradient using large batch\n",
 80 |     "B_large = B*100\n",
 81 |     "Z, y = generate_data(mode,N,d,B_large,shape_k)\n",
 82 |     "Z = Z.cuda()\n",
 83 |     "y = y.cuda()\n",
 84 |     "\n",
 85 |     "# redefine loss using newly sampled Z and y\n",
 86 |     "def eval_loss():\n",
 87 |     "    output = model(Z)\n",
 88 |     "    N= Z.shape[1]-1\n",
 89 |     "    diff = output[:,N,d]+y\n",
 90 |     "    loss = ((diff)**2).mean() \n",
 91 |     "    loss = loss \n",
 92 |     "    return loss\n",
 93 |     "\n",
 94 |     "loss = eval_loss()\n",
 95 |     "loss.backward()\n",
 96 |     "gradient = model.allparam.grad.data.clone().detach()\n",
 97 |     "model.allparam.grad.zero_()\n",
 98 |     "\n",
 99 |     "noiseList = []\n",
100 |     "for _ in range(n_sample):\n",
101 |     "    # compute stochastic gradient\n",
102 |     "    Z, y = generate_data(mode,N,d,B,shape_k)\n",
103 |     "    Z = Z.cuda()\n",
104 |     "    y = y.cuda()\n",
105 |     "    \n",
106 |     "    loss = in_context_loss(model, Z, y)\n",
107 |     "    loss.backward()\n",
108 |     "    stochastic_gradient = model.allparam.grad.data.clone().detach()\n",
109 |     "    model.allparam.grad.zero_()\n",
110 |     "\n",
111 |     "    noise = torch.norm(stochastic_gradient - gradient)\n",
112 |     "    noiseList.append(noise.item())\n",
113 |     "\n",
114 |     "filename = log_dir + '/stochastic_gradient_noise_layer{}_N{}_{}_sd{}.pth'.format(n_layer,N,mode,seed)\n",
115 |     "torch.save({'noiseList':noiseList}, filename)"
116 |    ]
117 |   },
118 |   {
119 |    "cell_type": "code",
120 |    "execution_count": 6,
121 |    "metadata": {},
122 |    "outputs": [
123 |     {
124 |      "data": {
125 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAJOCAYAAABBWYj1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABXY0lEQVR4nO3dd3xUVf7/8fckJCGEkkICBDWhS3EliFKUogjSFFGUphQF1F3LytdVXCv2srJ2RVFxxYKwoCxIN6hIQKQjBkIgIM2ETiCk3t8f/jLmZibkTslMJnk9H4889J7MPfczGZK8c+6Zc2yGYRgCAABAuYL8XQAAAECgIDgBAABYRHACAACwiOAEAABgEcEJAADAIoITAACARTX8XQCqtqKiIh05csTUFhMTo6AgMjsAIPAQnFChjhw5ori4OFNbZmamYmNj/VQRAADu489+AAAAiwhOAAAAFhGcAAAALCI4AQAAWERwAgAAsIjgBAAAYBHBCQAAwCKCEwAAgEUEJwAAAIsITgAAABYRnAAAACwiOAEAAFhEcAIAALCI4AQAAGARwQkAAMAighMAAIBFBCcAAACLCE4AAAAWEZwAAAAsIjgBAABYRHACAACwiOAEAABgEcEJAADAohr+LgAASkuctMB0nPHCAD9VAgBmjDgBAABYRHACAACwiOAEAABgEcEJAADAIoITAACARQQnAAAAiwhOAAAAFhGcAAAALGIBTAAVjgUtAVQVjDgBAABYRHACAACwiOAEAABgEcEJAADAIoITAACARQQnAAAAiwhOAAAAFhGcAAAALCI4AQAAWERwAgAAsIjgBAAAYBF71QHwu9J72QFAZcWIEwAAgEUEJwAAAIsITgAAABYRnAAAACwiOAEAAFhEcAIAALCI4AQAAGARwQkAAMAighMAAIBFBCcAAACLCE4AAAAWEZwAAAAsIjgBAABYRHACAACwiOAEAABgEcEJAADAIoITAACARQQnAAAAiwhOAAAAFhGcAAAALCI4AQAAWERwAgAAsIjgBAAAYBHBCQAAwCKCEwAAgEUEJwAAAIsITgAAABYRnAAAACwiOAEAAFhEcAIAALCohr8LAFD1JE5aUKH9ZbwwwKv9A4BVjDgBAABYRHACAACwiOAEAABgEXOcAAScknOemO8EwJcYcQIAALCI4AQAAGARwQkAAMAighMAAIBFBCcAAACLCE4AAAAWEZwAAAAsYh0nAD7n7b3sAMBXGHECAACwiBEnAB5jBAlAdUFwAmBJ6XDEVicAqiNu1QEAAFhEcAIAALCI4AQAAGARwQkAAMAighMAAIBFBCcAAACLCE4AAAAWEZwAAAAsYgFMAAGNhTkB+BIjTgAAABYRnAAAACwiOAEAAFjEHKcqbP369UpLS9P+/fslSY0bN1bLli2VlJTk58oAAAhMBCc37Nq1S2vXrtXPP/+stWvXav369Tp16pT98wkJCcrIyPBLbfn5+XrllVc0bdo0paenO31M8+bNNW7cOE2cOFEhISE+rhAAgMBFcLJoxYoVev755/Xzzz/r6NGj/i7HqbS0NA0bNkzr168/5+N27typSZMmadasWfriiy/UvHlzH1UIAEBgIzhZtHHjRi1ZssTfZZTp0KFD6t27t/bs2WNqb968udq2bSvDMPTLL7+YRqHWrVunPn36aPXq1YqLi/N1yQAABBwmh3soLCxMzZo182sNRUVFuv76602hqVGjRlq8eLHS0tL01Vdf6euvv9bOnTu1cOFCNWzY0P643bt3a/DgwTIMwx+lAwAQUAhOLggJCVH79u01btw4TZ06VevWrdOpU6c0bdo0v9b16aefas2aNfbj6OhorVq1Sn369HF4bN++fbVq1SpFRUXZ21atWqWZM2f6pFYAAAIZt+osGj16tO68807VrFnT36WYFBYW6oknnjC1TZkyRYmJiWWe06RJE02ZMkVjx461tz366KO6+eabFRRElgYAoCz8lrQoKiqq0oUmSVq5cqV2795tP27cuLFuueWWcs+79dZb1bhxY/txenq6Vq1aVSE1AgBQVRCcAtzcuXNNx6NGjVJwcHC55wUHBzsErDlz5ni1NgAAqhqCU4BbtGiR6bhnz56Wzy392IULF3qhIgAAqi6CUwDLzc3Vzp07TW2dO3e2fH7Xrl1Nx2lpacrLy/NKbQAAVEUEpwC2fft2FRYW2o/j4uJUt25dy+fXrVtX9evXtx8XFhZqx44dXq0RAICqhOAUwEqPNl1wwQUu91H6nLS0NI9qAgCgKmM5ggB2/Phx07E7q3+XPufEiRPlnpOVlWW5/8OHD7tcEwAAlRXBKYBlZ2ebjsPDw13uo/Q5JTcrLgvbswAAqitu1QWw0sHJnXWmSgen0n0CAIA/EZyqEJvN5pNzAACorghOAax27dqm45ycHJf7KH1O6T4BAMCfmOMUwPwVnDIzMy33f/jwYbVp08blugAAqIwITgGsXr16pmNX3u1WrHQIioyMLPec2NhYl68DAEBVQHAKYC1atDAd79mzx+U+Sp9Tuk+gLImTFvi7BADwOeY4BbBWrVqZNvTNzMy0tJxAsZMnT5rWWQoODiY4AQBwDgSnABYWFqZmzZqZ2lJSUiyfv2rVKtNxixYtFBYW5pXaAACoighOAa5v376m4xUrVlg+t/Rj+/Xr54WKAACoughOAW7w4MGm408++cS08W9ZCgsLNWPGjHP2BQAAzAhOAa5bt25q0qSJ/Xjfvn0OgciZGTNmaP/+/fbjZs2a6fLLL6+QGgEAqCoITpWMzWYzfZR36y04OFiTJ082tU2cOFEZGRllnpORkaH777/f1PbMM88oKIh/DgAAnItPf1Pedttt+umnn3x5Sa/at2+fMjIyHD4OHTpkelxBQYHTx2VkZJjexeYtI0eOVKdOnezHR48eVdeuXbVkyRKHxy5evFhdunTRsWPH7G1du3bV0KFDvV4XAABVjc0wDMNXFwsKCpLNZtNFF12kO+64Q7fccovq1Knjq8t7LDEx0a21kkoaPXq0pk+fXubnS+8dl5ycrJ49e5bb78GDB9W5c2ft3bvX1N6iRQu1bdtWhmHol19+0c6dO02fT0xM1OrVq9WgQQPLz8EVWVlZiouLM7VlZmayiGYACpR1mzJeGODvEgBUYT6/N2MYhjZv3qy7775b8fHxGj9+vNauXevrMqqcRo0aaenSpUpKSjK1p6Wl6auvvtLXX3/tEJo6dOigpUuXVlhoAgCgqvF5cCoeUTEMQ6dPn9aHH36ozp07KykpSVOnTlV2dravS6oyWrZsqTVr1uj5559X06ZNy3xcs2bN9Pzzz2v16tVq3ry5DytEIEmctMD0AQDw0626kkpe3mazKSIiQsOHD9f48ePVsWNHX5VWJa1bt047duzQgQMHJEnx8fFq2bKlLrnkEp/VwK26wBWoYYlbdQAqkk+D07Bhw/TVV18pLy+v3AAlSe3bt9edd96pESNGKCIiwldlwosIToGL4AQAjnx6q+6LL77Q/v379dJLL6lFixYyDMMemEq+Bb+4fcOGDbrzzjvVqFEj3XnnnVq/fr0vywUAADDx+RynmJgYPfDAA0pNTdW3336roUOHKjQ01GHEqeRcqOzsbL3//vu69NJLdemll+qDDz7QmTNnfF06AACo5vy64mHPnj31+eefa//+/Xr55ZfVsmXLckeh1q1bpwkTJqhRo0b661//qg0bNvjzKQAAgGqkUiwVHR0drf/7v//Tr7/+quTkZA0bNqzcUahTp05p6tSp6tixoy677DJ9+OGHjEIBAIAKVSmCU0k9evTQZ5995tIo1M8//6zx48crPj5ed999tzZt2uTnZwEAAKqiShecipUchVqxYoWGDx9e7ijUyZMn9c4776hDhw7q3Lmzpk+frpycHH89BQAAUMVU2uBUUvfu3fXpp59q//79+te//qVWrVqVOwr1008/6fbbb1d8fLzuuecebdmyxc/PAgAABLqACE7FoqOjNXHiRG3bts1hFKp0iJL+GIU6ceKE3n77bbVv315du3bVJ598ory8PH8+DQAAEKACKjiVVDwKdeDAAU2ZMsU+CiX9EZicjUKtWbNGY8aM0fnnn6/HHntMR48e9fOzAAAAgSRgg1Ox48eP6/fff9exY8dM7c7mQhUHqKysLD333HNq0qSJJk+erNzcXF+XDQAAAlBABqfCwkLNnj1bffr0UYsWLfTSSy8pKytLknm0qfi4rLlQp06d0lNPPaWkpCRt3LjRX08HAAAEiBr+LsAVu3bt0vvvv6/p06crMzNTkvM97orbw8PDNXToUB07dkwLFixQQUGB/TElg1Vqaqp69uypJUuW6LLLLvPhMwIAAIGk0gengoICffXVV3rvvff07bffmkaQJMewJEktW7bUnXfeqTFjxigyMlKS9Pvvv+vDDz/U+++/r4yMDIcAdfLkSY0cOVJbt25VWFiYj54dAAAIJJX2Vl16eromTZqk8847T0OHDtXy5ctVVFTkMPFb+iMwBQcH68Ybb9SyZcuUmpqqv//97/bQJEkNGjTQww8/rPT0dH366adq3ry5KYBJf4xoff755758mgAAIIBUquBUUFCgWbNmqXfv3mrVqpVefvllZWZm2keZnM1dio+P15NPPqk9e/Zo1qxZuuqqq855DZvNpuHDh2vLli0aN26cQ3iaO3duhT0/AAAQ2CrFrbr09HT73KWSk7wl86244nabzabevXvrrrvu0nXXXaegINfzX2hoqN59912tXLlS27dvt08YZ9NgAABQFr8Fp4KCAs2ZM0fvvfeeVqxYYWnuUnR0tMaOHas777xTzZo187iGoKAg3XHHHbr//vvt1zt8+LDH/QIAgKrJ58EpPT1d7733nj7++GNLo0uS1KlTJ911110aOnSo1ydut2jRwnTMmk5AYEuctMB0nPHCAD9VAqAq8mlwuvrqq5WcnCyp7GUEij8XERGhESNG6K677lL79u0rrKaYmJgK6xsAAFQtPg1O3377rf3/yxpdatu2re68806NGjVKderU8WV5AKogRqAAeJPPb9U5C0whISG64YYb9Ne//lXdunXzdUn2OgAAAM7FL5PDi0NKQkKCJkyYoNtvv11xcXH+KEWdOnVSUVGRX64NAAACi19GnPr166e77rpL/fv3dxiBAgAAqKx8Gpweeugh3XnnnUpISPDlZQEAALzCp8Hp+eef9+XlAAAAvKpSbbkCAABQmfl0xOn77783HXfo0EG1a9f2+nVOnTrlsHVK9+7dvX4dAABQvfg0OPXs2dM0GTwlJUWXXXaZ16+zbds207VsNpsKCgq8fh0AAFC9+OVWnS/XTCq9Bx4AAIC7mOMEAABgEcEJAADAIoITAACARVUyOOXl5ZmOQ0ND/VQJAACoSqpkcDp69KjpuE6dOn6qBAAAVCVVMjht2rTJdBwZGemfQgAAQJVS5YJTXl6eZsyYIZvNJsMwZLPZ1Lp1a3+XBQAAqgCfLoBZWsnFMD118uRJ/fjjj3rhhRe0c+dOU9/t27f32nWAqipx0gJ/lwAAlZ7XgtP333+vMWPGuHTO9ddfr7CwMI+um5+fr9OnT+vEiRNlPmbw4MEeXQMAAEDyYnDKyclRRkaG/RZZWYo/ZxiGDh486K3Lm5TcauWyyy5jxAkAAHhFhdyqK+sWXOlA5c1bdc6uVatWLX300UcVdg0AAFC9VKnJ4cX70hmGoQsvvFArV67UhRde6O+yAABAFeH1ESdXNtT15ua7oaGhat26tS655BINGTJE11xzTYWOaAEAgOrHa8GpR48e2r17d5mfNwxDTZs2NS0TMGfOHI/mH9lsNoWFhalOnTqqVauW2/0AAABY4bXgVLNmTSUkJLh0TqNGjVw+BwAAwF+q1BwnAACAiuTTBTC7d+9umndUt25dX14eAADAIz4NTitWrPDl5QAAALyKW3UAAAAWEZwAAAAsIjgBAABYRHACAACwiOAEAABgkUfvqnvqqaectj/++OMuPb6ilVUPAACAK2yGBxvGBQUFOd0PrrCw0KXHV7Sy6kHFy8rKUlxcnKktMzNTsbGxfqoIZUmctMDfJfhExgsD/F0CgADmlXWcSmYvK8HIm5v7loeNfgEAgLd4JTgVhxOrgchXYcaXAQ0AAFR9HgcnV8MJYQYAAAQqj4LTE088UaGPBwAAqEw8mhwOlIfJ4YGDyeEAUD7WcQIAALCI4AQAAGARwQkAAMAighMAAIBFXlnHCUDgqS6TwctT+uvA5HEA50JwAlCtEBgBeCJgg9O+ffu0cuVKHTp0SIWFhWrcuLE6duyo5s2b+7s0AABQRQVccFq7dq0efPBBff/9904/f+mll+rll19Wt27dfFwZAACo6nw6OTw3N1dt2rRR06ZN7R9333235fM/++wzdevWTd9//70Mw3D68dNPP+nKK6/UU089VYHPBAAAVEc+HXFavHixUlNTZbPZZBiGbDabbrjhBkvnrl27VmPHjlV+fr6kc28UXFRUpMmTJ6tevXq67777vFI7AACAT0ec/ve//5mOmzRpoquuusrSuePGjVN+fr5sNpspNJUcbSpWHMwefPBBbd++3TvFAwCAas+nwSklJcU02jRw4EBL582ZM0dbtmxxCExhYWG6+uqrNWzYMF166aUqve1efn6+7r//fq8+BwAAUH357FZddna2fv31V1Nb//79LZ373nvv2f+/OHR17txZs2fPVnx8vP1zP/74o4YMGaLMzEx7QFu8eLF2796tJk2aeOeJAACAastnI07p6ekOI0JJSUnlnpeZmanly5ebRpvq1KmjuXPnmkKTJF1++eX64osvHK7z2WefeVA5AADAH3wWnDIyMkzHsbGxio2NLfe8JUuWqLCwUNKfo0233XabGjRo4PTxPXr0UJ8+feyPlaTvvvvOs+IBAADkw+B06NAh03GjRo0snZecnOzQNnr06HOeM2TIEPv/G4ahTZs2WboWAADAufgsOJ05c8b+/zabTXXr1rV03vfff2+6TXfeeefp4osvPuc5HTt2NB0fPnxYx44dc6FaAAAARz4LTjk5OabjmjVrlnvO4cOHlZ6eLunP23RWli9ITEx0aDt+/LilOgEAAMris+AUEhJiOi45AlWWH3/80aGte/fu5Z5Xp04dh7aTJ0+Wex4AAMC5+Cw41atXz/7/hmHo4MGD5Z7jbFL3FVdcUe55xauLl1T6nXYAAACu8llwiomJMR3v3btXZ8+ePec533zzjWl+U1xcnFq0aFHutZzNZ4qIiLBYKQAAgHM+C07t2rUzHRcWFmrFihVlPn7z5s3asWOHpD/nN/Xo0cPStQ4fPuzQVnLECwAAwB0+C04tW7Z0GPV54403yny8s89deeWVlq61ZcsW03HNmjUVFxdn6VwAAICy+Cw4Fe9NVzx6ZBiGFi1apClTpjg8dsmSJfroo49Mt+mCg4M1aNAgS9dav3696ZjtVgAAgDf4bK866Y+FK2fOnClJ9vD0j3/8Q/Pnz9fAgQNVp04drV69WjNmzFBRUZFpQ+A+ffqoYcOGlq7zww8/SPrzFl/p24QAAADu8Glw6tu3r7p166YffvhBNpvNHoy+++470zvoSm6XUuzRRx+1dI3ffvtNa9eutfctSV26dPHekwAAANWWz27VFXv33XdNE7WLA07Jj5KhyWazacyYMercubOl/mfPnu3QRnACAADe4PPg1Lp1ay1YsEB169a1jwgVjz4VfxQzDENXXnml3n77bcv9v//++6Y+oqKiHLZgAQAAcIdPb9UV69q1q3799Vf93//9n+bMmaPc3FyHx0RFRemBBx7Qgw8+qODgYEv9zp8/X6mpqfZjm82ma665RkFBPs+HQKWTOGmBv0sAgIDnl+AkSQ0bNtSnn36qU6dOacWKFfrtt9904sQJRUZGqm3bturSpYvDNi3l2bp1q8M778aOHevNsgEAQDVmM9iLBBUoKyvLYQ2tzMxMxcbG+qmi6osRJ/dkvDDA3yUAqES4hwUAAGARwQkAAMAighMAAIBFBCcAAACLCE4AAAAWEZwAAAAs8ts6TqUVFBRoy5Yt2rFjh06cOKETJ07o7Nmz8sZqCY8//rgXKgQAANWdX4NTTk6OZs6cqY8++kg//fST8vLyKuQ6BCcAAOANfgtOs2fP1t13362srCxJ8srIkjMl960DAADwhF+C09///ne98cYbprBUEQGHRdEBAIA3+Tw4PfXUU3r99dclOQ9LhB0AAFBZ+TQ47dixQ88884xDYCoOSzExMUpKSlLz5s1Vt25dhYeH+7I8AACAc/JpcHryySdVUFBgCk6GYeiKK67Q448/rl69ejEnCQAAVFo+C04FBQWaP3++PRgZhiGbzaZJkybpueee81UZAAAAbvPZApirV69Wdna2pD9D04ABAwhNAAAgYPgsOGVkZDi0TZ482VeXBwAA8JjPglPxek3FGjRooKSkJF9dHgAAwGM+C05nz561/7/NZlNiYqKvLg0AAOAVPgtOkZGRpuNatWr56tIAAABe4bPg1Lx5c/v/G4bhcOsOAACgsvNZcOrYsaOCg4PtxxkZGSosLPTV5QEAADzms+AUFRWlnj172lcJz87O1qpVq3x1eQAAAI/5LDhJ0sSJEyX9uUdd8Z51AAAAgcCnwalfv37q16+fDMOQYRiaM2eOli9f7ssSAAAA3GYziu+d+UhWVpY6duyoffv2yTAMRUZG6ttvv1X79u19WQZ8JCsrS3Fxcaa2zMxMxcbG+qmi6itx0gJ/lxDwMl4Y4O8SAPiZT0ecJCk2NlZLly5V48aNJUnHjx9X165d9eabb6qgoMDX5QAAAFjm8+AkSS1bttSaNWvUvXt3SX8sjnnfffepSZMmeuSRR7RkyRLt379fOTk5/igPAADAKZ/eqiu5HEFpxWUUTxz3FpvNxkiWH3GrrvLgVp3nuFUHoIYvL1ZWRrPZbPbA5OMpVwAAAJb5NDhJjiNKxe+wK/6cN0ecCGGozhhhAgDv83lwKs3bt+YAAAAqis+DE6NAAAAgUPk0OBUVFfnycgAAAF7ll+UIAAAAAhHBCQAAwCKCEwAAgEV+f1cdAO9hCQIAqFiMOAEAAFhEcAIAALCI4AQAAGBRpZrjlJ6erpUrV+rHH3/U5s2bdfToUR09elQnTpxQUVERG/YCAAC/8ntwMgxDX3/9tV599VX98MMPpnZXvfvuu3r99dftx6GhoVq5cqVq167tlVoBAED15tfglJaWppEjR2rdunWSHMNSyX3srASpQYMG6b777lNBQYEMw5DNZtPs2bM1ZswYr9YNAACqJ7/Ncfr888/VoUMHrVu3ToZh2INOyQ9XNWrUSDfeeKO9L0n6+OOPvV06AACopvwy4vTVV19p1KhRKiwslPTnyFJZo0quhKiRI0fqiy++sPe3cuVKnThxQvXq1fOwaud2796tjRs36sCBA8rOzlajRo2UkJCgrl27KiQkpEKuCQAA/MPnwWndunUaPny4CgsLHW7F/eUvf9Ett9yibt26KTExUTt27FCPHj1c6v/qq69WRESEzpw5I+mPjYWTk5N1/fXXe/NpaPbs2ZoyZYpSUlKcfj46OlpDhw7VU089pfr163v12qX17NlT3333ndvnf/TRR9zOBCwovcBoxgsD/FQJAH/x+a26e++9V7m5uaZRpvr162vu3LnauHGjHnjgAXXq1EkNGjRQaGioy/2HhYXp6quvNo1eJScne63+7OxsDR8+XDfddFOZoUmSjh49qnfeeUft2rXT4sWLvXZ9AADgPz4NTv/973+VkpJiCk1NmjTRhg0bNGjQIK9d55JLLpH05y2+TZs2eaXfwsJCDR061H4rsFhsbKz69Omjm266SR06dDCNpP3+++8aNGiQVq5c6ZUaAACA//j0Vl3JidqGYSg8PFwLFy5UfHy8V6/Tvn1703VSU1O90u+kSZP0zTff2I9DQkI0ZcoUTZgwwTQ6tm3bNo0bN84+IpWbm6vrr79eW7ZsUaNGjbxSy7ns3r3bpcdX9K1EAACqCp8Fp9zcXC1fvlw2m83+rrf77rtPLVu29Pq1SveZlZWl7Oxsj9Zz2rVrl1577TVT26xZs5yOlLVp00bLly9Xr1697OHpyJEjmjx5st599123a7AqMTGxwq8BAEB15LNbdT/99JNycnJMbbfffnuFXCsyMtKh7dixYx71OXnyZOXn59uPx4wZc87bi+Hh4Zo+fbppJOqDDz7Qrl27PKoDAAD4j8+C04EDB0zH559/vpo1a1Yh13K29MDJkyfd7i8nJ0ezZ882tT300EPlnteyZUvTu/kKCgr02WefuV0HAADwL58Fp99//93+/zabTY0bN66wazlb9+ns2bNu97d48WL78gaS1KVLF1144YWWzh07dqzpeM6cOW7XAQAA/MtnwSk7O9t0HB4eXmHXcnZbLiwszO3+Fi1aZDru2bOn5XO7deumGjX+nEq2YcMGU4gEAACBw2fBqVatWvb/NwxDR44cqbBrHTx40KEtKirK7f62bt1qOu7SpYvlcyMiInTRRReZ2n755Re3awEAAP7js+DUoEED03FGRoaljXvdsWbNGtNxjRo1PFry4NdffzUdN2/e3KXzS8/l2rZtm9u1WHHffffpsssuU1xcnEJDQxUdHa0WLVro2muv1UsvvaQdO3ZU6PUBAKiqfBacSoeNkydPauPGjRVyreXLl5uO27Rp49amwdIfK4AfPXrU1HbBBRe41Efpx6elpblVi1Wvv/661q5dq6ysLOXn5+vYsWPauXOn5s+fr4ceekitW7fWDTfcoPT09AqtAwCAqsZn6zi1b99eYWFhysvLs7dNnz5dSUlJXr3O/v379fXXX5vWi+ratavb/R0/ftx0XKtWLUVERLjUR1xcnOn4xIkTbtfjDUVFRZo7d66WL1+uDz/8UDfeeKNL52dlZVl+7OHDh10tDwCASstnwSkkJES9evXSN998Yw81H3zwgR588EGvvsOueL2lkiNM1113ndv9eWNSe+lzTp065XY953LRRRepX79+at++vZo3b67IyEjl5uYqMzNTKSkpmjlzprZs2WJ//MmTJzV06FDNmzdP/fv3t3yd0kEQAIDqwqdbrtx6662mLUvOnDmjm2++Wd99953pnWfu+vLLLzVt2jRTaGrcuLF69+7tdp+lg1PNmjVd7qN0cCrdp6dGjBiht956S23bti3zMVdddZUeeeQRffrpp7rrrrvs4a14/73U1NQKXSICAICqwKeb/N50001q2rSppD/XWlq9erWuueYah1tirvrkk080atQo0wbCNptN999/v4KCvPc03Zkr5e78KqsmTJhwztBU0siRI7V8+XLTuxyzs7M1efLkiioPAIAqw6fBKSgoSP/+97/t76YrvmW3YsUKtW3bVtOmTTNta2JFenq6hg4dqjFjxpjmT9lsNjVv3lx33323RzWX3t+u9LYxVpQ+x5M987zh0ksv1TPPPGNq+/jjj3X69Gk/VQQAQGDw6a06Sbr22ms1YcIEvffee7LZbPbwdPDgQd1xxx164IEHNGDAAHXs2FGFhYUO5+/atUs7d+7Upk2bNG/ePK1evVpFRUX2ESbpj9Gm0NBQffLJJwoJCfGo3qoYnCTpr3/9q5588kn7VjR5eXlKTk7WwIEDyz03MzPT8nUOHz6sNm3auF0nAACVic+DkyS98cYb+u2337Rw4UJ7eJL+CDwnT57UF198oS+++ML++OIRKsMw1KJFC1NfJUevio9tNpumTZumyy67zONaS+97d+bMGZ0+fdqld9aVDhrONiH2tbCwMF155ZX6+uuv7W2bN2+2FJxiY2MrsjQAACotn96qKxYSEqK5c+fqzjvvNC2CWXIEqvijtJKfKw5JJUNTSEiIPv74Y91yyy1eqTUmJsZh1fG9e/e61MeePXtMx6XDn78kJiaajl1ZZgAAgOrIL8FJkkJDQ/X2229r1qxZioyMdBqgSoai8j5nGIZatmyplStXei00FWvdurXpeOfOnS6dv2vXrnP25y+l3+3nzm1IAACqE78Fp2I33nij9uzZo3/9618677zznI42lRWUij8SEhL02muvaevWrbr00ku9XmO7du1MxykpKZbPPX36tDZv3nzO/vyl9OKU9evX91MlQGBKnLTA9AGg6vPLHKfSateurYkTJ+q+++7Tjz/+qO+//14//PCDtm7dqqNHjyo3N9f0+KioKLVp00ZdunTRwIED1a1btwp9y3/fvn313nvv2Y9XrFhh+dwffvhBBQUF9uOkpCSHffv8pfSefp7s5wcAQHVQKYJTseDgYHXv3l3du3c3tZ8+fVonTpxQcHCw6tSpY1qDyBeuueYahYeH229lpaSkKDU1VRdeeGG5506fPt10PHjw4Ioo0WVbtmwxrSIuST179vRPMQAABAi/36qzIiIiQvHx8WrQoIHPQ5P0x/50Q4YMMbW9+OKL5Z63Y8cOzZ07135co0YNjRgxwuv1uaqwsFD333+/qa158+YsGwAAQDkCIjhVBk8++aRpTajp06dr3rx5ZT7+7NmzGjt2rGlRzttvv13NmjU753VKT34v77bgG2+8obNnz1p7Evpjvabx48dr+fLlpvYnnnjCch8AAFRXBCeLmjZtqvvuu8/UNmTIEL355pumcCRJv/76q3r16qVVq1bZ22JiYioknNx7771q0qSJ/vGPf2jNmjWm+VQlFRQU6Ouvv1anTp300UcfmT539dVXa+TIkV6vDQCAqsZmOFssCU4VFhbq2muv1cKFC03tcXFx6tChg+rUqaNdu3Zp/fr1pncFhoaGatmyZerWrVu51yg9yT05Ofmcc49KPz4sLExt27ZVo0aNVK9ePeXn5yszM1Pr1q1zurlwx44d9e2336pOnTrl1uaOrKwsxcXFmdoyMzNZRNNLeCdX5ZLxwgB/lwCggvl1cnhubq5SU1P122+/6eDBgzp9+rRycnJks9kUHh6u2rVrKz4+Xueff75atWrl8fYpngoODtaXX36pcePGaebMmfb2zMxMLVq0yOk5cXFx+vjjjy2FJm/Izc3V+vXry32czWbTPffcoxdffFE1a9b0QWUAAAQ+nwYnwzCUnJysefPmacWKFdq2bZvT/eicCQkJUbt27XTVVVfpuuuu0xVXXFHB1TpXu3ZtffHFFxoyZIheeeUVrV692unjoqOjNXToUE2ePLlCR1defvllJScna82aNTpy5Ei5j4+NjdXNN9+su+++29K7AgEAwJ98cqvu7NmzevPNN/XWW2/Ztytx97LFt6aaNWume++9V3fccYdfR6J2796t9evX68CBAzp9+rQaNmyohIQEXX755QoNDfVpLfv27dP27du1b98+HTlyRDk5OQoODlZUVJTq16+v9u3blzs53du4VVexuFVXuXCrDqj6Kjw4zZs3T/fee69+++03h7Dk6qKVzs5v2rSp3nzzTV1zzTUe1wrvIzhVLIJT5UJwAqq+Cn1X3cSJEzV48GDt3bvXtCGvsz3orCh9vmEYSk9P14ABA/TII49UwDMAAAD4U4XMcSoqKtKYMWP06aef2gOTM64Odjnb8Lf4ei+88IKysrJMW6MAAAB4U4UEp3/+85+aMWOG05Gl4rAUFBSkpKQkXXzxxbrwwgsVHx+vyMhIhYeHq6ioSLm5uTp69KgOHDig1NRUbdy4UZs3b7afX7Lf4tGnDz74QI0aNdLkyZMr4mkBAIBqzuvBadGiRXrppZfswaZ4xKk48HTu3FkTJkzQoEGDFBUV5VLfR44c0VdffaX3339fP/30k9Pw9Oyzz6pHjx666qqrvPekAAAA5OU5Tnl5ebrnnntMbcWBpmnTplq4cKFWrVqlMWPGuByapD9W37799tu1evVqLViwQImJiabbfTabTUVFRfrb3/5W5graAAAA7vJqcJo6darS09NNo02GYejaa6/Vpk2bvPrOt379+mnz5s0aOHCgw1ypHTt2aNq0aV67FgAAgOTF4GQYhl599VWHW3TXXnut5s6dq4iICG9dyi4iIkJfffWVKTwVj3BNmTLF69cDAADVm9eC07fffqvdu3eb2hITE/X5558rKKjiVj0ICgrS559/riZNmpja09PTlZycXGHXBQAA1Y/XEs2sWbPs/1882vTGG2+oVq1a3rpEmSIiIvTGG2843LIrWRMAAICnvBacFixYYHqX2yWXXKL+/ft7q/ty9e/fXx07djS9i++bb77x2fUBAEDV55XgtGfPHu3fv1/Sn6NN48eP90bXLhk3bpzp+LffftNvv/3m8zoAAEDV5JXgtGbNGtOxzWbTDTfc4I2uXXLjjTc6LLi5evVqn9cBAACqJq8Ep9TUVNNxu3btFBMT442uXRITE6O//OUvprZff/3V53UAAICqySvBafv27ZL+vE3XoUMHb3Trlg4dOpj2xyuuDQAAwFNe2XJl3759puPWrVt7o1u3lL526doAoKIkTlpgOs54YYCfKgFQUbwy4pSZmWmaWxQfH++Nbt1S8tqGYSgzM9NvtQAAgKrFKyNOR44cMR3HxsZ6o1u3lL526doAwFcYgQKqHq+MOJ09e9Z0HB4e7o1u3VKzZk3TcenaAAAA3OWV4JSbm2s6DgsL80a3bgkNDTUdl64NAADAXV4JTvn5+d7opkIUFBT4uwQAAFBFVNzuuwAAAFUMwQkAAMAighMAAIBFBCcAAACLKiQ4ld5oFwAAoCrwygKYJRmGoc6dO3u7WwAAAL/zenCS/ghPAAAAVU2FBCd/36ojuAEAgIrg1eDk78BUrLLUAQAAqhavBSdGeQAAQFXnleD0xBNPeKMbAOVInLTA3yUAQLVGcAIAALCIBTABAAAsIjgBAABYRHACAACwiOAEAABgEcEJAADAIoITAACARQQnAAAAiwhOAAAAFhGcAAAALCI4AQAAWOS1TX4BAOdWeq/BjBcG+KkSAO5ixAkAAMAighMAAIBFBCcAAACLCE4AAAAWMTkcAPyEyeJA4GHECQAAwCKCEwAAgEUEJwAAAIsITgAAABYRnAAAACwiOAEAAFhEcAIAALCI4AQAAGARwQkAAMAighMAAIBFbLkCVGKlt+QAAPgXI04AAAAWEZwAAAAsIjgBAABYRHACAACwiMnhAFBJlH4zQMYLA/xUCYCyMOIEAABgEcEJAADAIoITAACARQQnAAAAiwhOAAAAFhGcAAAALCI4AQAAWMQ6TgBQSZVc14k1nYDKgREnAAAAiwhOAAAAFhGcAAAALCI4AQAAWERwAgAAsIjgBAAAYBHBCQAAwCKCEwAAgEUsgAn4UckFDiUWOQSAyo4RJwAAAIsITgAAABZxqw6oRErfugMAVC6MOAEAAFhEcAIAALCI4AQAAGARwQkAAMAighMAAIBFvKsOAAIAi6UClQMjTgAAABYRnAAAACziVh3gQyxwCQCBjeAEAAGIOU+Af3CrDgAAwCKCEwAAgEUEJwAAAIsITgAAABYRnAAAACwiOAEAAFhEcAIAALCI4AQAAGARwQkAAMAighMAAIBFbLkCAFUAW7AAvsGIEwAAgEUEJwAAAIu4VQdUoNK3TwBf4dYdUDEYcQIAALCI4AQAAGARt+oAoBrg1h3gHYw4AQAAWERwAgAAsIjgBAAAYBHBCQAAwCKCEwAAgEW8qw7wMha9BICqixEnAAAAiwhOAAAAFhGcAAAALCI4AQAAWERwAgAAsIjgBAAAYBHBCQAAwCLWcQIAmNYfy3hhgB8rASo3ghMAVEMs1Aq4h1t1AAAAFhGcAAAALCI4AQAAWERwAgAAsIjgBAAAYBHvqgMAuKT0O/JYvgDVCcEJAGBCMALKxq06AAAAixhx8tDu3bu1ceNGHThwQNnZ2WrUqJESEhLUtWtXhYSE+LW29evXKy0tTfv375ckNW7cWC1btlRSUpJf6wIAIFARnNw0e/ZsTZkyRSkpKU4/Hx0draFDh+qpp55S/fr1fVZXfn6+XnnlFU2bNk3p6elOH9O8eXONGzdOEydO9Hu4A1D5ubrKOLf6UJVxq85F2dnZGj58uG666aYyQ5MkHT16VO+8847atWunxYsX+6S2tLQ0de7cWQ8//HCZoUmSdu7cqUmTJqlLly7auXOnT2oDAKAqsBmGYfi7iEBRWFio6667Tt98842pPTY2VklJSapXr57S09O1YcMGlfyyhoWFadmyZbriiisqrLZDhw6pc+fO2rNnj6m9efPmatu2rQzD0C+//OIQqJo0aaLVq1crLi6uQurKyspy6DszM1OxsbEVcj1fKO+vafYAA8wYcUJVwq06F0yaNMkUmkJCQjRlyhRNmDBBoaGh9vZt27Zp3Lhx9hGp3NxcXX/99dqyZYsaNWrk9bqKiop0/fXXm0JTo0aNNH36dPXp08f02EWLFmns2LE6dOiQpD/maA0ePFgrV66UzWbzem1VQXlBiKAEANUHt+os2rVrl1577TVT26xZs3T33XebQpMktWnTRsuXL1eXLl3sbUeOHNHkyZMrpLZPP/1Ua9assR9HR0dr1apVDqFJkvr27atVq1YpKirK3rZq1SrNnDmzQmoDAKAqIThZNHnyZOXn59uPx4wZo0GDBpX5+PDwcE2fPt0Uqj744APt2rXLq3UVFhbqiSeeMLVNmTJFiYmJZZ7TpEkTTZkyxdT26KOPqqioyKu1AQBQ1TDHyYKcnBzVr19fZ86csbf9+uuvuvDCC8s9d+jQofryyy/tx08//bQeffRRr9X23XffqWfPnvbjxo0ba8+ePQoODj7neYWFhUpISLAvVSBJP/zwg9fnYVWFOU7cigO8y5tznngHH3yNEScLFi9ebApNXbp0sRSaJGns2LGm4zlz5ni1trlz55qOR40aVW5okqTg4GDdcsstFVobAABVDZPDLVi0aJHpuOQIT3m6deumGjVqqKCgQJK0YcMG/f7772rQoIHfa+vZs6defPFF+/HChQsdbuFVR4wwAb51ru85RpBQ2RCcLNi6davpuOSk7/JERETooosu0oYNG+xtv/zyi1eCU25ursM6TJ07d7Z8fteuXU3HaWlpysvLc5jsDgCVhad/2JQ8n1AGd3CrzoJff/3VdNy8eXOXzm/WrJnpeNu2bR7XJEnbt29XYWGh/TguLk5169a1fH7dunVNq5oXFhZqx44dXqkNAICqiBGnchw9elRHjx41tV1wwQUu9VH68WlpaR7XJclhtMnVuorPOXz4sP04LS1N7dq187g2ACgLt8MRyAhO5Th+/LjpuFatWoqIiHCpj9LvKjtx4oSnZUlyrM2d1b/dqS0rK8ty/5mZmQ5tJYOaL3R4euk5P7/+sd6m48Iz3nl9AHju/Hs/c+nx5f18Kvn9XV7fpX82oHqIiYlRUFDZN+QITuXIzs42HYeHh7vcR+lzTp065VFNxfxVm6fbs7Rp08aj870t7g1/VwDAW7z5/czPhuqpvCVzmONUjtLhpGbNmi73UTqclO7TXZW5NgAAqiKCk4vc2c/NV3vAVebaAACoCghO5ahdu7bpOCcnx+U+Sp9Tuk93VebaAACoipjjVI7KHE78VZuzCd9lKSgosO/PFx0dbf/vuSbeIbAdPnzYYR7btm3bTEtfAHAP318VLyYm5pyfJziVo169eqbjM2fO6PTp0y69s6500IiMjPRGaQ61ufJut2Lu1ObqPnONGjVy6fGoeurXrx9Q+xMCgYTvL9/iz/5yxMTEKCoqytS2d+9el/rYs2eP6bhFixYe1+Wsn9LXsaKiagMAoCoiOFnQunVr03HphSfLU3yrqqz+3NWqVSvThr6ZmZkuLXVw8uRJ05pKwcHBBCcAAM6B4GRB6ZW0U1JSLJ97+vRpbd68+Zz9uSssLMxhOxdXalu1apXpuEWLFgoLC/NKbQAAVEUEJwv69u1rOl6xYoXlc3/44QcVFBTYj5OSkryywW8xT2or/dh+/fp5oSIAAKougpMF11xzjWmhyJSUFKWmplo6d/r06abjwYMHe7M0h/4++eQT08a/ZSksLNSMGTMqtDYAAKoagpMFtWrV0pAhQ0xtL774Yrnn7dixQ3PnzrUf16hRQyNGjPBqbd26dVOTJk3sx/v27XMIRM7MmDFD+/fvtx83a9ZMl19+uVdrAwCgqiE4WfTkk08qJCTEfjx9+nTNmzevzMefPXtWY8eOVV5enr3t9ttvd5iTVJrNZjN9lHfrLTg4WJMnTza1TZw4URkZGWWek5GRofvvv9/U9swzz7C2EgAA5eA3pUVNmzbVfffdZ2obMmSI3nzzTVM4kqRff/1VvXr1Mk2+jomJ0RNPPFEhtY0cOVKdOnWyHx89elRdu3bVkiVLHB67ePFidenSRceOHbO3de3aVUOHDq2Q2gAAqEpshmEY/i4iUBQWFuraa6/VwoULTe1xcXHq0KGD6tSpo127dmn9+vUq+WUNDQ3VsmXL1K1bt3KvUXrvuOTkZPXs2bPc8w4ePKjOnTs7rDHVokULtW3bVoZh6JdffnFYSiExMVGrV6/26oR1VG9ZWVmKi4sztZW32zgAa/j+8j9WDndBcHCwvvzyS40bN04zZ860t2dmZmrRokVOz4mLi9PHH39sKTR5olGjRlq6dKmGDRumDRs22NvT0tKUlpbm9JwOHTpo5syZhCYAACziVp2LateurS+++EKzZs1S586dy3xcdHS07rrrLm3dutVhyYCK0rJlS61Zs0bPP/+8mjZtWubjmjVrpueff16rV69W8+bNfVIbqo/Y2FgZhmH64K9hwDv4/vI/btV5aPfu3Vq/fr0OHDig06dPq2HDhkpISNDll1+u0NBQv9a2bt067dixQwcOHJAkxcfHq2XLlrrkkkv8WhcAAIGK4AQAAGARt+oAAAAsYnI4AI8VFhZq586d2rZtmw4cOKATJ04oLCxMUVFRatasmTp27KiIiAh/lwkAHiM4AXDL3r17NWfOHC1btkw//PCDTp48WeZjg4OD1bt3b919990aMGCAD6sEqrZhw4aZ3uUtSQkJCedcBBmeYY4TAJeNGDFCn3/+uVvnDhw4UNOmTWMZDMBD8+bN06BBgxzaCU4Vi+AEwGUdO3bUunXrHNobN26sFi1aqEGDBiooKNCuXbu0adMmFRUVmR7XsmVLfffdd2rYsKGvSgaqlOPHj6tt27b2d02XRHCqWNyqA+CRpKQk3XbbberXr5/TvRj379+vp556Su+99569bceOHbrpppv0/fffO6yWD6B8//d//2cPTXXq1NGpU6f8XFH1wbvqALjMZrNpwIABWrt2rdavX6+77767zA2sGzdurKlTp+qtt94yta9cudJhbgaA8i1btkwffvihJKlGjRp66qmn/FxR9UJwAuCyWbNmaf78+erYsaPlc/7617/qxhtvNLV98skn3i4NqNJOnz6t8ePH248nTpyo9u3b+6+gaojgBMBliYmJbp33t7/9zXScnJzshWqA6uPhhx+2z19q2rSpnnzySb/WUx0RnAD4TFJSkuk4JydHx48f908xQIBZtWqV6Zb31KlTFR4e7seKqieCEwCfqVHD8f0oeXl5fqgECCy5ubm67bbb7O9QHT16tK6++mo/V1U9EZwA+MzOnTtNxzVq1FD9+vX9VA0QOJ588klt375dkhQbG6tXXnnFzxVVXwQnAD4ze/Zs03HHjh0VFMSPIeBc1q9fr3/961/241dffVUxMTF+rKh64ycWAJ/Izs7WBx98YGobPHiwn6oBAkNBQYFuu+02FRQUSJL69u2rESNG+Lmq6o3gBMAnHn74YR06dMh+HBkZqXHjxvmxIqDye+GFF7Rp0yZJUkREhN555x0/VwSCE4AKN3fuXL355pumtmeffVbR0dF+qgio/LZt26ZnnnnGfvz000+7vRQIvIfgBKBCbdq0SaNGjTK19enTR3fddZefKgIqv6KiIt1+++3Kzc2VJF1yySW69957/VwVJIITgAq0d+9eDRgwQNnZ2fa2hIQEzZgxgz3qgHN47bXXtHr1akl/vPt02rRpCg4O9nNVkAhOACpIZmamevfurf3799vbGjZsqKVLlyo2NtaPlQGV265du/Too4/aj9lWpXIhOAHwuqNHj+rqq6/Wjh077G3169fXsmXL1KJFCz9WBlRuhmFo/PjxOnPmjCS2VamMCE4AvOrEiRPq06ePtmzZYm+LiorS0qVL1bZtWz9WBlR+77//vr799lv7MduqVD6O+x8AgJtOnTqlvn37at26dfa2unXratGiRdxqACx44okn7P/fv39/NW/e3L6pb1lKLvMh/bH2U+lz4uPjFRoa6q0yqzWbYRiGv4sAEPhOnz6tvn37auXKlfa22rVra/HixeratasfKwMCR2RkpE6cOOH1fjds2MAfL17CrToAHsvJydHAgQNNoalWrVpasGABoQlAlUJwAuCRs2fP6rrrrtOKFSvsbTVr1tS8efPUvXt3/xUGABWA4ATAbXl5ebrhhhu0bNkye1tYWJi++uor9erVy4+VAYHp+PHjMgzDpY/k5GRTHwkJCQ6P4Tad9xCcALiloKBAN998sxYuXGhvCwkJ0ezZs3XNNdf4sTIAqDgEJwAuKyws1MiRI/X111/b22rUqKGZM2dq4MCBfqwMACoWyxEAcNltt92mL7/80tT23HPPKSkpqdy3TpfWsGFD1axZ04vVAUDFYTkCAC7z5j5zycnJ6tmzp9f6A6qbFStW6Morr7QfJyQkuPwHDKzjVh0AAIBFBCcAAACLuFUHAABgESNOAAAAFhGcAAAALCI4AQAAWERwAgAAsIjgBAAAYBHBCQAAwCKCEwAAgEUEJwAAAIsITgAAABYRnAAAACwiOAEAAFhEcAIAALCI4AQAAGARwQkAAMAighOASikxMVE2m830kZGRYencMWPGOJw7ffr0Cq0XQPVQw98FAL6UlZWlHTt2aO/evcrKylJOTo4KCgpUt25d1atXT/Xq1VN8fLzatWun8PBwf5cLAKhkCE6o0nJycrRw4UJ9/fXX+v777y2PWAQHB6tVq1ZKSkpSnz59dN111ykyMrJCawUAVH4EJ1RJWVlZeuWVV/T+++/r6NGjLp9fWFiobdu2adu2bfr0008VEhKiq6++WqNHj9ZNN92koCDucqN6+emnnxy+ly677DJFR0f7qSLAP/jpjyqlqKhI//73v9WiRQu9+OKLboUmZ/Lz87Vw4UINGzZMbdq00YwZM1RYWOiVvoFA8OCDD6pfv36mj82bN/u7LMDnGHFClZGZmanhw4fr22+/LfextWrV0gUXXKCYmBiFh4crPz9f2dnZ9rlP57J9+3bdeuutOnz4sP7+9797qXoAQCAgOKFK2L17t/r06aOdO3c6/XyNGjXUv39/DRo0SFdccYVatGghm83m9LEnT57U6tWrtWLFCs2ZM0fbt293+riCggKv1Q/vmj59Ou+iA1AhuFWHgPf777/rqquuchqagoKCNH78eKWnp+vrr7/WbbfdppYtW5YZmiSpbt266tOnj5577jmlpqYqJSVFw4cPZ14TAIDghMCWn5+v6667zum75Ro1aqTk5GS99957uuCCC9y+RufOnfXZZ59py5Yt6t+/vwfVAgACHcEJAe3JJ5/UTz/95NCekJCgH3/8Ud27d/fatdq0aaMFCxbos88+Y2kCAKimmOOEgLV9+3a99NJLDu0RERFasGCBmjRpUiHXHT58uDp16qQdO3ZUSP8AgMqL4ISA9eijjzqdoP3iiy+qbdu2FXrtpk2bqmnTphXS95EjR7Rhwwalp6fr+PHjys/PV506ddSuXTv16tXLcj+HDx9Wamqqdu3apePHj+vUqVOqWbOmoqOjFR0drYsuuqjCnkNpaWlp2rBhg3777Tfl5OSodu3aaty4sf7yl7+oVatWPqnBm86cOWN/jYpXoI+MjFSDBg103nnn6dJLL1WNGr798Xro0CGtW7dOu3bt0smTJ1W7dm3Vr19fLVq0UMeOHav9HL3K+JodPHhQmzZtsr9mhYWFqlevni699FJ16tTJ7X4zMjK0ZcsWZWRk6NSpU5KkqKgode/e3aWfjdnZ2Vq3bp3S09N15MgR5ebmqmbNmoqLi1OLFi2UlJSkmjVrul2nOwoKCrRx40alpqYqMzNTZ86cUVhYmGJjY3XzzTerVq1aFV+EAQSgPXv2GEFBQYYk00fr1q2NwsJCf5fnoHSdpb/18vPzjenTpxtdunRx+rwkGT169DjnNQ4dOmRMmzbNGDlypBEfH++0j9If8fHxxq233mqsX7/e68/57Nmzxquvvmq0atXqnDU0adLEeO6554wTJ06Yzk9ISHB47O7duy1de/To0Q7nfvTRRx49n5ycHOP99983evbsadSoUeOcz6levXrGTTfdZCxdutTt6+3evduh34SEBNNjCgsLjc8++8zo3LmzYbPZyqwnOjramDBhgrFv3z7L1//oo48s/Ruy8jF69Gi3vw6eqIyv2enTp43XX3/d+Mtf/mL565WcnFzuz4MjR44YzzzzjNG8efMy+33iiSfKfQ65ubnGhx9+aPTo0cMIDg4+59csNDTUGDBggDFnzhyjqKjI7a+ble/Xn3/+2bjllluMunXrllmP1Z8PniI4ISA9/vjjTr9xPvzwQ3+X5tS5gtP69euNdu3alfvLp6zgtGLFCqNXr17l/pAr76N///7G/v37vfJ8V69ebTRr1syl65933nnGsmXL7H1UluBUVFRkfPjhh0aDBg3c/rru2LHD5euW90s4LS3N6Ny5s0u1hIeHG++9956l6wdycKqsr9myZcuc/rsu7+tVXnD6/PPPjfr165fbb3nBaf78+Zbqc/Zx8cUXG2vWrHH5a2YY5/5+PX36tDFu3Lhz/mHg6s8HT1XvsVsErLlz5zq01apVSzfddJMfqnHf4sWLdcUVV2jr1q1u97F06VItX77c45XMv/nmG3Xs2FGrV6/2qJ85c+aoR48eSk9Pd+m8ffv2qV+/fpo5c6ZH1/emkydPavDgwbrtttv0+++/u9XHN998o06dOunHH3/0Wl2rVq1Sp06dXH6tcnJyNGHCBL3yyiteq6Wyqayv2UcffaRrrrlGe/bs8VqfkvT0009r+PDhOnz4sNt9GIahhx9+WAMHDnS7vk2bNumKK67Qu+++63YdpR0/flzdu3fXtGnTZBiG1/r1FHOcEHD279+vLVu2OLT36tVLtWvX9kNF7tm0aZNuvPFGnTlzxtReu3ZtJSQkKDY2VidPntT+/fvd/gUQFxenBg0aqG7duqpZs6a9vwMHDjh9/MGDB9WvXz9t2LBBiYmJLl9v+fLlGjZsmPLz851+Pjw8XE2aNFFcXJyOHTumvXv36tixY/bP5+fna9SoUUpISHD52t527Ngx9enTRz///HOZj4mJidH555+v6OhonTp1Snv37nX6Wh07dky9e/fW/PnzddVVV3lU19atW9W/f3+dOHHC1B4VFaULLrjAXsuuXbvK3HLowQcf1OWXX67OnTt7VEtlU1lfs0WLFmn8+PEOf9xERUXp/PPPV0xMjI4ePap9+/bpyJEjlvudNm2aHn/8cYf22NhYNW7cWFFRUcrMzNS+ffsc/r2UdN999+mNN94o8/P16tVTQkKCYmJilJmZqYyMDJ0+fdrhcfn5+brrrruUl5ene++91/LzcKawsFCDBg3SunXrTO1BQUH2nyHBwcHav3+/9u3bV+bPnArhk3EtwIvmzJnjdJj2lVde8XdpZXJWb9u2bU3HgwYNMr799lsjLy/P4fxdu3YZM2bMcNr3I488Yu+jdevWxqRJk4xly5YZmZmZZdbz+++/G1OnTi1znkWnTp2M/Px8l57jsWPHjIYNGzrtr3Xr1sYXX3xhnD592nROYWGhsWzZMqNv376mx7do0cJpX766VVdUVGQMGDDA6XOJjIw0HnnkEWPLli1O53Vs2rTJmDBhgtNbpw0bNjSysrIs1eDstk+DBg1Mc8aCg4ON22+/3VizZo3D3L7CwkJjxYoVRteuXZ0+jzZt2pxzXkpmZqaRkpJi/0hKSnLo46233jI9pqyPnTt3Wv7au6uyvmYxMTGmf8tBQUHG2LFjjZSUFKfzMbdu3WrMmTPH1ObsVl2TJk2MWrVq2Y9DQ0ONiRMnGhs3bnT6tVm9erWxZMkSh899/vnnZd76uvrqq41FixY5/CzIyckx/vvf/zr9N1H87zIlJcXS18wwnH+/lv752KRJE+ODDz5w+nMtOzvb+M9//mP5dfIUwQkB57HHHnP6zbp8+XJ/l1amsn4wSX/MO5k3b57bfT/11FPGqFGjnP7ALE9BQYHx3HPPOZ0/MHPmTJf6Gjt2rNPnN378eOPs2bPlnv/OO++UO4/BV8Hp5Zdfdnr9AQMGGEeOHLHUx48//mjExMQ49HH99ddbOt/ZL+GSH40bNzbWrVtXbj8FBQXGyJEjPf6e6dGjh8P5ycnJls+vaIHwmtWvX99YvXq1y8/NWXAq+dGsWTNj+/btLve7d+9eIzIy0qE/m81mvPnmm+Wen5+fbzzwwANl1nTq1ClLdTj7fi35MW7cOEs/Q3yF4ISAU9YvAW9NbK4IZf1ACAoK8jjwefJulmL/+te/HGq7/PLLLZ+/adMmp89v+PDhLtX3+uuv+z047dq1ywgJCfH4uRiGYWzcuNGoWbOmQ19WAs+5fgnHxMQYGRkZluvIyclxOln/lltusdxHZQ5OgfCaRUREGL/88otbz+9cwSk+Pt44cOCAW/2OGTPGaZ9vvfWWS/3cc889Tvt5+umnLZ1/ruA0duxYd55ahSI4IeA4+wEeFBRkFBQU+Lu0MpX1Q2HixIn+Ls0wjD9u63To0MHtoHLXXXc5nNugQQOHJQas6N27t1+Dk7NfAhdddJHbf/H++9//dujv1ltvLfe8c/0S/uqrr1yu47333nPop/Rb5c+lMgenQHjNXn/9dbdqMYxzByd3R6uzsrKcBsRrr73W5b7y8/ONNm3aOA11zqYelFZWcLrgggssj1r5EsEJAcfZvJzo6Gh/l3VOzn4ohIWFGYcPH/Z3aXavvvqqQ42fffZZuedlZ2c7XVvlnXfecauOLVu2+C04HT161IiIiHA4d9GiRW49F8P4Y7QnLi7O1F9oaKiRnZ19zvPK+iXcsWNHt+o4duyY01uhVv8NVtbgFAivWcOGDV2eM1hSWcHJ3X8LhmEYzz//vEN/wcHBRnp6ulv9LV682GmNVm75lxWcrNwu9AeWI0DAOXv2rEObp6vXNm/eXDabzeWPJ5980u1rDho0SDExMR7V7U2XXXaZQ5uVt7v/+OOPOnnypKmtZs2aGjFihFt1tGvXzqNVkz0xb948h3cLtWjRQtdcc43bfdasWVNDhgwxteXl5TndY9GKcePGuXVeZGSkmjVr5tAe6FsHBcJrNmrUqApZlfy2225z+9zFixc7tPXq1cvt3QT69Onj9Fxn17EiLCxMI0eOdOvcikZwQsBxts2KzWbzQyWeufLKK/1dgklcXJxDm5W1mNasWePQ1qdPH9WtW9ftWm6++Wa3z/XE999/79B24403etxvt27dHNrcXSOoR48ebtfRvHlzh7ZzvU09EATCa1ZR3+vu9ltYWKi1a9c6tA8fPtyjepz9sZSSkuJWX+3bt6+0m6mzjhMCjrPRpdIjHoHgkksuqbC+N23apOTkZG3ZskVbt25VZmamTp06pZMnT7q03snx48fLfYyzv8I7duzoSrkOKvJrcy4//PCDQ5unz0WS0zWxNm/e7HI/wcHBHu3t5yzMBnpwquyvmVQx/55r166tli1bunXuli1bnK7D5Om6Xs7OT01N1bFjxxQVFeVSX/76GWAFwQkBx9kmjtnZ2SoqKgqojUzPO+88r/aXn5+vt99+W++//75++eUXr/RpJThlZGQ4tF188cUeXdfT892Rm5urtLQ0h/YzZ854vJr63r17HdrKWpzyXKKiojwaXQ0PD3doc3brO1AEwmsWGhqq2NhYj2pxJj4+3u2fd86+Z2vVquV2ECuWlJTk0GYYhn777TeXg5O3fz56E8EJAadhw4YObYZh6MSJEy5/cxabPXt2ub9ABg8erEOHDrnVvzP16tXzWl9r1qzRbbfdpm3btnmtT0lO/yotreTK38Wc3fZzRWRkpEJDQ5WXl+dRP64o65fiqFGjfHq9c4mIiPB6HUYl2srCVYHwmnnz+9xb/Tr7nm3QoIHHf3g2atRINpvN4d+Us+uVp6K+bt5AcELAOf/8852279271+3g1L59+3IfExYW5lbfZXE2cuaO7777TgMGDLAUclxl5Zeqs1EpT+Y3lezDk/23XOXKVhfeEOi3yCqDQHjNvPV97s1+nQUZb3zP2mw21alTx2HqhDvBqaK+bt5AcELAadOmjdP2n376yS+3ePxp//79uvbaa8sMTYmJibr88svVqlUrnXfeeYqLi1NYWJjCw8MVHBxseuzBgwd1ww03uFxDbm6uQ1toaKjL/ZTm7aBaHl8HGU83ZQavmbtycnIc2pzdxnVHeHi4Q3CqiD/q/InghIBT1qTBn376SePHj/dxNf71j3/8Q6dOnXJov/766/XYY4+pQ4cOlvvauXOnWzXUq1fP4S9/ZzW5ytcT/ksHSVR+vGbucTa65I3v2bL6qcy33dxBcELAueSSS1S7dm1lZ2eb2p29u6Yqy8rK0syZMx3aH3roIb3wwgsu9+fOcLr0x3yk0sHJ05GAoqIih9e3otWpU8ehLSgoSGfOnPH56Bes4TVzj7MpDd74Q6WgoEBnzpxxaI+Ojva478okcN6CBPx/oaGh6tOnj0P79u3b3V5nJRDNnz9fRUVFprY2bdro2Wefdas/d+eL1K9f36HN2TudXJGWlubzScvO5s4VFRVp9+7dPq0D1vGaucdZcDpw4IDH77Asa903d+eeVlYEJwSkslalfuedd3xcif+sW7fOoW3EiBFu375w1p8VzuaVbdy40a2+vHW+O+rWravGjRs7tG/atMnntcAaXjP3OJsnWlhYqC1btnjU74YNGxzawsLC3F6NvLIiOCEgDRo0SPHx8Q7ts2bN0vbt2/1Qke/9/vvvDm2tW7d2u7+VK1e6dZ6z7VGWL1/udh3eON9dXbp0cWibN2+eHyqpfCrr6vy8Zq5r0qSJGjRo4ND+3XffedRvcnKyQ1uHDh2q3G1TghMCUo0aNfTPf/7ToT0vL0+33HKLS6tjBypn84hq167tVl/79u3TsmXL3Dq3a9euDm3bt2/Xzz//7FZ/ubm5mjVrllvnemrgwIEObfPnz3d7/ldV4uyXX2X4PuM1c4+z79sZM2a43V9ubq6+/PJLh3ZnwTbQEZwQsCZMmOB0hOXnn3/WI4884oeKfMvZO1UOHDjgVl9TpkxxugegFRdeeKHTLS5efPFFt/p79913La1YXhEGDx7sED5Pnjyp5557zi/1VCbOJmJXhreZ85q5p/QmxtKfWzW548MPP3T6fXvTTTe51V9lRnBCwAoJCdF//vMfp7uOv/zyy3rssccCelXk8ji7Vblw4UKX+0lJSdHrr7/uUS0TJkxwaJs9e7aWLFniUj8HDx7UE0884VEtnqhbt67TJS1ee+01t0fkqgpnE3ydbd3ha7xm7rnpppuc7sJw7733uvxH1OHDh/Xoo486tHfs2NHj/e8qI4ITAlrHjh316quvOv3cM888o/79+3vlh/vBgwcrxV/XJTnbvf2///2v1q9fb7mPX3/9VUOGDPF4Yb8RI0Y4naQ7bNgwyxNOjx8/rr59+/p9Re1HH31UMTExprb8/HwNGTJEK1as8Lj/tLQ0/ec///G4H19zNqG4sryLldfMdSEhIfrb3/7m0L5161aNHTvW8h+dZ86c0aBBg5xuRzNx4kSP66yMCE4IeH/729+czneSpEWLFql169a68847nb7jozzr16/Xfffdp6ZNm/p0+w8revfu7bB3WWFhoQYOHGjpHXKzZ89W9+7d7bf3PFlMMCIiQm+//bZD+7Fjx9SjRw998skn5zw/JSVFXbp0se8+HxQUpJo1a7pdjyeio6P1/vvvO7SfOHFCvXr10sMPP+zynmWnTp3SrFmzNHDgQLVq1UqfffaZt8r1GWcLz86bN8+t7ytv4zVzzz/+8Q9ddNFFDu0zZszQ0KFDy/2Zl56ert69e2vVqlUOn+vfv7+GDx/utVorExbARJXw7LPPKjIyUg899JDDX0pnz57V1KlTNXXqVCUkJOiKK67QxRdfrISEBMXExKhmzZrKz89Xdna2Tp48qZ07dyo1NVU//vij0x3SS/Ln+iT16tXTPffc47DY5cGDB9WlSxeNHDlSw4YNU1JSkqKiopSdna0DBw7o22+/1eeff66UlBTTef/85z/19NNPu13PddddpxEjRjj8gjl27JhGjRql559/XjfffLPatWunuLg4HTt2TLt27dLcuXO1cuVK0+t27733au7cudqzZ4/b9Xhi8ODBevrpp/XYY4+Z2ouKivTCCy/o9ddf15AhQ9SzZ09deumliouLU1RUlPLz83XixAkdPXpUqamp2rhxo9auXasVK1Y43ZomkHTt2lXx8fGmeXR5eXm6/PLLNWLECHXr1k3nnXeeatWq5fAOvNjYWDVr1qxC6+M1c11YWJg+/fRTXXrppQ7PddasWUpOTtbo0aN1ww03qGnTpoqOjlZWVpZSU1M1a9YsffLJJ04XvIyNjdWHH37oq6fhewZQhSxdutRo1KiRIalCPxITE41Zs2ZZrstZH95w8uRJo02bNh4/n/Hjxxu7d+92aE9ISHCpnjNnzhhdu3b1qJZu3boZubm5RkJCgsPndu/ebamO0aNHO5z70Ucfufz1ffbZZyvk388111xT7rW98XqU5unXxd2vx+jRoz2q2xVV7TUzDMNITk526LdHjx4e91ts9uzZRo0aNbzydapTp46RkpLi0vW99f3qK9yqQ5Vy9dVXKzU1VQ8++GCF3Opp1aqV3n33Xe3YscPpu1J8rU6dOvrf//6nhIQEt/uYMGGC1xYODQ8P15IlS3Tttde6df5VV12l//3vf17ZJNgb/vnPf2r+/PlO17zxREhIiFf785UHH3xQV1xxhb/LOCdeM9fdeOON+uabbxzmibmqSZMm+uGHH6rkhPCSCE6ocurWrasXX3xR+/bt0/PPP6+2bdt61F+DBg00YcIEJScnKzU1VXfccUel+iHatGlTrVu3Ttddd51L58XFxemTTz7R1KlTvbpZakREhObNm6epU6da/uUVGRmpF198UUuWLKl0G4IOGDBAO3bs0COPPOLRnlshISHq16+fvvjiC7+tU+WpGjVqaOnSpbrnnnsqTbh1htfMdb1799Yvv/yiMWPGOH2n8rnUqlVLDzzwgDZv3ux0J4GqxmYYVfj92sD/l56eru+//15r167V9u3btXfvXh0+fFg5OTnKz89XaGioIiIi1KBBAzVu3FgXXnihLrroInXt2lVt2rSptKsml7Z27Vq98847Wr58udP5WVFRUbriiis0ePBgDRs2TOHh4fbPnTp1Sh9//LHp8XXr1tWoUaPcric3N1f//e9/NX/+fK1fv1779u3T2bNnVatWLZ133nm66KKL1LdvXw0ZMsTpOkGVTU5OjhYsWKAFCxYoJSVFaWlpDvsFFouPj9eFF16oDh066KqrrlL37t0dJvMHsiNHjujLL7/U6tWrtXnzZh06dEinTp3SmTNnHOYZjh49WtOnT/dLnbxmrtu7d68+/fRTffPNN1q3bp1ycnIcHlOnTh117txZAwcO1IgRI5zuWVlVEZyAKurIkSM6fPiwTp06pfDwcMXGxiouLs7fZVUp+fn52r9/v06cOKG8vDzVqlVLderUUUxMTLX8hRsIeM1cYxiG9u/fryNHjigvL09hYWFq0KCB12+FBhKCEwAAgEXMcQIAALCI4AQAAGARwQkAAMAighMAAIBFBCcAAACLCE4AAAAWEZwAAAAsIjgBAABYRHACAACwiOAEAABgEcEJAADAIoITAACARQQnAAAAiwhOAAAAFhGcAAAALCI4AQAAWERwAgAAsIjgBAAAYBHBCQAAwKL/B1+MUP8natiQAAAAAElFTkSuQmCC",
126 |       "text/plain": [
127 |        "<Figure size 600x600 with 1 Axes>"
128 |       ]
129 |      },
130 |      "metadata": {},
131 |      "output_type": "display_data"
132 |     }
133 |    ],
134 |    "source": [
135 |     "# Plot stochastic gradient noise\n",
136 |     "\n",
137 |     "import torch\n",
138 |     "%matplotlib inline\n",
139 |     "from matplotlib import pyplot as plt\n",
140 |     "import numpy as np\n",
141 |     "\n",
142 |     "fig_dir = 'figures' \n",
143 |     "os.makedirs(fig_dir, exist_ok=True)\n",
144 |     "\n",
145 |     "# hyperparamters\n",
146 |     "mode = 'normal'\n",
147 |     "N = 20\n",
148 |     "seed = 1\n",
149 |     "n_layer = 3\n",
150 |     "\n",
151 |     "filename = log_dir + '/stochastic_gradient_noise_layer{}_N{}_{}_sd{}.pth'.format(n_layer,N,mode,seed)\n",
152 |     "loaded_dict = torch.load(filename)\n",
153 |     "noiseList = loaded_dict['noiseList']\n",
154 |     "noiseArray = np.array(noiseList)\n",
155 |     "\n",
156 |     "fig, ax = plt.subplots(1, 1,figsize = (6, 6))\n",
157 |     "ax.hist(noiseList, bins=100, density=True, alpha=1.0, edgecolor = 'black', linewidth = 0.001)\n",
158 |     "ax.tick_params(axis='both', which='major', labelsize=30, width = 3, length = 10)\n",
159 |     "ax.tick_params(axis='both', which='minor', labelsize=30, width = 3, length = 5)\n",
160 |     "\n",
161 |     "ax.spines[['right', 'top']].set_visible(False)\n",
162 |     "ax.spines['left'].set_linewidth(3)\n",
163 |     "ax.spines['bottom'].set_linewidth(3)\n",
164 |     "ax.set_ylabel('Density',fontsize=40)\n",
165 |     "ax.set_xlabel('Gradient error',fontsize=40)\n",
166 |     "\n",
167 |     "plt.tight_layout()\n",
168 |     "plt.savefig(fig_dir + '/heavy_tail_noise_layer{}_N{}_{}.pdf'.format(n_layer, N, mode), dpi=600)"
169 |    ]
170 |   }
171 |  ],
172 |  "metadata": {
173 |   "kernelspec": {
174 |    "display_name": "pytorch",
175 |    "language": "python",
176 |    "name": "python3"
177 |   },
178 |   "language_info": {
179 |    "codemirror_mode": {
180 |     "name": "ipython",
181 |     "version": 3
182 |    },
183 |    "file_extension": ".py",
184 |    "mimetype": "text/x-python",
185 |    "name": "python",
186 |    "nbconvert_exporter": "python",
187 |    "pygments_lexer": "ipython3",
188 |    "version": "3.11.3"
189 |   },
190 |   "orig_nbformat": 4
191 |  },
192 |  "nbformat": 4,
193 |  "nbformat_minor": 2
194 | }
195 | 


--------------------------------------------------------------------------------
/plot_loss.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "code",
  5 |    "execution_count": 1,
  6 |    "metadata": {},
  7 |    "outputs": [],
  8 |    "source": [
  9 |     "import torch\n",
 10 |     "%matplotlib inline\n",
 11 |     "from matplotlib import pyplot as plt\n",
 12 |     "import math\n",
 13 |     "import torch.nn.functional as F\n",
 14 |     "from torch.nn.functional import relu\n",
 15 |     "from torch import nn\n",
 16 |     "import torch.optim as optim\n",
 17 |     "import torch.optim.lr_scheduler as lr_scheduler\n",
 18 |     "import random\n",
 19 |     "import numpy as np\n",
 20 |     "import gc\n",
 21 |     "from pylab import *\n",
 22 |     "import os\n",
 23 |     "import random\n",
 24 |     "import json\n",
 25 |     "import pandas as pd\n",
 26 |     "from scipy.stats import norm\n",
 27 |     "pd.set_option('display.float_format', lambda x: '%.5f' % x)\n",
 28 |     "import sys\n",
 29 |     "import matplotlib.pyplot as plt\n",
 30 |     "import time\n",
 31 |     "\n",
 32 |     "from linear_transformer import Transformer_F, attention, generate_data, in_context_loss, generate_data_inplace\n",
 33 |     "\n",
 34 |     "np.set_printoptions(precision = 4, suppress = True)\n",
 35 |     "torch.set_printoptions(precision=2)\n",
 36 |     "device = torch.device(\"cuda\")\n",
 37 |     "torch.cuda.set_device(0)"
 38 |    ]
 39 |   },
 40 |   {
 41 |    "cell_type": "code",
 42 |    "execution_count": 2,
 43 |    "metadata": {},
 44 |    "outputs": [],
 45 |    "source": [
 46 |     "# Set Hyperparameters\n",
 47 |     "\n",
 48 |     "# Fixed\n",
 49 |     "n_head = 1\n",
 50 |     "d = 5\n",
 51 |     "B = 1000\n",
 52 |     "ma = 1\n",
 53 |     "var = 0.05\n",
 54 |     "shape_k = 0.1\n",
 55 |     "\n",
 56 |     "# Number of Iterations to run\n",
 57 |     "max_iters = 10000\n",
 58 |     "hist_stride = 1 # stride for saved model paramters in `train.ipynb'\n",
 59 |     "stride = 50 # stride for computing loss\n",
 60 |     "\n",
 61 |     "# We vary the following parameters\n",
 62 |     "n_layer = 3\n",
 63 |     "mode = 'normal'\n",
 64 |     "N = 20\n",
 65 |     "seeds = [0,1,2,3,4,5]\n"
 66 |    ]
 67 |   },
 68 |   {
 69 |    "cell_type": "code",
 70 |    "execution_count": 3,
 71 |    "metadata": {},
 72 |    "outputs": [],
 73 |    "source": [
 74 |     "log_dir = 'log'\n",
 75 |     "loss_plots = {}\n",
 76 |     "\n",
 77 |     "for sd in seeds:\n",
 78 |     "    for (alg, toclip, lr) in [('sgd', True, 0.02),('adam', True, 0.02)]:  \n",
 79 |     "        filename = log_dir + '/train_layer{}_N{}_{}_{}_{}_lr{}_sd{}.pth'.format(n_layer, N, mode, alg, toclip, lr, sd)\n",
 80 |     "        loaded_dict = torch.load(filename)\n",
 81 |     "        hist_list = loaded_dict['hist_list']\n",
 82 |     "\n",
 83 |     "        np.random.seed(99)\n",
 84 |     "        torch.manual_seed(99)\n",
 85 |     "        Z, y = generate_data(mode,N,d,B,shape_k)\n",
 86 |     "        Z = Z.to(device)\n",
 87 |     "        y = y.to(device)\n",
 88 |     "\n",
 89 |     "        model = Transformer_F(n_layer, n_head, d, var)\n",
 90 |     "        model = model.to(device)\n",
 91 |     "        \n",
 92 |     "        test_losses = torch.zeros(max_iters//stride)\n",
 93 |     "\n",
 94 |     "        for t in range(0,max_iters,stride):\n",
 95 |     "            allparam_loaded = hist_list[t]\n",
 96 |     "            with torch.no_grad():\n",
 97 |     "                model.allparam.copy_(allparam_loaded)\n",
 98 |     "            test_loss = in_context_loss(model, Z, y)\n",
 99 |     "            test_losses[t//stride] = test_loss.item()\n",
100 |     "\n",
101 |     "        loss_plots[(alg, toclip, lr, sd)] = test_losses"
102 |    ]
103 |   },
104 |   {
105 |    "cell_type": "code",
106 |    "execution_count": 4,
107 |    "metadata": {},
108 |    "outputs": [
109 |     {
110 |      "data": {
111 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAJNCAYAAADHzfpbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAC0TElEQVR4nOzdd3wURf8H8M+l95BKT+hC6EWUAIINbCggTQQBQRFF7MJPfRQfLNgAxYZKE0E6iIViAQUJSJUSakhoAZIQEtKTu5vfH/vcJXu3d7l+l8vn/XrtC3Zvdnb2srf3vZnZGZUQQoCIiIiIquXj7gIQERER1RQMnIiIiIgsxMCJiIiIyEIMnIiIiIgsxMCJiIiIyEJ+7i4AuYZWq8XVq1dl22JiYuDjw9iZiIjIUgycvFR2drZsPScnB0lJSbJtWVlZiIuLc2WxiIiIajQGTl4qPj7e3UUgIiLyOmynISIiIrIQAyciIiIiCzFwIiIiIrIQ+zh5qaysLNm6UudwIiIisg4DJy/Fp+WIiIgcj4GTl1IajoCIiIjsw8DJS3E4AiIiIsdj53AiIiIiCzFwIiIiIrIQAyciIiIiC7GPk5ficARERESOx8DJS3E4AiIiIsdj4OSlOBwBERGR4zFw8lIcjoCIiMjx2DmciIiIyEIMnIiIiIgsxMCJiIiIyELs4+SlOBwBERGR4zFw8lIcjoCIiMjxGDiR1bYNGABkZkKl0QBaLeKmTkXSww+7u1hEREROx8CJrNbgt9/QqrRUv/73rl0AAyeX02q1UKvV0Gq17i4KEXkQHx8f+Pr6wtfX191F8UoMnLyUMwfA1KpU8g1qtcPyJvPUajUKCgpQUFCAoqIidxeHiDxYQEAAwsPDERYWhuDgYKgM791kEwZOXsqZA2AaBk6CgZNLFBUV4fz58xBCIDQ0FPXq1UNAQAB8fHx4QyQiAIAQAlqtFhqNBkVFRcjPz8fVq1cRFBSExo0bw8+PX/v24jtIVmPg5Hq6oCk0NBT169fnzY+IqhUREQEhBEpKSnDx4kVkZGQgISEBAQEB7i5ajcZxnMhqWh+Dy4aBk1Op1Wp90NSwYUMGTURkMZVKhZCQEDRp0gQqlQpnz56FEMLdxarRGDiR1Vjj5FoFBQUQQqB+/frwMQxaiYgs4O/vj4YNG0KtVqOwsNDdxanR+NPVSzlzAEx2DnetgoIChIaGsqaJiOwSFBSEoKAg5OfnIzw83N3FqbF4J/ZSzhwA06ipTqNx2rFqO61Wi6KiItSrV8/dRSEiLxAZGYmsrCxotVrWYNuI7xpZjYGT66j/V5vHzpxE5AgBAQEQQkDD+7bNGDiR1QQ7h7uMbnBL/jIkIkfQDYrJwMl2vBuT1YRhHyd+AJ2O4zQRkSPwXmI/Bk5kNQ5HQEREtRU7h3spZ065YtRUxxonIiKqJRg4eSlnTrnCwImIiGorNtWR1bQGM26rGDgREVEtwcCJrMbO4UREVFsxcCKrCdY4ERFRLcU+Tl7KmVOuGPVx+t9YQ0RERN6OgZOXcuaUK6xxIqoZioqKsH//fpw+fRp5eXkoKipCUFAQwsPD0ahRIzRp0gStWrWCv7+/w45ZVlaGw4cPIy0tDZcuXUJxcTF8fHwQGRmJyMhINGvWDO3bt0doaKjDjknkSgycyGqGNU4MnIg8h1arxYoVK/D111/jzz//1I8+b0pgYCA6dOiAW265Bf3798ctt9yCwMBAq45ZXFyMpUuXYuXKlfjrr79QXl5uNr2Pjw9atmyJ++67D0OGDMFNN91k0cCMGRkZaNq0qdk0KpUKgYGBCA4ORlxcHOrXr49WrVqhQ4cO6NmzJzp16mTXIJCm9t24cSPuuusum/J8/fXXMWPGDKPtffr0wbZt22zKk5xIUK2QlZUlAMiWrKwsm/L6rX17IQD98ndSkoNLSzolJSUiNTVVlJSUuLsoVAOkpqaK7t27G33WrVnWrVtn8fHUarWYM2eOiI6OtuuYbdq0Ed99951Qq9Vmj5eenm7XcQCIBg0aiOeee06kpaXZ9B6bynfYsGE25afRaERCQoJinn369LEpT3N4T7Efa5zIaqxxIvI8Bw8exO23347c3FzZdl3tTsuWLREREYHy8nLk5ubi+PHjyMzMtPl4mZmZGD58OHbs2KH4eoMGDdC0aVPExMQgLCwMOTk5uHLlCs6cOYOCggJZ2mPHjmHUqFE4cOAAPvzwQ5vLZGm5Z8+ejY8//hiPP/443n33XdSpU8fufH/44Qfk5uYiOjraqv1+++03nDt3zu7jk+swcCKrsY8TkWcpKirCgAEDZEFTREQEpk6dikcffRT16tVT3O/y5cvYsmUL1q1bh40bN6KsrMyi4509exZ9+/ZFRkaGbHujRo3w/PPP45577sENN9yguG9FRQVSUlKwefNmzJ8/H1euXNG/VlpaatHxq0pPTzfaVlJSgry8POTl5eHYsWP4559/8Ndff+HSpUv6NFqtFl9++SU2b96MH374Ae3bt7f62L6+vhBCQKvVoqysDMuWLcPkyZOtymPBggX6//v7+6OiosLqcpCLubvKi1zDkU11W268UdZUt6tZMweXlnRYrU6WeP3112Wf7fj4eJGammpVHllZWeKtt94S27dvN5uutLRUdO3aVXY8lUol3nzzTVFaWmrVMUtKSsTnn38uGjVqJACIp556ymx6paY6S1VUVIgVK1YYlR2AiI2Ntfj9qrpfaGiouPPOO/XrXbp0sbg8QgiRm5srAgMD9fsPHjyYTXU1AMdxIqsZ1ThxOAIit1q2bJlsfdasWWjTpo1VecTFxeHVV19Fr169zKabPHky9u3bp1/38/PDihUr8Prrr1vdqTwoKAiTJk3C4cOH8dBDD1m1r7X8/PwwbNgw7Nq1Cy+99JKsk3dOTg4GDx6M4uJiq/N99NFH9f/fv38//v33X4v3Xbp0qb6Wz9fXF4888ojVxyfXY+BE1mPgROQxLl26hNOnT+vX/f39MWTIEKcc6+DBg/jmm29k2/7zn/9g6NChduVbp04dLFu2DC+//LJd+VjCz88P77//Pv7zn//Ith8/fhwzZ860Or9BgwYhKipKv75w4UKL963aTNevXz80bNjQ6uOT6zFw8lLZ2dmyJScnx2F5s8aJyHMYdvCOjY21uubHUoaBRfv27fHqq686LP+EhASH5VWdN954A7fddpts25w5c3D9+nWr8gkMDJTVli1durTa4RgAKQg9cOCAfr1qzRV5NgZOXio+Pl62OGrUcABGNU4+DJyI3EatVsvW8/PzoXHCAxsXLlzA6tWrZduefvpp+BrcD2oKHx8fvPHGG7JtBQUF+P77763Oq2rQk5OTgx9//LHafebPn6//f0xMDO6//36rj0vuwcCJrMcaJyKPER8fL1svLi7Gb7/95vDj/P7777KALCQkBCNHjnT4cVzplltuQadOnWTbNmzYYHU+Xbt2RceOHfXr1TXX6Z7A03n44YcREBBg9XHJPRg4kfUYOBF5jKZNmxoNNzBx4kQcO3bMocf5888/Zetdu3b1imlT7rzzTtn6zp07IYSwOp9x48bp/79p0ybZ0AeG1q9fLxs6gs10NQsDJ7Ken3z4LzbVEbnXqFGjZOtnz55Fp06dMGrUKPz0008oLCy0+xg7d+6UrXfv3t3uPD1Bjx49ZOt5eXk2DUg5atQofa2RRqPBt99+azJt1U7hnTt3ltVWkefjAJheKisrS7aek5PjuH5O7OPk8bRaLa5everuYni1mJgY+Ph4xm/PqVOn4vvvv8fFixf128rLy7F06VIsXboUvr6+aNu2Lbp3744bb7wRN998M9q1a2dV+Q07obdq1cph5Xen1q1bG207d+4cEhMTrconJiYGAwYMwJo1awBIzXVTp041Snf+/HlZUyprm2oeBk5eKi4uzml5q1jj5PGuXr1q1PeFHCsrK8upnzNrxMbG4ueff8a9994rC550NBoNDh06hEOHDumHE4iKisLtt9+OUaNG4Z577oG/v7/J/NVqtdE0KY6YpsQTVB1KQCc7O9umvB599FF94HTixAmkpKQY1WgtWrRIP/FyYGBgje8nVht5xs8lqlkMAieVDf0BiMixOnbsiAMHDmDChAnw86v+N/G1a9ewevVqDBw4EElJSVi3bp3JtEq1l5GRkRaX7dNPP4VKpbJoWbRokcX5OoJS4FRSUmJTXv3795eNxVS1SQ4AhBCy83vggQesntuO3I+BE1mPTXVEHikuLg5ff/010tLS8M4776BLly4WNcedPn0agwcPxpQpU/S1IbWF0vlWHVXcGoajf69YsUI2Gvm2bdtw5swZ/XrVDuVUczBwIusZNtWxxonIoyQkJOD//u//sG/fPuTm5mLjxo1466238OCDD6JRo0Ym95s7dy5ef/11o+1KtSL5+fkOLbO75OXlGW0LDg62Ob+qwVBBQYG+6Q6Qj93UqFEj9OvXz+bjkPuwjxNZj32cPF5MTIzRAwLkWDExMe4ugkUiIyNx11134a677tJvO336NFasWIG5c+fiypUrsvTvvvsuRo4cKXuYxN/fH+Hh4bJ+TtYETmPGjMF9992n+NqIESOwe/dui/NytGvXrhlts6fvWsuWLdGrVy/s2LEDgNRcN3r0aOTn52Pt2rX6dGPGjPGYhwvIOgycyGoqw6Y61jh5HB8fH4/puEyep0WLFnj11VfxzDPPYPz48Vi5cqX+Na1Wi9mzZ+Prr7+W7VO/fn1Z4HTixAmLjxceHo7w8HDF14KCgqwsvWOlpqYabbP2iTpDjz76qD5w+vPPP5Geno7NmzfL+k6NHTvWrmOQ+zDcJaupDJ6+YeBEVDOFhYVh6dKl6Nq1q2z7li1bjNImJyfL1vfs2ePUsrnKrl27ZOvR0dFo3LixXXkOGzYMYWFhACo7hFftKH7LLbegRYsWdh2D3IeBE1nNaDgCBk5ENZafnx+eeeYZ2bZz584ZPVnWt29f2frevXtRVFTk7OI5neH0ND179rQ7z9DQUAwdOlS/PnfuXFmgyU7hNRsDJ7KeQeDky8CJqEYznK8NMO77c/vtt8sm9C0uLrZpQlxPsm3bNvz777+ybY6abLfqwJZV38uwsDBZUEU1DwMnshprnIi8i69Bv0XAeJymRo0aGX3hf/LJJ7KJf2sSrVaLN998U7YtIiICI0aMcEj+vXr1UhxdfdiwYV4xx19txsCJrMbAici7GHaQjoyMVPxynzZtmmz98OHDePvtt51aNmeZPn06tm3bJtv2/PPP6/smOYJSkxynWKn5+FQdWc2wczib6ojcp6CgAFeuXLGrs/FXX30lW7/tttsU03Xs2BGPP/64LP2MGTOQlJSEIUOG2Hx8V1Kr1Xj11VfxwQcfyLYnJSXh5ZdfduixnnrqKfTq1Uu/rlKpHNKHityLNU5kNdY4EXmOq1evonXr1njkkUdw9OhRq/efPn06fv31V9k2c/OnffLJJ7jxxhv162q1GsOGDcPbb7+N8vJyq46tVqtRWFhoXYFtpFarsXr1avTo0QPvv/8+RJX7VlxcHNasWWPXwJdKwsPD0atXL/3CoMk7sMaJrGYYOBn3jiAiV9JoNFiyZAmWLFmCbt26YfTo0bj99tuRlJSkOH2IRqPBtm3b8M477+CPP/6QvdanTx+ztUeBgYFYs2YN+vTpg/T0dADSI/evvfYa5s2bhxdffBH33nsvmjdvbjKPc+fOYd26dfjkk09kU5DYIiMjw2hbaWkp8vPzkZeXh2PHjmH37t34888/cenSJaO0zZo1w4YNG9C6dWu7ykG1BwMnL2U4u3dOTo7D8mZTHZHn2rt3L/bu3QtA6qvUpk0bxMbGok6dOigpKcHly5dx+PBhXL9+3WjfpKQkLF++vNpjNG7cGDt37sSwYcOwfft2/fbz58/jmWeewTPPPINGjRqhadOmiImJQVhYGIqKinDt2jUcP34cly9fVsy3S5cuuOmmm6w636ZNm1qVXsfX1xePP/443n33XasmLCZi4OSl4uPjnZa3j2Hg5LQjEVF1QkJCkJCQgHPnzhm9lp+fbzTAoykPP/ww5syZg9jYWIvS16tXD1u3bsVnn32G//73v7h69ars9QsXLuDChQsW5dWmTRu88sorePjhh22eYNdSDRo0wPDhwzF58mQ0a9bMqcci78TAiaxm1FTHGicit4mPj8fZs2dx4MAB/PTTT9i+fTt2796tWKNkKDY2FkOGDMGjjz4q67dkKV9fX0yZMgUTJkzA0qVLsWLFCmzfvr3avk4+Pj5o3bo17rjjDowePRrdunWz+timqFQqBAQEIDg4GLGxsWjQoAFatWqFDh06oFevXujUqZPTgzPybioh+K3njSy5MWRlZdk0n1nK11+jx+OP69e1YAdxZyktLUV6ejqaNm3q9jm9qObQarXIyMjAqVOncO7cOVy/fh3FxcUICQlBREQE6tWrhw4dOtg9J5uSsrIyHDp0CKdPn8bly5dRXFwMX19f1KlTB3Xq1EHjxo3RqVMnjmXkJryn2I81TmQ1H8On6gBAqwU40zeRR/Dx8UGzZs3c0hQVGBiIG2+80aYaLKKagIGTl8rKypKt5+TkICkpySF5+wQEGG/UaBg4ERGR12Pg5KVsaYKzlGHncABS4KS0nYiIyIuwioCsZtg5HIAUOBEREXk5Bk5kNcNxnAAAarXrC0JERORiDJzIaib7OBEREXk5Bk5kNV/WOBERUS3FwImsxhonIiKqrRg4kdWUnqoTrHEiIqJagIETWU0pcNKUlbmhJERERK7FwIms5hsYaLRNU83cVERERN6AgRNZTanGScvAiYiIagEGTm5QUFCA999/HzfffDNiY2MRHByMpk2b4pFHHsFff/3l7uJVy1ehc7i2osINJSEiInItTrniYvv378fgwYNx9uxZ2faMjAxkZGRgyZIlePrpp/Hxxx9DpVK5qZTm+fj5QQt51M2mOiIiqg1Y4+RCGRkZuPvuu/VB06OPPorNmzdj9+7d+PLLL/Uzmc+dOxfTpk1zZ1HN8vX1heHgA2yqIyKi2oCBkwu98MILyMrKAgB88sknmD9/Pvr164fu3btj4sSJ2LNnD1q2bAkA+PDDD3Ho0CF3FtckX19fGA4+wKY6IiKqDRg4ucjx48exdu1aAECPHj3w9NNPG6WJjo7G3LlzAQBarRYzZ850aRktpVjjxMCJiIhqAQZOLrJq1Sr9/x9//HGT6fr164fExEQAwI8//oiSkhKnl81abKojIqLaip3DzUhPT8fBgweRmZmJwsJC1K9fH4mJiUhOToa/0nxtZmzbtk3//9tuu81kOpVKhdtuuw0LFy5EYWEh9u3bh169etl6Ck7h6+sLwzCJI4cTEVFtUGMCpzNnzmDPnj3Yu3cv9uzZg/3796OgoED/emJiIjIyMhxyrNWrV2PWrFlISUlRfD06OhrDhw/Hf//7X8TGxlqU59GjRwEAYWFhSEhIMJs2KSlJ///U1FSPDJxY40RERLWRRwdO27Ztw7vvvou9e/ciNzfX6ccrLCzEY489huXLl5tNl5ubiy+++AJr167F4sWL0b9/f7Ppy8rKcOXKFQCoNmgyTGM4bIEnYOdwIiKqrTw6cDp48CC2bNnikmNpNBoMHz4cv/zyi2x7XFwcOnfujMjISKSlpeHAgQMQQgAArly5ggceeAC//fab2VqhqjVj4eHh1Zalapqq+3oKxRonNtUREVEtUCM7hwcGBqJ58+YOzXPatGmyoMnf3x9z587FhQsXsHnzZqxcuRL79u3DkSNH0KNHD326srIyDBw4EJcuXTKZd9UO3gEKo24bCqwyF1xxcbG1p+J0Pj4+RoGTYFMdERHVAh4fOPn7+6NTp06YMGEC5s2bh3379qGgoADffPONw45x5swZfPzxx7Jtq1atwuTJk40CnaSkJPz++++y4Onq1at48803TeYfHBys/3+5BQFGWVmZ/v8hISHVpnc1Hx8f46Y61jgREVEt4NFNdWPGjMETTzyBoKAgpx7nzTffREWVPjpjx47FAw88YDJ9cHAwFi1ahPbt2+sDofnz5+Pll1/Wj/5dlbVNb4WFhYr7ehKjGif2cSIiolrAo2ucoqKinB40lZSUYPXq1bJtU6dOrXa/Vq1aYeDAgfp1tVqNZcuWKaYNDAxEfHw8AOD8+fPV5l21Q7glncndQWuwzsCJiIhqA48OnFxh8+bNsn5EPXr0QOvWrS3ad9y4cbJ13cjgStq1awdAqnE6d+6c2XxTU1P1/2/btq1FZXE1tcEExHyqjoiIaoNaHzht2rRJtt63b1+L9+3duzf8/CpbOw8cOKAfdsBQnz599P//448/TOYphMDWrVsBAKGhoejatavF5XElrUHgxBonIiKqDWp94HTkyBHZetVO39UJDQ1F+/btZdt0A10aGjJkiP7/X331lck8t2zZoh/I87777pN1LPckRoETO4cTEVEtUOsDp2PHjsnWW7RoYdX+hsMiVG1mqyopKUnfJyolJQWffvqpUZrc3Fz95L8+Pj6YNm2a2WNnZ2dbvOTk5Fh1XtUx7BwOjdEWIiIir+PRT9U5W25urtGI5NZ2xjZMf+rUKZNpZ82ahR07diAnJwdPP/00Dhw4gIceegiRkZE4ePAgZs6ciTNnzgAAnnvuOXTq1MnssXUdzt2BTXVEZI1FixbJ+oUuXLgQY8eOdV+BiGxUq2uc8vLyZOshISEIDQ21Kg/D4CU/P99k2qZNm2Ljxo1o3LgxAGDBggW488470b17dzz++OP6oOnJJ5/EBx98YFU5XE3DpjqiGmHLli1QqVSyxdPmvySqSWp14FR1vCQANvUnMtynunGaunXrhiNHjmDmzJno3r07oqOjERgYiMTERDz88MPYtm0bPvvsM6gMAhNPwz5ORDXDggULjLb9/fffOH78uBtKQ1Tz1eqmOsPAyZYxowwDJ8M8lURERGDq1KkWjRflqdhUR+T5cnNzsX79esXXFixYgPfff9+1BSLyArU6cDJkSy2PO2uGsrKyLE6bk5ODpKQkhx1b4yOvrBTsHE7kcb777jvZFE5Vffvtt3jnnXdkQ6oQUfVqdVNdWFiYbL3qZLyWMtzHME9niouLM7k4m2GNE1jjRORx5s+fr/+/j48P7r77bv36lStX8NNPP7mjWEQ1GgOnKmpa4GROfHy8bHFkbRMACMOmOtY4EXmUvXv34tChQ/r122+/Ha+++qosjVL/JyIyr1YHTpGRkbL14uJiFBUVWZWHYXNZnTp17C1WjWDYVAd2DifyKFVrmwBp8vKePXuiZcuW+m2//PILLl265OqiEdVotbpxOyYmBlFRUbh27Zp+27lz59CmTRuL86g6IS8A2U3JmxnWODFwIvIcJSUl+P777/XrERERGDRoEAApgNLVPGk0GixevLjawXbNKSwsxLZt23Du3Dlcv34d9evXR0JCAnr16gV/f3/7TqSKtLQ0HD9+HGfPnsX169ehUqkQFRWFBg0a4KabbnJKF4WCggJs374dFy9eRHZ2NiIjI9GpUyf06NEDPoY/Hg2cPXsWKSkpOH/+PDQaDerWrWvVXKjkwUQNtXXrVgFAvyQmJtqUT3JysiyfDRs2WLV/ly5dZPv/9ttvNpXD0aqWydSSlZVlc/6/1KkjBKBfDt9/vwNLTzolJSUiNTVVlJSUuLsoVIN8++23ss/6Y489pn/t/PnzwsfHR/9ay5YtbTpGZmamGD16tAgKClK8v8TGxooXXnhBXLt2TQghxMKFC2WvL1y40Gz+hYWFYvny5WLYsGEiPj6+2vtZly5dxHfffSfUarXF5zBmzBhZHunp6UIIIdLS0sSoUaNEcHCw4rESEhLEihUrFPPcvn276N27t8ly3njjjWL37t0Wl9HReE+xX61uqgOAdu3aydZTUlIs3reoqEjWh0ApP3dJTU2VLTt27HBo/lrDX1vs40TkMZSa6XQaNWqE22+/Xb9+6tQp/PXXX1bl/9tvv6FNmzZYsmQJSktLFdPk5OTgo48+QqdOnWwaM+qhhx7CiBEjsHLlSoueIN6/fz9GjRqFfv36ITs72+rj6WzatAkdO3bEd999Z7Lf67lz5zB8+HC8+eabsu3vvvsu+vTpg+3bt5vMf8+ePejduzc75tdgtbqpDgDuuusu2aS727Zts3jf7du3Q12liapz586oW7euI4tnM0d3BjfEpjoiz3T69GlZINSyZUskJyfL0owdOxa//vqrfn3BggW45ZZbLMp/27ZtuP/++42CisTERLRv3x6hoaG4cOECdu/eDbVajbNnz6J///6YMmWKVeeh1Wpl6xEREUhKSkJ8fDzCw8NRWlqKCxcu4NChQ7Ky/PHHH7jrrruwc+dOBAYGWnXM3bt3Y+zYsfpgsFGjRujUqRPCw8Nx6dIlpKSkyIZ3mD59Ojp27IiBAwfiww8/xCuvvKJ/rW3btmjZsiUCAwORlpaGffv2QQgBACgvL8dDDz2E1NRU/UwSVHPU+sCpf//+CA4O1n/wUlJScPz4cYvaoRctWiRb1/UhqA0Ma5z4VJ2H0WqBq1fdXQrvFhMDVNPPxR0WLFig/4IGgDFjxhilGTRoECIjI/VTRK1atQqffPIJIiIizOadl5eHUaNGyQKVVq1a4fPPP5fVYgHA1atXMX36dHz66ac4d+4c3n77bavPpWPHjhg1ahTuuecekz8Gi4uLsWzZMrz66qv6mqn9+/fjjTfewMyZM6063qRJk1BaWoo2bdpg7ty5RueUnZ2Nxx57DD/88IN+2wsvvID4+Hh9P7H77rsPH374IW644QbZvkeOHMGwYcP0E8sXFhbiP//5j9H3CNUA7m4rtJWj+jgJIcTo0aNleY0dO7bafU6cOCECAgL0+/j5+YnTp0/bXAZHg5m+ALrFnj5O6+rWlfdx6tvXgaUnHZv7I2Rlyf4+XJyw2PH5cRa1Wi0aNGig/4z7+PiIc+fOKaZ97LHHZPeDefPmVZv/008/LdsnKSlJ5OTkmN1n9uzZivef6vo46fobWers2bOicePG+vwjIiJEfn6+2X0M+zgBEF27dhV5eXkm96moqBDt27eX7RMVFSUAiIkTJwqtVmty34yMDFm/qZCQEHH9+nWrztNe7ONkP8/7ueQG06dPlz39sWjRImzYsMFk+tLSUowbNw7l5eX6bePHj0fz5s2dWk5PItjHicjjbNy4EZmZmfr12267zWRT0Lhx42Trhv2iDBUUFGDhwoX6dV9fXyxduhQxMTFm93v22Wdx3333VVd0I02aNLEqfUJCAmbPnq1fv379On788Uer8vD398fy5cuNhqqpys/PDy+99JJs27Vr15CUlIRPPvnE7GwSiYmJGDFihH69uLjYqn615Bk8PnC6cOECMjIyjJbLly/L0qnVasV0GRkZyMnJMXuMZs2a4ZlnnpFtGzJkCD799FNZcAQAx44dw+23346dO3fqt8XExOCNN96w80wdKysrS7akpqY6NH8GTkSex1yncEM9evSQNSf9888/OHLkiMn0a9askc3FOXjwYHTq1Mmicr311lsWpbPXgAEDEBAQoF+vep+2xNChQ9GiRYtq0915551G255//nnZsU3p16+fbP3gwYMWl488g8f3cerVq5fRWElKLl68iKZNmyq+NmbMmGrbkWfOnImjR49i48aNAICKigo8/fTTmDFjBrp06YLw8HCcOXMG+/fvl/UfCAgIwLp161C/fn3LT8oFnD3tCgMnIs+SlZWFn3/+Wb8eERGBwYMHm91nzJgxsg7NCxYswKxZsxTTGj6Z+9BDD1lcto4dO6Jt27Y4evSoxfuYotVqUVRUhIKCAqMftgAQHR2t/2Gt609kqapT0phTr149hIWFyQLJu+66y6J9Dcf6u3LliuUFJI/g8YGTq/j6+mLlypWYMGECVqxYod+elZWFTZs2Ke4THx+PxYsXo3fv3q4qpsUMH8etrtbNWhyOwMPFxABWTAJNNqimicrVvv32W1RUmTNy2LBhCA4ONrvPI488gtdee03/BNuSJUswc+ZMxZqTvXv3ytZvuukmq8p300032RQ4FRUV4aeffsIPP/yAgwcP4tSpU7Knmc2pOrixJawZ/DgyMlIfOIWHh6Nhw4YW71fV9evXLS8geQQGTlWEhYVh+fLlGDJkCD766CPs2rVLMV10dLR+DA9XTKhri/j4eKfmL3x9ZesqBk6exccH8NBrk5zDcN45c810Og0bNsQdd9yBLVu2AJB+YG3YsAFDhgwxSlu1ZiQkJAQNGjSwqnytWrWyKj0AfPPNN3jllVdsHpdJ99SgpaKioixO6+dX+fVpzVRbVfcDIAt2qWbw+MApIyPD5cccMmQIhgwZgvT0dOzfvx+ZmZkoKipCvXr1kJiYiJ49e1rUlu3VDGqcGDgRuc/OnTtlzVItWrRAz549Ldp33Lhx+sAJkPpJKQVOVWtvqhu2QIm5DtdKnnvuOcyZM8fq41RlOBZUdaqbRsXR+1HN5PGBkzs1bdrUZL+p2k5rUOMEK29QROQ4hp3CT58+bfbpLnO2bNmCCxcuoFGjRibT2Jq3pVatWmUUNLVt2xYjRoxA9+7d0aRJE9StWxdBQUFGg1w2adLEon6xRLZi4EQ2MewczhonIvcoLCzEypUrHZafVqvFokWL8Nprr8m2R0VF6TtdW9sEZu0+hk8pz5gxA6+++qpFARv7DJGzsX7RSzl7OAI21RF5hhUrVsie7nIEw9HHAcimkyouLpaNF2WJkydPWpTu1KlTsmbHW265Ba+99ppFQVNJSQny8vKsKheRtVjj5KWcPhwBm+qIPIJhM913331ncf+mqsaMGaOf4y49PR1bt27Fbbfdpn+9W7du+Pfff/Xru3btqna4g6p2795tUbpTp07J1q0ZPDMlJcUo4CNyNAZOXsrZwxEYBk4+rHEicrljx47JRp6OjY3F8OHDjZ7cssSoUaNkkwPPnz9fFjj16tVLFqR9//33FgdO//77r8VDERjWGFnTqZzzvpErsKnOS8XHx8sWUxNk2szwKRLWOBG5nGFt09ChQ20KmgDpaeKqTwuvXbtWFsQMHjwYYWFh+vV169bJaqDMMewvZY7hkADHjx+3aL+9e/di+fLlFh+HyFYMnMgmwuDmrGLgRORSFRUVWLJkiWzbyJEjbc4vKipKNvp1aWkpli1bpl+PiIiQjQ2l0Wjw8MMPIzc312y+c+bMwU8//WRxOTp27ChbX7x4sdEUW4bOnDmDoUOHckwkcgkGTmQbgxonNtURudaPP/6IrCqjwyckJNjUt6kqw8DLsEZrxowZsoEvjx49ih49euD33383yis3NxdTpkzB888/D8DywSUbNGiAXr16yfK59dZbFQckLisrwzfffIObb74ZGRkZCAoKktWKETkD+ziRbVjjRORWhkHNQw89ZPf4Svfff79sDrb9+/fj4MGD+sl869Spg6VLl+Luu+9GaWkpAOlpuTvuuAOJiYno0KEDQkNDcfHiRezatUtfA9S4cWNMmTIFL730kkXleO+999CnTx/91CrHjx/XT0rcrl07BAQE4MqVK/jnn39kTxTOnTsXb731lsOfMiSqijVOXsrpwxEYdg5n4ETkMhcvXsTmzZtl2+xpptMJDg7GwIEDZdsMA7S+fftiw4YNRqOHnz17Fj/++COWL1+O7du3y4KmTZs2ITY21uJyJCcn4+uvv4a/v79s+4kTJ7BmzRp8//33+OOPP/QBkq+vLz7++GNMmDDB4mMQ2YqBk5eKi4uTLdbctCxhNAAmAycil1m0aBE0VZrHk5KS0KFDB4fkbRiALV26VF+7pHPnnXfi+PHjGD16NIKCghTziY6OxpQpU3Dw4EGbHk4ZO3Ystm/fjr59+5pMExQUhAcffBD//PMPpkyZYvUxiGyhEhz0olbIzs42mvg3KyvL5vGelvTvj9FV5rc6FxODBAcPeUBSB9309HQ0bdrU5BcUkTsVFBRg69atOH/+PAoKClC3bl0kJiaiV69eDpvT89y5c/j777+RmZmJiooKxMfHo0GDBujZsyfCw8MdcozagvcU+7GPE9mG4zgREYDw8HDcf//9Tj1GQkICEhISnHoMIksxcPJSzh4A0zBwUrHikoiIagEGTl7KsFnO4QyeqmPncCIiqg3YOZxsomLgREREtRADJ7KN4ThObKojIqJagIET2cagj5Mva5yIiKgWYB8nL1V1KgZA6hzuyIl+DZvqWONERES1AQMnL2Xr+EwWM+zjxMCJiIhqATbVkW0Mm+oYOBERUS3AwIlsYvRUHQMnIiKqBRg4kU1UBpNvcjgCIiKqDRg4kU0Ma5zYVEdERLUBO4d7KWdPuWJU4+TQ3ImIiDwTAycv5ewpVwxrnPxY40RERLUAKwrIJoaBEwCA/ZyIiMjLMXAimxg21QEANBrXF6SWEKzRIyIH4L3EfgycyCY+SoGTWu36gng5Hx/pI6plbR4ROYDmfz9wfQ3G4iPLsY+Tl3L2lCuGA2ACYI2TE/j9r0m0vLwcoaGhbi4NEdV05eXlUKlUDJzswMDJSzl7yhXFpjrWODmcj48PQkNDUVBQgKioKHcXh4hquPz8fISFhelrs8l6fOfIJopNdaxxcorw8HAUFRVBzcCUiOxQWlqK0tJSREZGursoNRoDJ7IJAyfXCQ8Ph0qlwqVLl9jXiYhsUlFRgYsXL8LPzw9hYWHuLk6NxsCJbMKmOtfx8/ND48aNUVRUhIsXL7LmiYgsJoRAcXExMjIyIIRAYmIiVCqVu4tVo7GPE9mENU6uFRoaisaNG+P8+fM4deoUQkNDER4ejoCAAPj4+PBGSEQApEBJq9VCrVajqKgIhYWF0Gg0CAoKQuPGjfUPnJDt+A6STXwCAow3MnByqtDQULRo0QIFBQUoKCjA5cuX3V0kIvJggYGBqFOnDsLCwhAcHMwfWA7CwIlswnGc3MPPzw9RUVGIiorS/6pkvyciqsrHxwe+vr4ccsBJGDiRTdhU534+Pj4IUKr5IyIip2Hg5KWys7Nl6zk5OQ7NnzVORERUGzFw8lLx8fFOzd/Hzw9aGDyWyRonIiLychyOgGzi6+sLozCJgRMREXk5Bk5kE19fXxg1zLGpjoiIvBwDJ7KJYuBUUeGOohAREbkM+zh5qaysLNl6Tk4OkpKSHJa/r68vygCEV91YVuaw/ImIiDwRAycvFRcX59T8dYGTDAMnIiLycmyqI5v4+vqi1HAjAyciIvJyDJzIJko1TqLUKJQiIiLyKgycyCZKgZO2pMQtZSEiInIVBk5kEx8fHwZORERU6zBwIpsoNtUxcCIiIi/HwIlswj5ORERUGzFwIpswcCIiotqIgRPZRGk4AgZORETk7Rg4kU1Y40RERLURAyeyCQMnIiKqjRg4kU0UAyeOHE5ERF6OgRPZRHGuOtY4ERGRl2PgRDZh4ERERLWRn7sLQM6RnZ0tW8/JyXFo/kojh3OSXyIi8nYMnLxUfHy8U/NXGo6AgRMREXk7NtWRTRSb6srL3VEUIiIil2HgRDZRaqpTscaJiIi8HAMnsolKpUKFSiXfyBonIiLycuzj5KWysrJk6zk5OUhKSnLoMSp8fACNRr/OGiciIvJ2DJy8VFxcnNOPwcCJiIhqGzbVkc00fgZxN5vqiIjIyzFwIpv5hYXJ1lnjRERE3o6BE9ksIDxcvoE1TkRE5OUYOJHNAiMiZOs+FRVuKgkREZFrMHAimxkGTr4MnIiIyMsxcCKbBdepI1v31WoBrdY9hSEiInIBBk5kM8PACQD7ORERkVdj4EQ2C4mKMt5YajT1LxERkddg4EQ2C4mONt7IIQmIiMiLceRwFyooKMD+/fuxd+9e7N27F/v27cPp06chhAAApKeno0mTJu4tpBXCYmKMNzJwIiIiL8bAyYVuueUWHDx40N3FcJjw2FjjjQyciIjIi7GpzoV0NUsAEBkZib59+6JevXpuLJF9Ihg4ERFRLcMaJxd69NFHERcXh27duqFFixZQqVTo27cvLl++7O6i2aROVBTKAARW2SZKS6FyV4GIiIicjIGTC02ZMsXdRXCoOnXqGAVOpfn5CHZXgYiIiJysRgdO6enpOHjwIDIzM1FYWIj69esjMTERycnJ8Pf3d3fxvJ4ucKqq8OpVBk5EROS1HBY4nTlzBnv27MHevXuxZ88e7N+/HwUFBfrXExMTkZGR4ZBjrV69GrNmzUJKSori69HR0Rg+fDj++9//IlapHw45REREBC4ZbCvOzXVLWYiIiFzBrsBp27ZtePfdd7F3717kuuALs7CwEI899hiWL19uNl1ubi6++OILrF27FosXL0b//v2dXrbayM/PDxUqFVCl03vxtWtuLBEREZFz2RU4HTx4EFu2bHFUWczSaDQYPnw4fvnlF9n2uLg4dO7cGZGRkUhLS8OBAwf0T69duXIFDzzwAH777Tf06tXLJeWsbdS+voBarV8vyctzX2GIiIiczCnDEQQGBqJ58+YOzXPatGmyoMnf3x9z587FhQsXsHnzZqxcuRL79u3DkSNH0KNHD326srIyDBw4EJcuGTYqkSNo/OSxd2l+vptKQkRE5Hx293Hy9/dH27Zt0a1bN9x4443o1q0b2rdvj7///hu33nqrI8qIM2fO4OOPP5ZtW7VqFR544AGjtElJSfj9999x++236/tAXb16FW+++Sa+/PJLs8dZv349CgsL7S5vcnIymjVrZnc+NYHG3182P13Z9etuLA0REZFz2RU4jRkzBk888QSCgoIcVR5Fb775JioqKvTrY8eOVQyadIKDg7Fo0SK0b98e5eXlAID58+fj5ZdfNhvQPPvsszh79qzd5V24cGGtCZxEQIBsvbzKAwFERETexq6muqioKKcHTSUlJVi9erVs29SpU6vdr1WrVhg4cKB+Xa1WY9myZY4uHjFwIiKiWsTjx3HavHkziouL9es9evRA69atLdp33LhxWLlypX597dq1eO2110ymP3ToELRare2F/Z+QkBC786gxDAJntQOaOomIiDyVxwdOmzZtkq337dvX4n179+4NPz8/qP/31NeBAwdw5coV1K1bVzF9RESEzeWsrVQGgZOmSpBLRETkbTx+kt8jR47I1qs+MVed0NBQtG/fXrbt6NGjDikXSXyC5eOEa0pK3FQSIiIi5/P4wOnYsWOy9RYtWli1v+GwCKmpqXaXiSr5GjRLahk4ERGRF/Poprrc3FyjEckTEhKsysMw/alTp+wul6fIzs62OG1OTo5TyuAXGipbF1WGJiAiIvI2Hh045RmMQh0SEoJQgy/q6sTHx8vW8904QOPp06exY8cO2bbLly/r/7969WrZ3HphYWEYMmSIyfwMz80d/MPC5BvKDKf9JSIi8h4eHTgZDkYZbNCfxhKG+xS48XH5HTt2YNy4cSZff+mll2TriYmJZgMnT+AfHi5bV5WVQQgBlUrlphIRERE5j0f3cTIMnGwZM8owcHLEyOBUKdAgcPIXQjZ8BBERkTfx6MDJkC21GJ5U8zF27FgIISxeMjIy3F3kaoVER8vWgyBvfiQiIvImHt1UF2bQf6bEhie2DPcxzLMmy8rKMvmaYWfw3Nxc9OrVy+FlCIqMlK0HArhw4YLDJ3kmIiLyBAycarC4uDiTr7ms43hgoHwVwPkLF1xzbCIiIhfz6Ka6SIPajOLiYhQVFVmVh2GtTJ06dewtFlWlFDidP++eshARETmZRwdOMTExiIqKkm07d+6cVXmcPXtWtt6yZUu7y0VVKAROF1jjREREXsqjAycAaNOmjWz99OnTVu1/5swZs/mRnRg4ERFRLeLxgVO7du1k6ykpKRbvW1RUhEOHDpnNz1tlZWXJFqdNNcOmOiIiqkU8PnC66667ZOvbtm2zeN/t27dDrVbr1zt37oy6des6qmgEAAZjawWBNU5EROS9PPqpOgDo378/goOD9U/HpaSk4Pjx42jdunW1+y5atEi2PmjQIGcU0SO586m6rKwslJWVIdDgNSIioprO42ucQkJCjKYdee+996rd7+TJk1i3bp1+3c/PDyNHjnR4+Wo9hcAJAC5evOj6shARETmZxwdOADB9+nT4+/vr1xctWoQNGzaYTF9aWopx48ahvLxcv238+PEclNEZTARObK4jIiJvZHdT3YULF2T9iHQMp91Qq9UmpxAJCwtDbGysyWM0a9YMzzzzDD788EP9tiFDhmDWrFl4/PHHERAQoN9+7NgxTJgwATt37tRvi4mJwRtvvGHpKZE1DAInfwAqsIM4ERF5J5UQQtiTQZMmTYzGSrLWmDFjjPojGdJoNBgwYAA2btwo2x4fH48uXbogPDwcZ86cwf79+1H1lAICAvDbb7+hd+/edpWxpsnOzpat5+TkICkpSbYtKyvL7OjjFklLA1q0kG0KBjB95kxMnTrVvryJiIg8jMd3Dtfx9fXFypUrMWHCBKxYsUK/PSsrC5s2bVLcJz4+HosXL651QRNgfjoWhzJ4qg7gk3VEROS9akQfJ52wsDAsX74cq1atws0332wyXXR0NCZNmoQjR44YDWdQW2RnZ8sWw0l/HSY42HgT2FRHRETeye6mOndKT0/H/v37kZmZiaKiItSrVw+JiYno2bOnrN9TbaRSqapN45CmutJSo+CpBYA6Xbti79699uVNRETkYWp04ESmuSxwEgLw9ZX+/Z8OAM6EhuLKlSsIDQ21L38iIiIPUqOa6sgDqVRASIhsUwik6W5WrVrlnjIRERE5CQMnsp9C4AQA8+bNc31ZiIiInKjGPFVH1snKypKtKw1H4DAmAqddu3bh6NGjaNu2rXOOS0RE5GKscfJScXFxssXcAKN2M+jHVDWMYq0TERF5EwZOXsplwxEAJmucAGD16tXQarXOOzYREZELsanOS8XHx7vuYGYCp0uXLuHvv/+ulYOQEhGR92GNE9nPIHBqVreubH358uWuLA0REZHTMHAi+xkETt0MOqGvW7eOzXVEROQVGDiR/QwCp3bNm8vWL126hO3btwMAiouLodFoXFY0IiIiR2Lg5KWysrJkS2pqqvMOZhA4xQYHGw1BsHDhQpSUlODy5csoLi52XlmIiIiciJ3DvZTdU6lYwyBwQnExhg8fjtdff12/admyZWjcuDGys7PxxBNPoFOnTq4rHxERkYOwxonspxA4jR8/Hv7+/vpNFRUVeOuttzBv3jzceuutyM/Pd3EhiYiI7MfAieynEDg1aNAAI0aMUEyel5eHDRs2uKBgREREjsXAyUu5cwBM/K8P03PPPWdyl/379zuvPERERE7CPk5eyp0DYKKoCADQuXNn9O3bF9u2bTPa5eDBg84vFxERkYOxxonsZ6LGCQC++OILREVFGe3y77//Qgjh7JIRERE5FAMnsp+ZwKl169Y4deoU5s6dK0ty7do1XLhwwRWlIyIichgGTmQ/M4ETAERHR2PAgAEICwuTbWdzHRER1TQMnLyUOwfANAycVCoV6tWrZzR2EwMnIiKqadg53Eu5ewBMQ4GBgejSpQt27Nih33bgwAFnl4yIiMihWONE9jMMnEpLAYVJfVnjRERENR0DJ7KfYeAEACUlRpsMA6f09HRkZWU5qVBERESOx8CJ7KcUOCk01yUlJcmmYQGAF154wVmlIiIicjgGTmQ/CwOnwMBAPPjgg7Jt3333HX7//XfFbLOzs1FYWOiQIhIRETkCAycv5dIpV4KDjbcpBE4A8MEHHyA8PFy27amnnoJarZZt02q1KCgoQGlpqcOKSUREZC8GTl4qPj5etiQlJTnvYH5+QECAfJuJwKlRo0Z46623ZNtOnDiB1atXIy8vD8X/26+iogIVFRUMnIiIyKMwcCLHsGBIAp2nnnoKbdq0kW375JNPcOHCBWRlZUEIgfLyclRUVKCsrAxahSf0iIiI3IGBEzmGFYGTr6+vUafwlJQUvPHGG9i7dy9KS0tRVlaGwsJCqNVqo2Y8IiIid2HgRI5hReAEAA8//LDRIJ1r167F0KFD8cUXXyA5ORnJycl45plnUFZW5ujSEhER2YSBk5dy6ZQrgNWBU1BQECZNmqT42gsvvIBTp05BCIHNmzdj+fLljiolERGRXRg4eam4uDjZEhsb69wDhobK16sJnACpr5MlU8MsWbLE1lIRERE5FAMncgwra5wA6cm/TZs2YeTIkWbT/fvvv/aUjIiIyGEYOJFj2BA4AUDnzp3x9ttvY8SIESbTXL9+HWfPnrWndERERA7BwIkcw8bASaVSISIiArfeeqvZdNu2bbOxYERERI7DwIkcw8bACQCCg4PRp08fs2m2bNmCS5cuQaPR2FI6IiIih2DgRI5hZ+DUsmVLjBo1ymSaZcuWYefOnSgqKrK1hERERHZj4ESOYRg4WRng+Pj4YPr06QgLCzOZZsiQIZg8ebItpSMiInIIBk7kGHbUOOk0b94cBw4cwOzZs/HPP/8gMDDQKM2SJUtw+PBhW0vpdkIIXLt2jU2OREQ1FAMnL5WdnS1bcnJynHtABwROANC4cWPcc889iIyMxB133KGYZv369Tbl7QnUarVsMmMiIqpZGDh5qfj4eNmSlJTk3AM6KHAKDAxEcHAw1Go1Zs6ciaFDhxql+emnn/T/LygoQHZ2tk3Hcoe0tDQcPXqUgRMRUQ3FwIkcw0GBEwBER0ejUaNGaNu2LWbNmoUvv/xS9vqePXv0wVJubi7y8/NrxETAn3/+Odq2bYsBAwZgypQp0Gq17i4SERFZiYETOYYDA6fQ0FBERERApVKhTp066N69O0Kq5C+EwPr16/Hwww+jefPmuP/++x0+QKZWq3VorZBGo8Frr72mD5ZWrlyJnTt3Oix/IiJyDQZO5BgODJyqCg0NRXx8PHr37i3b/vjjj2PZsmXQaDQ4duwYnn76aYccTycvLw8XLlxwWE1WWloarl27Jtv29ddfOyRvIiJyHQZOXiorK0u2pKamOveAdg5HYIpKpULDhg0xbNgws+k2btyIEydOWJSnWq1GWVmZydffeust1KtXDz179sTmzZsV01jbzHbo0CGjbX/88QeEEBbnodVqcf36dav2ISIix2Lg5KXi4uJkS2xsrHMPWKeOfD03F3BgH557770XPj7mL9fZs2dXm49Wq8Xly5dx+fJlxQBky5Yt+M9//oOKigrk5OTgmWeeMUqXn5+PlJQU5OfnW1x+pcDpwoULFjfXbd68GTfeeCPuuusu7Nu3z+LjEhGRYzFwIseoX1++rlYDV686LPu6deti0KBBZtMsWbIEubm5ZtOkpqZi9erVOH78uFEfJrVajeeff162LS0tDRkZGfr1xYsXIzExEb169cLgwYNRWlpqUfmVAidAGhG9Onl5eRg5ciT279+PlJQUDBs2rEZ0hici8kYMnMgx4uMBlUq+7dIlhx7imWeeMft6cXExPvnkEwAwGmAyNTUVgwYNQqdOnfDcc8/h/vvvx4YNG2Rpvv76axw9etQo3x07dkAIgddeew1jx47V1zT98ccf+Pzzz7F79278/PPPOHLkCM6fP6/YDGgqcFqzZk2108j89ttvsoAwPT0dv/zyi9l9PEVxcbHZZlEiohpHUK2QlZUlAMiWrKwsxx4kLk4IoHLZvNmh2Wu1WtGsWTOj86i6hIaGiq+++kq8+eabIiMjQwghxOXLl0VERIRR2piYGJGeni40Go24du2aiImJUczzscceE1988YXZ4+qWG264QSxevFiUlpbqy52fn292n5kzZ5o974kTJxrtc++99zr0vXWW8+fPi9zcXHcXg4jIYVRCsKdpbZCdnY34+HjZtqysLMTFxTnuIB06AFWnQ1m8GHjkEcflD2DBggUYP368RWmjo6Px448/YteuXXjhhRdMphs2bBgOHjyIkydPKr7euHFjlJSUWDX6+kMPPYTPPvsMQUFBmDdvHp577jmTaWNiYnDmzBlEREQovt6iRQukpaXJtvn6+uLcuXNo0KCBxWVyteLiYvz7779o2rQp6tWr5+7iEBE5BJvqyHEM+zk5uKkOAEaPHo0+ffoAkJ64++qrrzBw4EDFtLm5ubjrrrswdepUs3muXLnSZNAEAOfPn7d6yprvv/8e0dHRiIyMNBs0AcDVq1cxa9Yso+1CCBw7dswoaAKkpsgFCxaYzFMI4dan706fPo22bdsiOTkZDzzwAEdKJyKvwcCJHMewVuHyZYcfwt/fH7/++it++uknbNu2DSNHjsTbb78NlWH/qv8pKCiwuiO1v78/AgICHFFcVFRUGG0bOnSo0Tx833zzDYQQ0Gg0KCkpwd9//42xY8eanSpnyZIlits3btyIJk2aoH79+hZ1PgeMh1coKyvDpk2bsH//fov2N/T222/rO9X/888/WL58uU35EBF5HHe2E5LruKSP09Sp8j5Ow4c7Nv8qSkpKRHZ2ttBqtUIIIR555BGL+iCFhISIjz/+WPj7+5tM89hjj4nevXsrvqZSqcT27dtFp06d9Nt8fX3FrFmzxP/93/+JkJCQasvw1ltviZSUFKPt77//vhg4cKBo3ry5RecCQPz+++9i4cKF4tChQ0Kr1YqioiIRHR2tfz0wMFBcunTJ7HtZUFAg9u7dK4qLi4UQQpSWlor27dsLAMLf31+sX7/e6r+PYTk7duxodR5ERJ6IgVMt4ZLAafZseeDUp49j8zejqKhIjBkzRnTo0EFMmTJFdOvWTTHQGDBggDh37pxYs2aNmDZtmhg3bpyoW7eu/vWEhASRnp4uXn31VcX9Bw0aJLRarbh48aIYMGCA6NGjh5g/f77Izs4WGRkZYvny5Yod0asuf/31l9BqtaJhw4YWB0jVLX5+fuKpp54S8+bNM3ptxowZJt+3DRs2iMDAQAFAjB8/XgghxKpVq2T79+jRw6q/RWZmplEZGjdubNffl4jIUzBwqiVcEjgtXy4PnFq1cmz+1aioqBBpaWni5MmT4sCBAyIgIMDonBctWiQ0Go3Iz88XZ8+eFadPnxZHjhwR77zzjpg6dapISUkR5eXlYteuXUb7dunSRVy8eFF/vOvXr4vDhw+L8+fPC41GI9Rqtbh27Zr46aefRGxsrACkp/y6du0qoqOjhY+Pjxg9erR+//Hjx1sVHPn6+or+/ftbHVQlJCQItVpt9H4VFRUZPUn4119/ib59+xrlodFozL73BQUF4urVq0IIIdauXWu0f7NmzRz0VyYici8GTrWESwKnbdvkgVNEhGPzt0BJSYl+KIB33nnHqFam6qPxWq1WH/BcuXJFHDt2TPb6zJkzRb169USXLl3ErFmzxNmzZ/VNg0IIodFoxJUrV2RDD+jyTUtLEwsWLBB///23yMrKEpmZmeLvv/8WeXl5+nSGNTuGS0BAgAgPDxcqlUr4+vqKadOmiQ0bNthUI/Xjjz8KIaTg8vnnnxddu3YVDRo0MEo3duxY0bNnT6Ptp06dMvmev/XWWyI4OFjUq1dPbN++Xbz00ktG+/v7+ysGb9YqKSlx/HVLRGQFDkdQS7hkOIKTJ4EbbpBvKyoynsfORcrLy3HrrbfqpzWZPHky5s6dq5hWCIGioiIEBwfD19dXvz0vLw+XL19GZGQkYmJiLO40LoRATk4OtFot4uLioFKpcP36dYSEhMDf3x8AcO3aNcTGxirOe7dkyRK0bdsWgYGBKCwsRFlZGTp27Ag/Pz+EhoZa+1agf//+2LRpE6ZNm4b33nvPZLqIiAjExsbizJkzsu1r165VHLn9xIkTaNOmjf4JvhYtWqBevXrYsWOHUdr09HQ0adLE6rJXlZeXh2vXriExMbHaKXiIiJzBz90FIC+iNFbP5ctAs2auLwuAgIAA/Prrr/j+++8BACNGjDCZVqVSISwszGh7WFgYGjdujJCQEJNP7pnKzzAojYyMlK1HRUWhdevWRhMwv/nmm+jTpw/CwsIQEBCAkpISFBYWIigoCAEBARgwYAB+/PFHi8sCSHPdzZ8/H3PmzDGb7vr167h+/brR9iNHjhgFTkII/Pzzz7JhD06fPo3Tp08r5p2WlmZX4JSSkoKPPvoI8fHxeOedd1DHcH5EIiIX4E82L5WdnS1brB2HyCbh4UBwsHybE8ZyskZISAgeeeQRPPDAAwixoeZLV8NjTdBkjWHDhsnWY2Nj8eSTT6JRo0aIiopCaGgoYmNjkZCQoK/tMtynZ8+e+OWXX2Q1ZUomTJhg8/QnhlPGaDQanD9/Hnv37rU4D6XxqCx1+fJl3HnnnVizZg2++OILvPzyyzbnRURkDwZOXio+Pl62mBsPyGFUKuNBMJ0wlpO1/P39ERsb67Tgxx6PPfaYrCbqtddeUyxr1WapkSNHYuzYsfD390f79u0xb9483H333fj000/1wdOtt96KTz/91GHlPFxlRPgdO3agefPmaN++vb42zxKmaqIsMW/ePNmcfl9//bXNeRER2YN9nLyUJUGCw/s4AUDPnsD/+hQBAD79FHjqKccew8ukpaXh22+/RZMmTfDQQw8hKCjIov2uXr2KkpISNGjQQB9YnThxAseOHcNtt92G0NBQdO3aFf/++6/i/jExMcjLy0P//v3xxx9/oLS01OSxfH19UVhYCD8/PzRr1gznz5+3+jwHDRqEtWvXQqvVoqioCGFhYRYHs507d8bBgwdl24qLixFsWMPpQrp+cUpNvETkvVjjRI7lgmlXvE3z5s0xbdo0DBo0yOKgCZD6SNWvX19WG3XDDTfg7rvvRnh4OHx9ffH5558rNuFNnz4dly5dwpkzZ7BixQrcf//9Zo+l0WgQHByMRo0a2RQ0AVKAmJmZiSlTpuD555/HuXPnTKYtKytDYWGhfv3ixYtGadLT06s9ZtWR2wsKCnD58mVoNBorS66svLwc165dQ3l5uUPyI6KagYETOZYLpl3xRsHBwVZ3dvbx8VEMigIDA/U1OcnJyVizZg369u2L7t2745577sGbb76JZ599Fv7+/khISEBYWJhRvylTrly5YnH57rnnHtn6oUOH0LNnT3z22Wf45ptv0KdPH2RnZxvt9/fffyMhIQHh4eHo0aMHPvjgA8V0hk/+GSoqKtIHShkZGZg3bx7+/PNPXL582eppeAyp1WqsWLECGzZsYOBEVMvwqTovlZWVJVvPyclxTT8n1jh5nAceeAB33HEHLl26hNLSUgQEBBg1cd19990ICQlx2GS8Y8eOxfjx4/HLL7/ItuvmrwOAs2fPYtiwYfj111/h51d5K3ryySf11++uXbuwa9cuxWPoAieNRoPMzEzUr19fn09aWhpmzpwJX19fjBkzBnfeeae+j9STTz6JN998E7GxsTaf34gRI7BmzRoAUqBn6ZyARFTzsY9TLeGScZwAYMECYPz4yvUOHQATfWzItUpKSnDlyhX4+/ujYcOGRq/37dsXf/75p835v/3227h27Rrq1auHe++9F9HR0WjUqJHiRMdVTZ8+HW+88QYA4Pjx42jTpo1Fx5syZQqeeuop3H777bhw4QLuuusu/Pjjj8jPz0f79u1xyUzQ/tBDD2Hp0qU2PTBw/vx5JCQkyLZdu3aNwyMQ1RKscSLHatRIvq7QN4XcQ9dHydRvpUmTJhkFTnXq1EFeXp5F+Y8dOxbx8fEoLS2Fv78/AgIC0KJFCxw7dszsfnPnzoWPjw82btyI/fv3W3QsQKpVeu6553DhwgUAwKZNmzBv3jykpqaaDZoA4Pvvv8f9999vdmwvUw4cOGC07dSpU7jxxhtl2yoqKkw2pxJRzcU+TuRYhoHT1atASYl7ykJG/Pz89COXGxo0aJDsy/+FF17A2bNnsWDBAmzcuBE9evTQv6ZSqVC3bl39+tChQ9GgQQP4+fkhLCxM38/q0UcfrbZMV69exeuvv46UlBSrxpnav3+/UVPg5MmT8fnnn1u0/5IlSwDAZCBpyokTJ4y2Ve1vJYTAhg0b8Pzzz+PXX3+1Km8iqgHcMM0LuYFL5qoTQojr1+Xz1QFCnDzp+OOQU2RnZ4tPP/1UfPnll6K8vFwIIUReXp64ePGiyM7OFsOHDxcdOnQQc+bMEQUFBeK7774T8+fPFyUlJYr5abVa8dlnn4nQ0FD9dRcfHy+io6MtmmMvOTnZprn5LF2aN28uoqOjxdtvv62fh3DHjh1i3Lhx4o033hAVFRVG5zRmzBijfN566y3968uXL9dvDwoKEnv27HHCX0qiVqtFTk6OKCsrc9oxiEiOgVMt4bLASQghIiPlgdMffzjnOOQUWq1W9kWs1Wr1QYVWqxXZ2dmiuLjYqjwPHz4snnrqKTF69Gjxzz//iEmTJlUb1ERHR4uSkhLRr18/pwZPumXixIniwIEDIigoSL9t0qRJRufSrVs3xcmRdXr06CF7beDAgTb+Jcw7fvy4uOGGGwQAMXjwYIdMokxE1WPn8FrCZZ3DAaBdO+Do0cr1xYuBRx5x/HGoRhFCQK1Ww9/fH+vXr1ecNLiqcePGYcGCBSgvL8fKlSuh1WoxceJEswN12is8PBwFBQX6dV9fXxw6dAhJSUkoLi7GxYsX0bFjR5QYND/37t0bf/31F/Ly8hAVFWWU79mzZ406lNtr5MiRspHbN2zYgAEDBjj0GERkjH2cyPEaN5av/6/zLtVuKpVK37/q1ltvNdtpWqVS4bHHHgMgTdZ83333ITk52eJJgqOjoxW3Vxe8VA2aAGmog5dffhkpKSmoV68eWrVqZRQ0AZXTyfzxxx+K+X711VdG29avX4/7778fL7/8sk3BoOF0N46cYoeITGPgRI5n2EHcxpGmyXtFRkYa1YACQP/+/dG/f38sXLhQ1hk9PDwcMTExaNGiRbV5N2nSBKtXr0br1q2NXnvrrbesLuvPP/+M5ORko6CqqkuXLqGkpARbtmxRfH3BggWyQTcPHjyIwYMH48cff8QHH3yADz74wKoyXVYYWFZpdHUicjwGTuR4hjVOZqbWoNpr4sSJsvUmTZpg48aN2LRpE8aMGSN7zdfXF1FRUWjWrJliXuvWrcOBAwewZs0aHDp0CLfccgsefvhho3QjR45ETEyMbFuDBg0QGBho59lIwyNs3rxZ8bVLly5h69at+vUPP/xQ9jTfrFmzFPcz9ZSh0rANWq3WmuISkY0YOJHjGdY4samOFDz22GOIjIzUr7/66qvVDkjZoUMHo22vvPIKBg4ciE6dOmHw4MH6efoeffRRWf4vvPCCfiRxHR8fHyxcuBA///wzQkJC7DqfzZs3y0ZGNzRz5kz07NkTffr0wdKlS2Wv5eXlYfbs2Rg+fDjWrl2LU6dO4cEHH8Q999yDv/76yygvpcDpzJkzDJ6IXICdw2sJl3YO//VXoF+/yvXoaGk8JyIDJ0+exNKlS9G8eXOMHj262sCpoKAAffr0wYEDBxAeHo7Zs2djfNWR6g3s2rULn3/+ORo1aoRp06YhIiICRUVF+M9//oN9+/Zh+PDhmDhxInx9fXH8+HF88MEHCA4OxgsvvIDOnTsjPz/f0adcLZVKJauNqlOnDk6dOiWbImbw4MFYt26d0b7p6ekW9wMjItswcHKxtLQ0bNq0CX/99RcOHTqECxcuoKysDHXq1EFSUhL69euH8ePHywYXdASXBk7HjwOG02YUFQF2/qIn76RWq6FWqxEUFGRR+vLycuzdu1ff56m6kblLS0tRVFQka6ITQuD8+fMIDw+XPQWn0WhQXFyMsLAwvPjiiyab0Czh5+dn92TCOlOmTMHHH3+sX09MTMQ5hSbwTZs2oX///g45JhEpY+DkQmPHjsXixYurTRcREYHPPvsMo0aNctixXRo4FRYC4eHybSdOAK1aOf5YVCtptVoIIeyazqSsrAx+fn4m8zhz5gyaN29uU95BQUFYtWqVw4YHCAwMxKlTp9C4cWPk5OSY/Nx+8sknmDx5sk1z8BGRZdjHyYV0c2qFhIRgxIgR+Oqrr/Dnn39i//79+OGHHzBq1CioVCpcv34djzzyCFasWOHmEtsoLAwwnPCU/ZzIgRwxB1xgYKDZPJo1a2YU+Lz88suYOXMmnnzySZNP6MXHx2P9+vW49957HTbxb1lZGaZPnw5Aea48ndTUVFy8eNFoeAOlbfbib26qtdwx6mZtNXr0aP1UFaZ8//33+hGHY2NjRVFRkUOO7dKRw4UQol07+ejhixY571hETnLmzBlRr149AUAkJSWJgoICodFoxKVLl8ShQ4eMPlM33nij2LVrl37k9TvuuMNhI5sHBgaKgoIC8e6775pNN2XKFPH3338LjUYjSktLRd++fQUAkZCQIC5cuGB0jhqNRuTl5QmNRqP4HpSVlRlNPVNeXi4uXrzI0cqpVmKNkwt9++23eOaZZxAWFmYyzYgRIzBw4EAAQE5ODn777TcXlc7BDIck4FhOVAM1bdoU586dw6FDh3DgwAGEhYXBx8cH9erVQ/v27fH8888DkMaZmjp1KpYuXYoOHTogICAAAGSTJturrKwMv/76K3bs2GE23SeffILbb78dM2bMwJdffolt27YBAM6dO4cpU6bI0q5evRoJCQlo3bq1UQ23EAIfffQR7r//fixcuFD2WmlpKYqLi1FeXm7/iRHVNO6O3MjY3Llz9b8eP/jgA4fk6fIap8cfl9c4PfaY845F5EaXL18Whw4dEufOnTOaw2/t2rU21zDFxMSIpKQk2baRI0fKJky2Zbl8+bIQQohly5YJlUql3+7v7y8OHjyoL/unn34q22/Hjh1CCCGuX78uZs+eLWbMmCEuXbrkujeayEP4OSkec5n09HQcPHgQmZmZKCwsRP369ZGYmIjk5GT99A41TdVfcfb243CbxET5+pkz7ikHkZPFxcUhJCQEoaGh8PGRV+Ir1TjVr18f165dQ2xsrL7fY1XvvPMO0tLSMHLkSBw6dAjPPfec/rVly5bZXd7Fixejffv2GD16tKyfUkVFBSZMmIBdu3bBx8cHkydPlu23dOlSJCcnY8iQIfoR0vfs2YMffvjB7jIR1SiOjMLS0tLE8uXLxYsvvij69OkjwsPDZb9YEhMTHXasVatWGc1CXnWJjo4WkyZNEtnZ2Q47pqvce++9+vPYvHmzQ/J0eY3TihXyGqeEBOcdi8hDabVa0aVLF/1nLiIiQmRnZ4tz586Jo0ePigYNGhh9LtVqtcjIyBD5+fkiLS3NbO1Rx44dRcuWLa2qcQoLCxPR0dEmX3/vvffEwYMHjbYHBweLzZs3G21PT083ef7Xr18XH330kZgxY4Y4f/686954IieyO3DaunWr6Nevn9kPoiMDp4KCAjFixAiLbxJ169YVmzZtsvu4rrJ7927h4+MjAIgGDRroO5nay+WB07598sBJpRKitNR5xyPyUAcOHBB9+vQRSUlJYtmyZUIIKaA4e/asmD17tuwz+fbbbwshpA7ZWq1WCCFE27ZtTd7fXnjhBfHtt98KX19fAUB069ZNdOzY0a6mvKCgIDF8+HDFH6N333230fZ58+aZPPdRo0bp07Vt21b8888/4qmnnhJvvfWWKOX9gGoouwMnww++MwMntVot7rnnHqN84+LiRL9+/cTQoUNFly5dZO32gPQ0yvbt2+09VafLz88Xbdq00Zd7kQOfRHN54JSfLw+cACGOHXPe8Yg8WEVFhTh//ry+D5RGoxElJSWisLBQDB06VERERIj+/fuLwsJCo32nTZtm8p6qq5E+ffq0WLdunTh8+LCsj6QlS1RUlF2B1tChQ4VWqzV68q6goMDoXlx1uf/++0VFRYXYvXu3vt8VUU1g9wCYc+bMkbXB6wQGBqJRo0ZIS0vTb0tMTDQ7l1N1XnrpJXz44Yf6dX9/f8yaNQuPP/64/ikWQBrLZMKECUhJSdFvi4mJweHDh1G/fn2zx1i/fj0KCwttLqNOcnKyyQlJlWg0Gtx///345ZdfAADDhg1z6DhOLh0AU6duXSArq3L9xx+B++5z3vGIPFhFRYXJfpd5eXnQarWIjo42eu3gwYPo0qWL0bhJgYGBuHbtGoKDgwFIT7pptVqo1WrUqVPHonGWoqOjsWvXLnz44Yf46quvbDgrybvvvguVSoWbbroJvXv3hq+vL3bs2IHevXtbtH9iYiI2bdqE1q1b21wGIpexN/KaPXu28Pf3F506dRITJkwQ8+bNE/v27RPl5eVi69atDqtxSktLE/7+/rL81q9fbzJ9cXGxUR+oiRMnVnucxMREu3596ZaFCxdafG5arVaMGTNGv2+3bt3MjvVkC5fXOAkhRHKyvMZp1iznHo/IS7311ltGn9++ffuaTH/fffcZpX/rrbdEUFCQ4n2qoKBANG7c2CH3vmbNmonMzEwxZ84cq/Zr06aNw+97RM5gd+CUm5srSkpKFF9zZOD0yCOPyPIaO3ZstfucOHFCBAQE6Pfx8/MTaWlpZvdxdeCk1WrFxIkT9ft17NhRXL161aJ9reGWwOmRR+SB05NPOvd4RF5s2bJl+qEIVCqVfngAJX/99Zfss3733XcLIaR74rBhw0SHDh3Ehx9+qO9HJYQQmzZtcsi9D4AYM2aMGD16tNX76Zr9iDyZU8dxclTgVFxcLEJCQmR5HbOwv8ywYcNk+82YMcNs+vz8fHHt2jW7F0s7dT/11FP6srVr185pTwG6JXCaMUMeOPXr59zjEXm57OxssXDhQtl4S6a88847IjY2VnTt2lUcPXpU9lpxcbFRnyQhpOCse/fuonnz5qJFixayH57WLEFBQaJ+/fo27bt69WohhPSjMjs722hE8/LycrFr1y6Rm5trxztJZLsaETitW7dOlk+PHj0s3nfjxo2yfTt37mxTGZxh8uTJ+nK1bdvWqYGMWwKn77+XB05Nmzr3eEQkc+XKFXHy5ElRXl5u8T6lpaXi8uXL4syZM2L9+vWKwU3VIVNsWaoLqm677TZ9s2JkZKT4+eefhRBCXLp0STRq1EgA0hQyhg/9lJaWio0bN4o///zTZTVXRUVFYsqUKaJ58+bi0Ucfddg0WeS5akTgVLUpC4D4v//7P4v3LSwsFH5+frL9PeEJjqeffloWNF25csWpx3NL4LRnjzxw8vERwkHDKxBR9QoLC20e3VtX46MU2GRlZYmmTZvq13v16iVuu+22agOmwMBAsXjxYlFWVia2bNkiUlJSLHoyu06dOqKoqEiMGzdOtj0yMlLs2LFDVFRUiE8++UQ2LtZrr73m4HfTWF5enujVq5esTLNnz3b6ccm9akTg1LNnT1k+GzZssGr/zp07y/b//fffbSqHo0yZMsWlQZMQbgqcrl0zHpLgxAnnHpOIZOyteQkMDJTdN3Sd0q9cuSLefvttMWfOHJGZmSl++uknk4FPUlKSyM/Pl/WHLS0tFXl5eaKsrMzsWFW6Zd68eUaDKgPScApKQZuPj4/YvXu3KC4uFqmpqWL37t121wZlZmaKqVOnimnTpokLFy6Irl27Gh13wIABivtaU+tHnq1GBE6Gg2umpqZatf+QIUNk+8+dO9emcjjCs88+K7uZ2BM0ZWVlWbykpqa6PnASQojYWHng9L8qdyKqGarW+Pv4+IiUlBTFdFqtVtxwww2KQc8jjzxi9hirVq2qNnByxNKkSROb77kajUZ069at2mPccMMNRvuWlpaK8+fPK/Yro5rH4+eqy83NRW5urmxbQkKCVXkYpj916pTd5bLF1KlTMWfOHABAbGws5s6di6ysLGRVHevIQFRUFBo2bKj4muG4TB6pRQsgJ6dy3U3vPRHZ5r///S/Kyspw6NAhPPbYY7j55psV06lUKsyYMQPDhg0zeq1Lly5mjzF48GB07NgR//77r0PKbEpGRgbmzJmDd955R/F1IQS+/PJLrFu3Dp07d8bUqVP1Y2sdOXIEe/furfYYaWlpsjG7SktL8dNPP6GwsBBDhgxBWFiY406I3MOZUZkjapwM52oKCQmxOo+ZM2fK8hgzZozVeTiCLUMdmCurtXkZLi6pcRo5Ul7j9Nxzzj8mETmcWq22qNnvm2++kY0X5efnJ06ePFntfmfOnBHdu3cXcXFx4uWXXxaFhYUWTeVl7dK6dWv9MdPS0sTx48f15/XKK6/I0iYkJOhHZ//8888tPsaJ/3VJ0Gq14sEHH9Rvt2QsQfJ88qm8PZDhKN66UXKtYbhPQUGBXWUiKyQmytfPnnVPOYjILr6+vlCpVNWmGz9+PHbs2IH7778fbdu2xRdffIGWLVtWu1/Tpk2xe/dunDp1CjNmzEBoaChGjRqlmDYoKMhky8N//vMfs8c5fvw4Fi9ejDvvvBPNmzdH69at0aBBA4SFhRnVRJ07dw4PPPAA9uzZg7///rvac9CZPXs2Pv74Y6xZswZr1qzRb//qq69w9epVi/Mhz+TxTXWGgVNQUJDVeRgGTo6YUsUW9kw3U2MZ3tzOnXNPOYjIZbp27YoffvgBGo0GPj7W/T6PjIzU//+JJ57AF198gYqKClmawYMHo0uXLnjxxRdl25s3b47p06cjMTERn3zyCWJjY3HPPfdg+vTpsvv+2LFjZftdvnzZZHlKS0vx8ssvm7x/N27cGDExMTh48KB+25dffqmYVgiBrVu3YsiQISaPR57P4wMnQ5b84nHEPjWBub5ROVX7FUHqK9arVy9nF8kYa5yIai1fX1+79m/Tpg2+//57vP322zhx4gTKysrQrVs3zJ07F+Xl5Zg6dSo0Go0+/fjx4+Hj44Px48fjwQcfRG5uLmJjY3Hy5Em75uLbtm2b4vauXbti0aJFmDVrlixwMufXX39l4FTDeXzgZNiRrqSkxOo8DPfxls555ibo9ZiO44Y1TtnZQEkJYEOTKxHVPg8++CAGDRqE0tJSAFILgu7H8NixYzF//nwA0v1wwoQJ+v3q1KmD8PBw+Pr6YuDAgVYFTs8++yyWLl2K7OxsxdcjIyORm5sLlUoFlUqFG264weK8//jjD4vTkmdi4ETOpdQP4dw5wIobDRHVbj4+PggJCTHa/sknn6BFixY4ceIEnnjiCaMfk7oar9tuuw1hYWFG3TSefvppDBw4EH/++ScKCwsRFBSEG2+8Ef3790dCQgKef/55xfLcdNNNsibIVq1aWXwup0+fxpQpU9C1a1dERETg448/xpEjR/DII4/go48+krWQ5OXlYePGjUhMTERycrLFxyDn8vjAqWp7NwAUFxejqKgIoaGhFudh2KRVp04dRxSNLBEeDkRFAdeuVW5j4EREDhASEoJp06ahoqICfn6mv84CAwMxfPhwfe0UAEybNg3vvPMOVCoVbrrpJhQUFCA4OBghISHw9/fHpEmT8OGHHyIzM9MoP8NuD9bUOAHA3LlzjbbNnj0bN954Ix566CEAUveK5ORknDhxAgDw8ccfY8qUKVYdx5S0tDRotVo0a9bM7ubU2sjjn6qLiYlBVFSUbNs5KzsYnzXoV2PJEx7kQOznRERO5O/vX21f1pkzZ2LkyJFo164d3nnnHbz99tv6fUJDQ1GvXj1ERkbqx18KCgrCV199pRhYGAZOzZs3d8h5fPHFF/r/P/300/qgCQD+7//+T9aJXQhhVd7bt2/Hl19+icuXL2PmzJlo1aoVOnXqhHfffdfq79TazuMDJ0DqIFjV6dOnrdr/zJkzZvPzRrqBNXVLamqq+wrDJ+uIyM1iY2OxdOlSHDhwAFOnTrXoab97770Xv/76K5o1a6bf1rlzZ9xyyy2ydIGBgYr7jxo1Cvfee6/JYRUMbd++HSdOnMCaNWuwbNky2WvFxcWYMWOGPt3777+vH8xZCAGtVitLX15ejtTUVJSUlOCzzz7DLbfcgkmTJqFdu3b45ptvAEiDer7yyitYsWKFReUjicc31QFAu3btsHPnTv16SkoKBgwYYNG+RUVFOHTokFF+5EKscSIiD2GuSU9Jnz59sGnTJvz000/Izc3F448/rlgLdd999+Gnn37Sr48ePRrffvstACAzMxPr16+3aCictm3bmizjV199hdjYWMyYMQNCCMycOROTJk3C119/DbVajaeeegqvv/468vLy0K9fP8WR2JXGkVIa7Z3McObomo6aq27t2rWyfHr06GHxvhs3bpTt27lzZ5vKUNPAgtFtXTJyuBBCfPCBfPTw/00SSkRUE6jValFYWCiuXbsm1Gq1YppffvlFf28NDw8XaWlpstc3b94sbr/9dvHggw+KH3/8UZw8eVLs3r1bPP300w4dGb1NmzYiISHB4vQ333yzK95Cr1Ijapz69++P4OBg/dNxKSkpOH78OFq3bl3tvosWLZKtDxo0yBlFJHNY40RENZivr2+1DyTdfffdSElJwaZNm3D33XfLmvcAoF+/fujZsyeys7MRHR2N8PBwqFQqREREKHYWt9WxY8esSj906FCHHbu2qBF9nEJCQowGDHvvvfeq3e/kyZNYt26dft3Pzw8jR450ePmoGoZ9nC5cAKoMWkdE5A1uvvlmvPbaa7jxxhsVXw8NDUVCQgIiIiL0HdNbt26NO+64QzH9gAEDcObMGbNj9tlDpVIxcLJBjQicAGD69On6px0AqSZpw4YNJtOXlpZi3LhxKC8v128bP368w55+ICsY1jhVVABmpjggIqqp/Pz8zHY8V3pt4cKFGDBggH6MwYCAADz//PNYvHgxmjRpgtmzZzulrN27d0fjxo2dkrc3c0jgdOHCBWRkZBgthvP/qNVqxXQZGRlGU4QYatasGZ555hnZtiFDhuDTTz+VBUeAVFV5++23yzqUx8TE4I033rDzTGsOj3qqLj4eCAiQb+OTdUREAIBGjRphw4YNuHz5Mi5fvoxLly7hvffeQ1RUFFQqFUaOHIl+/frJ9mnZsiUSExMRFBSEKVOm4OrVq0bjPA0ZMgTbtm3Dq6++isWLFxsd19Kn/UhOJYSVg0EoaNKkidFYSdYaM2aMUX8kQxqNBgMGDMDGjRtl2+Pj49GlSxeEh4fjzJkz2L9/v2yMi4CAAPz222/o3bu3XWWsSQynCsjJyUFSUpJsW1ZWltOqgI20bAlUHUbi+++BESNcc2wiohru4sWLuPPOO3Hs2DEkJSXh999/R3x8PK5fv46QkBAE/O/H6datW7F48WIkJCTg1VdflQ2VMHjwYH33lbi4OBw7dgwxMTFuOZ+arEZ0Dtfx9fXFypUrMWHCBNm4E1lZWdi0aZPiPvHx8Vi8eHGtCpoAD5qrTichQR44scaJiMhiDRs2xJEjR3Dx4kU0aNBAPySC4UwYt956K2699VbFPObPn48WLVrg3LlzePbZZxk02ahGBU6ANM/c8uXLMWTIEHz00UfYtWuXYrro6GgMHz4cb775putqVcg0PllHRGQXHx8fu/okRUVF4f3333dgiWonhwROGRkZjsjGKkOGDMGQIUOQnp6O/fv3IzMzE0VFRahXrx4SExPRs2dPfdUleQCOHk5ERF6gxtU4GWratCmaNm3q7mJQdVjjREREXqDGB06kLCsrS7au1DncpVjjREREXoCBk5fyuH5dhjVO+fnSEhnpnvIQERHZoMYMgEk1XKNGxttY60RERDUMa5y8lNI4Tm4VFATUqycfMfzsWaB9e/eViYiIyEoMnLyUx43jBEj9nKoGTqxxIiKiGoZNdeQ6fLKOiIhqOAZO5Dp8so6IiGo4Bk7kOqxxIiKiGo59nLyUx43jBLDGiYiIajwGTl7K48ZxAoxrnDIzgYoKwN/fPeUhIiKyEpvqyHUMa5yEAE6fdk9ZiIiIbMDAiVwnKgqIjpZv+/FHKYAiIiKqARg4eans7GzZ4vYBMAFApQL69ZNv27QJUKvdUx4iIiIrsY+Tl/LIATAB4L77gOXLK9dTUoDiYs5ZR0RENQJrnMi17roL8Kly2ZWWAlu3uq88REREVmDgRK4VEwMkJ8u3/fKLe8pCRERkJQZO5Hr33itf37TJPeUgIiKyEgMnL5WVlSVbUlNT3V2kSnffLV8/fx4oKHBPWYiIiKzAzuFeyiMHwNRp0sR427VrQHi4y4tCRERkDdY4keuFh0tDE1R17Zp7ykJERGQFBk7kej4+QJ068m0MnIiIqAZg4ETuERUlX2fgRERENQD7OHmp7Oxs2bpHjBxelSWBU04OEBwMhIa6pkxERETVYODkpTx25HAdw8ApL0++XloK5OZKc9sxcCIiIg/Bpjpyj+pqnK5fl6ZiKS11XZmIiIiqwcCJ3MNc4FRWBuTnAwEB0v+1WteWjYiIyAQGTuQe5p6qKy6WAqawMECtlhYiIiIPwD5OXiorK0u2npOTg6SkJDeVRoFhjVNubuX/NRopWLp2DfD1BSoqpNonIiIiN2Pg5KU8euRwwHxT3d9/A08+CWRmAjffDPzwAzuIExGRR2BTHbmHqcDp55+BESOkoAkAdu0CVqxwbdmIiIhMYOBE7qE0HMG+fcDAgcZP0u3Y4apSERERmcXAidxDqcbp+++VO4IfPswO4kRE5BEYOJF7GAZOFRVARoZy2lOngMJCpxeJiIioOgycyD0MhyMAgPR05bRqNbBnj1OLQ0REZAkGTuQe1gROgPSkHUcRJyIiN2PgRO7h6wtERMi3KU30q3P4sPF8dkRERC7GcZy8VHZ2tmw9JyfHTSUxIypKmpPOEsePS4FTZCQQHOzUYhEREZnCwMlLxcfHu7sI1YuKAs6etSztyZPSVCxFRd4VOAkhjZTux48iEVFNwKY6ch/DJ+vM0XUQz8+XAg1vUVoKZGVJARQREXk8Bk7kPtUFToYdyGfMAK5eBUpKnFYkl1OrpaEYtFp3l4SIiCzAwIncR+nJuqqmTpWvX7wIzJkDFBQ4q0Sup9VKwRMDJyKiGoGBk5fKysqSLampqe4ukjFzNU4BAcDkycCNN8q3f/89sHWr94wkrgucvKn5kYjIizFw8lJxcXGyJTY21t1FMmYucIqMBIKCgHfflf6t6pVXvKfWSaNhjRMRUQ3CwIncx1zgVKeO9KRZq1bAs8/KXzt7Fnj7bWeWzHXU6son64iIyOMxcCL3qS5wAoCwMGDECKBLF/nr8+fX/Fqar74CuncHRo2SxqkiIiKPx8CJ3Cc62vRrusApIECqeXr3XfnreXmAJw7qaanz54FJk6R///1Xan4k+7HmjoicjIETuU+jRqZf09VGBQRIS7NmgEolT5OZ6byyOdu338przH77zX1l8RZCAFeueNdwFUTkcRg4kfskJJh+TVfj5O8PhIZKA0XGxcnT1OTAyWBKHADS4J5kO61WGhOLtU5E5EQMnMh9wsNN93Oquj02VgqevClwUppi5fx515fDm3BoByJyAQZO5F6map2qDo4ZEADUrSstVV286LRiOd21a8bbzp1zfTm8iVYrBU01/aEBcqyKCqCszN2lIC/CwIncKzFRebvhqOIhIUDz5vJtNbnG6fJl422WTnhsK2+viTl6FNi5Eygvd3dJyJMUFUlTNbmTENLYc540J6VaLd1DvWUwYRfilOzkXqYCJ6UmPMPO5BcuOL48rnLpkvE2e5vq1GrAx0daDOXnS18gDRrYdwxPNX8+8Nhj0hdTz57A9u3GDxNQ7XPwILBiBdC6NTB6tPJnwxVKSqRa5qAgqd+mu02YINXMxsUBSUnA3XcD8fHuLlWNwcCJ3MuSpjodwy99V9Y46Qaq1N30Cgrsuwkq1TjZGzhlZ0t9wSIi5NvLyyuHbhDCOwOKV16p/DX/999SzVPPnu4tU22kq1UxvAbd4fhx4KabKmsgg4KA4cPdU5bycukBF7Xa/YGTEMCyZfKnT//6i4GTFdhUR+5laVMdYBw4KdXaOEtBQWXwodUCubm2P/au0QBZWcbb7QmctFqguNi4OU4I4PBh6Vf33r3e2f9HqzV+P9eudU9ZzKkNTSIlJVLNpidYulTebPv00+4rS0mJVBZPaC7Pzja+dzVp4pai1FQMnMi97AmcsrJc92VUUVEZmJSVSb8ebQ1Crl5VvoHa0/RYUSEtVb8oCgqA558HevQAXn1VGoH9m29sP4anUhrawZW1alpt9UG0RiMF+qbS5ee7vx+OKSUllneuLi31nI7YX30lX8/Odk8fozNngGnTgA8+UP7BZKuSEtvufxkZ8nU/P+9twncSBk7kXtY01TVsKF/XDXgIAIWF0lN2GRnSjersWccOhKj7Qigrk4KTsjLbgzZTNUsXL9p2Y09PB268EWjbFpgyRfoiFwLo3x+YM0cKqHTmzbOpyB5NqcnWVb/shZCaBg8fNn/M8nLpGi0sVH69qEj6Yi8qkq6r7Gz3ByBaLbBkCfDEE8DKldX/UDh0SLq+Dh70jJpNpR9laWmV/8/Ls++HlxDVP4igVgP33CMNePvtt8CYMbYfz1Burm21e4aBU+PGgK+vQ4pUW7CPk5fKNvgVnuOp05OYaldXCpxiYqT+AVUDgcxMKaAqKKjsfKlSSV9QPj7Sa/Z0CFWrpZvd6tVAp07Sv35+0vaq5bBUWZn09JeS0lKp1iE21ro8X3tN+uIGpC+4Rx6RAtKUFOO0zn5yzx2UAidTA4yq1dJ15AhCAA8+CKxbJ11z8+cD48Yppy0vl5b8fOnBh6rjeAlRWauTkyNdr9euSdvj46UyFxRInwlX1qQtXy5dS4AUQPn7S7WWSg4fBrp2rXxAYcsW4PbbXVdWJYGBxtt27ZIeMpk1Swognn1W6hxti+Ji6e/UoIHpe8z27cCJE5XrO3ZIAY+56aYsodVK9wulc6yOYeBkqtafTGKNk5eKj4+XLUm23hyczdQNR+mG4OMD1K8v36arpSkpAYKDpWELgoOByEjpS8re0biXL5c6UpaXA//8A7zxhnTDNAzgLCGEFBgZ3riqsrafk1Yrla+qb78FTp9WTp+b69iaDN1j1vn5pmtTqnJG06pS4FS1/1tFBTBkiBR43H6748b/2rlTCpoA6X147TXlGsPiYuCzz4BFi6SyFhfLX9eNdh4ZKb2X169Lnfzz86XrLi9Pum5cPczCypWV/xcCeOghqSxK5syp/NtqtdK6uyk1i+3eDTz5pNR0/fXXwG232fa+bt8udTR/6aXKWm8lO3YYbzt50nzeQkgBtLnPilotLaWllpW3KsMfTwycrMbAiWoWw7b4s2eB//xHuol9/nnlzcbHRwqgcnLsCxS++06+vnAhsHWr9IWZlWV501pmpvTkSm6u9OVoirWBU2qq8baLF83XLJm70Velm8LEnPJy4NgxaRLmb74xnV6jkb78L1ww/fcoKDD/3piiFDhVfWpxyRJgzRrp///+C7z/vvXHULJqlXE50tON002cKH1Rz5kj1eBcvCj/stb1T/P3l2qjysqk60D3CPvVq9IXpKsDpz//NN72xhvKaRcskK//9JPjy2MtpcBpzx7pM6xz5Qqwfr11+V69Ctx3H/Dzz9IPq6eeMp3277+Ntx0/bj7/wkLp726uq4EucCovt7553/CHGzuGW41NdeR+oaGWt9UbBk4zZ1Z+Se7dK1XFJydLN57Bg6VfUzk5Uk2VYe1Wbq7UZGLu0emqfSJ0HntM+vfTT6WaHcP9i4qkWoO6daVj/vgjMHSo9IXYoYP5G5W1TWl//WW8rbzcfD6XLpmfJ1BHdx6GfcuqOnoUuPfeyicOz58HPvqo8nUhpAmMDx4E2reXmknUauMaRa1W+kJSqYCwMOUmqbIy5ZpIpcCpanD4yy/y1wxrAUpKpHytbdJVqtVbuFAaG6dPH6BjR+kaW7q08vWzZ4G33wZef12qYdI1xeXkSPv+8osUiAJAixZA376V0w09+qg0TZE7ff65VGNzww3Vp3Xn0Be6mjpDu3cbbzt0CBg2TPq/EFKQrzQlks6330qfC51165TPVa02HTiVlUnBcliY/LUffpDuLyUl0jXy0kvG+1+7Jl0L+/ZJPxjffdd8eQ0ZBk5Nm1q+L0kEeSUA1S5ZWVnuLqakUychpFtP5WLK5MnGac0tw4YJ8eefQuTkSEt2thBFRUKMGSNE48ZCPPqoEOXlpo/XsqX5/BctktIVF1fuk5srxOHD0vGEEOLGGy0v74svWv6+FRQI8cADxnnExAgxaJDpY6xbZ1n+ublCnD4tREWF8usnTggRHS3POzpaCK22Ms2cOZWv+fgIceedQhw5onwuR44IkZoqfy91ioqEOHtWuSwDBiifZ3m5VJbYWPn2oCAh1Gpp39JSIdLThSgstOw9qaiQ8tRqpffZ1HusUgnxzz9CrFmj/PratUIcOyad9+TJQgQGVn9tDBxoWRkd4fp10+WYPFme9to15XQXLpjOv+o14gznz1v+mXv55cr9rl8X4tIl83krXW8XL1a+npcnXa+7dysf75ZbhPj0UyF++UWer0Yj3ZN06fz8hEhLMz7+U0/J8/v1V8vfF61WiJAQ+f7btlm+PwkhhGBTnZfKysqSLalKTTqe4oMP5Ovmqr6tfWx25UqpWn3iROmX/rp1wOzZwOLFUu3IggXGfYR0tNrqa4COHavs1FteLjU1/fKL1DH7xAmpn8q//1peXkub6oSQasOUmlOuXpV+RZti6fhXujm+dM1vVfsxCQGMHSvVqFSVmyu9FxqNlPbNNytf02qBX3+VnjIyfALt+nXpF7tGY9wHCKh8Ik2pmc/UQKhZWVJTpuGDEaWl0pOXuuNev27Z01Gffw5MmiTVIpw6ZX74ACGAL74Afv9d+fVnn5WaLb/8Uqq5tKQ5+YcflAdOtdW1a/KOy1WZmzdx2TL538jUveX4ceVhO65fl/5mznzyzprH/qv2g9QNO2Ku+Ut37VSlq33UaKT3tKBAuTYYkLZPnizdl3R95ADpPa/6+VerpeujKiGk/nJVffyx6bIaunrV+PPFpjrruTtyI9fIysry3BonrVaIF16Qait69DD/S3XRIutqnCxZOndWPtaFC9XvO2SIVGtw8qQQV64I0by5/PUmTawrS3Jy9e9XWZkQffrYfr6vv179MVJShBg5UojnnxciK0v6G6WnS+cohBCnTpnOf/t2qbbtww9Np9mzp/JYJSVCHD8u5X/ypBBnzki/vnV++kmIdu2kv9MffxiXtX5908f49FPl19askY51551Sjee330p5VVRINW2GNSJvvlm5b0CAEI8/Xv37rFIJccMNpl9PSBAiKsq6v92CBdX/7SyxapVUowEIMXq08eu//GK+HF9/XZn2q6+U03zyifS3zM6uTJueLsTUqUJ88IFUM+MsGzda/p7ec4+0j1ot7bdzZ2WNpKHcXOnvqvR3OXdO+psC0n3szjurP/add1bmvWmT8ett2kg1V+npUpqzZ43TREYal7OoSPqsGl7He/bI9/X1NV2jTCbB3QUg1/DowEknN1eqYjdn507bAwZzS1W6m+b27dXv17lzZdPcF1/YX46EBHlZ8vONb2zz59t3jMceM/8eX7gghL9/Zfrnn5eatI4fFyIjQ0rzww+m858/X0rXpYvpNKtWSfkcPizEtGlCTJwoxIwZUrPB0aPSjV8IIS5fFiIsrHK/li2lL4PCQqmJqKJCagJUOsaPP0qBrdJrzz0nD2qDg4XYsEGIUaOEGD/eOHivrsnWUUvnzlIz3mefCXHrrcavDx9u/m9nCY1GiAYN5Pnu3StP8+WX5svZvbsU8AohxLPPmk4XFibEE09Iza/bt0tf8rrXpkxxXpOdNT+wOnSQrqMePaT1kBApgFLy88/Kefzf/wnx4IO2/c11Pv7YdJqAACmYXb7c+DU/v8q/hRDSe3rhgvTZMgxOV62S72t4vyGLsHM4eY6AAOnjbE737lKHWqWq+K5dpad+WrWSqqOnTpWahixRVCQNZZCfL3UqbdBA+QkpQ2lpldMpHDxo2bHMycyUqvt9faWqel1zUESE9N4UFZluArDUpUtSs0BkpLQYWr1a/nTcrFnSY/ybNkmDajZsWNmBWcmxY0C9esD+/abTnDsnNYPedJO86UClkjq8zpghdY7++mv5MAenTgHvvQccOAC0ayc9pWaqyefSJeWmTEBqrq2qpAS4//7K9awsYMMG6f+lpdJxHaFZM+WmHkB6zzZsqOykft990oCS77xTmeb336XztWdssiNHjJs316+XPj861TVR//OP1Aw+apTppjpA+tt9+aXUXLp5s/ypyS+/BF58URqAUUcI6ZoPD7dtjCIda5rqLl6U3nfduGfFxcCHHwJ33SVPV14ufQaU7N1r+b3GkO6hB1PNprpjv/gicMcdxq+p1dLxe/WS3r+PP5aum8RE4K23pAdSAgKktIYdwy15SISMuTtyI9eoETVOlpo4UflX2bRpUidN3XLhghAffSRE69bV/+rbuVNqUti1S4gDB6TajKrNM7rlttuMt33zjRBLlgjRu7djah10HU2LiqQO07rmsaIiqekjKckxx+ndW/5LVeeee8z/8v33XyEeecR0mrvuqv69ePppecdxw6VxY+m8ExPN5zNmjOnXHnrI9vdGpapsYjp40DHv9513Sk0luuYcw+XZZ+XX7+HDyrWeu3fb9/n56CPjPO+9V+pMP2WKEOHhxq8/+aQQcXHybdHRUhkNa6+sWd54Q17rlJ8vXfPVddCuzosvWlcOwyZ2QJ5fcbH02eva1THXQtXl2DHpGEr3FsPF1LUzc6ZUa/boo/Ltt9xSef8QQvo7Vn191Cj73udaip3DqeYZMkR5e5s28nWVShrp+Ndfpc7h5iQnSyMI33yzVKv1zTfGNU6PPCI9am/46O+ECcDo0dKgeNWZOlX+K++WW4xnSz9/Xqr1evll4JlnpMf5c3OlX5Hz5xv/wo+Pt62D5/btxh1Nq1NeLnV6NlfjtGlT9e+FbmocU86fl2ozqqv5WLzY9GvLl5vf1xwhKjt2O+rBihtvlIYWMDUW0tChlTWeBQXSKPidOxs/Lm44vIISIUx3OP/tN+NtO3dK86l98onyWFotWwL//a98W26u9Bi/qc75lli0qLJGMSensvN7fr7y4I4FBZbN6WfpWGU6SsOO6MaEE0KqwVqyxDG1yoZ0HcvN1TjpmOq0v2OHdA8yHE/rr7+kIRiEkNZ37pS/zo7htnF35Eau4VU1TuXlyr+6/v1X6kSp+8Welib9Ij56VIgVK6z7FejrK0S9evJtM2ZIj+c3bWr7r8tVq4T4+28hJk0SYsIEKT/DDuSrVgkxdmzlenCw6V+avr5SDZm5Ph3Nmpl+rU4d4/e3RYvqz6Nqv6PqlqgoIV59Vb6tQwfloRQ8aRk3Tno/XntN+X0HpFqX+fOljvTV5bdpk1RzkZ5u3Pfq5pulzsWpqVJN6cmTUq2nEMY1rN27m/98qNVSLcPJk1IfsbKyytfKyowfR7dkWbtWqhnq18/x7/P770udoHXrCQlSnyHDWqeSEiE2bxbi0CH5wwNK+ve3v1y6fm4bNgjRrZvt+fj5CTF0qOnX58yRHjBx1nV8111S7WnVe4pu+eYb8+8jKYK7C0Cu4VWBkxBC3Hef/AbQrp0ULOm+eI4dkzozZ2dLX1alpfbfgFaskDqC29Mkd/q01BSha4rJyzPO7803lZ/cUVratJGa8PbvN51mxAjzeZSWSl+0paXSF6suKHDEEhAgxNKlUkftqtujooTo2FG+zZ6A1BlLo0ZSsGA4JtbkyVJQcuiQvAPu6dNCDB6snFdwcOW4XtevS03DnTtLr4WHSwHBqVOVTxSWlVU2Yf30kzwvlUqIzEzlz0VpqfSAxeHDUnkOH5Y66uvy2rbNtvdC13k8I8P8+FXmFpVKaqK19EnTwYOlpjshpM+xbry3yEj5U5lKdO+tPcuuXY554KNTJ/NPmD71lPFDLyqVEFu2VP/ZtfR9N9UMyDGcbMKmOqqZpk2TN5mNGiV1KNVqpWr+wECp02lsrDT1SmCg/TOTN28udaZu2dK2/aOjpQ7AKpV02wKkTuBVO8cCUnOc7vXq3Hij1Km9eXPTabp3Nz+C808/SU1rublS86ThGEvVCQtTnpQZkP5OvXtLHb2runatcmJinaeftu641lKar9FUsxkgjbN08qRxU12nTtLDA8HBUidtXcfbunVNXxvt2lWOEh0WJo2gvnCh1Gy2f7/0emmp9BCALk/d3+yOO+QjTAshdeA3VFgoNeVkZ0t/j8xMKf+UFKmJdf9+aSRyW+ialxMSpCarqCjL9uvWDRg4ELj7bmDFCqnJXDdxcHXWrpU+11qtNFq5rpksP9/83w2wvqlOyfnz0oMRSsaOtXzE7WHDgAEDpM+pklOnpKlgqmrcWOo2UPWBBVsJAfzxh/H2hg2lLgpkPXdHbuQaXlfjJIQQ330njeL7yivSL1O1WmqeO3NGudPzwYNSDYitv9x0NQazZlWftlcv423h4VIHzlOnpBqxo0el2rCpU20v0yefVJ5f3brKaX74QYj4+Orz6t/f+iZNQBp24OabjbfHxgqxb5/U5FRcXH0+v/1m/bhXli716hk3Z/73v9LfoOoj8obLBx8Y18Dt2iW930VFUs1P1TF/vvtOOZ9Jk+SdoIuKpBqho0elGsizZ+XDMBgaPlye3y23yF8vLZU6WuuaUA2v86FDba9J9PWVl72sTPpbVR3lOihIqiFauLByW2SkVJOiezT+yBGpNuzqVdPXqtIybJjxtsBA6TOuNJxBcXHlGFW6xZrj6RZTwyzMmiWNmq/UHKhSSfejqp+NI0ekv8/69aZnETAci+z226Wax+xs6b1V2sdUU3fz5sadxA2Xbt2kvwnZBO4uALmGVwZOZWVSEHLqVOWXV2GhdJMyZds2qa3/k0+Mv6Sjo4V46SXlG014eOVN2pLB9aZNM97Wv7+0//Xr0pfmyZPSOZgapNGSpeoTVs88o5zm4EHjZjFTi+EUKpYsI0cqT/Hy1FNSgKgLYs0FbyqV1FdL6UtSl5ep8ZosWYYOla6ROXOkfjrTp0t9itRq6QutQQPpyzUiQr6fUpl1zUdKdu1SPv6SJcZpKyqkQOn8eal/Xlqa6b47a9fK8/PxkQYmFULKw9R4VdYupgbsNFRYKJX5o4+kgPTEicofAnv2SEHcli3SZ7G8XGqazs+vPD9T40R16GA6UDBcvv22cjolIaTP5/HjUp8pw7SGTfuWLEqDk6alSYHQ5cvKT/cOGiT9XU+ckMp36JB8LKWcHCGWLav+2E8+WblPcrLycQoLlfvWLVpkfgy6iAgpaMrNNX0dk1kKnwhyhvLycvHDDz+I119/Xdx7772iTZs2Ij4+Xvj7+4uwsDDRqlUrMXLkSPHDDz8IrRMGhfPKwEkIqTbj8mXr9snJkW7kzz0nv6F89510w1N63L9Dh8r9zY2arVuWLjWuSao6R1xZmVR2rdb8YJLVLVWDxPJy47n8YmKkNH37OuaLVWl56y3pi9Jw+7ZtUkdo3fVsbs6+evWkL6R33jF+LTBQ+sL69FOplsPUvG7mvnA//bTyfTp/Xgpaq9ZKVlRIAe3SpebPtWFD89dWdrbyfkpzjlU9dmZmZQCgpKjIuFP3vHlS8GTNXIhVlzfflAfUcXHSvGeG6aKilMtUWFjZj+rUKek91Q3WevmyfMRwpXNu21Z+nLZtpUDbkhrdqkv37lIA1bOn6b6BStentct990nX8/Hj0rWjNGDlv/9K56fVSvcYXQd/nevXzfdH1C0ff1y5j2HZIyKkgEwI6T5SdYTyO+6Qgle12nRftP79pc+auR8AZBbcXYDa4vz588IwcDG1JCcni0xTnT9t5LWBkxDWjz6sVks37mvXpKkzuneXxkEpLJR+EaenS50yq96E58yp3N/UU31Vl0OHpPyHDJE6GT/xhOnaBEtupErLHXcon9u8edIXYuvWUlAmhPI4NZYsPXtKtTXvvSd1cFZKs26d8YSmAwbIJzoWwnytyE03SV++f/xh/FqXLpVNWJmZ0uS5SnlMmSLEww8rv3b8eGU58vJMT/dRVGQ8KXB173lVWq3yftU9Babb1xzDkalvvtn2v+uWLdIXZ1qaNOXKwIFSoHv8uNQ8WTXtSy+ZLlNZmfQ3SU2tnBZECOk6rO6ct2ypvKaCgqSn165ckYIPR3Tu1i3h4dUHxJYsM2ZI17Tuab+LF+VNok8+KX+CUenvWVEhXXt16pg/VkpK5T4XLggRGiptV6mkv0/VoL+sTJr2Zs4ceYBu6kGF116TAlRLJ7YmI3B3AWqL8+fPi+joaDFo0CDx9ttvi2XLlok//vhDHDhwQGzbtk18/PHHon379kIX1LRp00aUKPXTsZFXB0720GqlJ4UuXZLf6EpLpRv7c89JgYjh3FXVPX6uu4FqNNKvcnPvtalaCsOlfn1pYL+4OCkw+ucf5fwyMyv7lOgCDksel1davvtOCmgOHZK+XA1f9/OrrPF77z3pUfJbb5WObdhn5/nnTR9n+HApnwsXjKc3+fzzyr9NQYFUY6RU63T0qPR3Mxz4snVr64JrpYFPdcuUKdXvr7SfI3z7bfV/r5AQqaZz6VIp6FVK07NnZW2nEJUDraamSoFuaakQd98tpe3Y0fzckUJUToFTUGDd+ZSUSDVc778vTWWSl1c5b6FSoGPrQJvNm1s2fVJ1y7Ztxv0n9+2Tpuh59dXKgWst0bCh6eMMHWp8vZ49K12XCxcq1+Tl5krXf9VaJFNPBKakSO+1qfn4qFoMnFxEo9EITTW/wCoqKsSAAQP0gc1nn33msOMzcDKjqEiqRTKkVktf5ko3xJwcKYgZMUL6Qqv6y7N3b3naK1fMT2iq1ZpuZlqyRBrd9777pC+ZoiLpC07Xn0SJRiMFamlplU15W7bY9mVx4IB0oz5/Xlpuu02ax87PT+oTVLVJQQgp7ZEjUrB4+rS81sHcXFxTp0p/g7IyqQZL17m3Z0/5yMdarfR3mTRJvv/dd1cGaYWFUgfdxEQh2reXmnGskZNT+QvfcNm8ufr9DcfLee89645vysmT5v9WDRsKsXWr1Hfr4kWpBkcp3UcfyfOtqJD+ttevV25Tq6VaCXNNjI6QlSUF5cePV16reXnS3/Ctt6TrIDJS+nv++69tffAmT1aeHNeapXFj6VpWCsCvXpU+j9bU4Bh29gekYSe2bZNf74bHOX1a+V5VUSHdq6rOa5mRoXwuDJjsxsDJw+zcuVMf2AwZMsRh+TJwspFWa7rJQTfx7eHD0jgtzZpJ/U327ZOnM3XDrUppEtmbbpK+RHRj8+gCpcxMqRbI3KzmWq38l7FGI1Xxd+smTZfy8suVxwkIUJ7+RKWS8tBoKm+2uhqBEyekAKnql60QUnl1tU2GXwDr15v+YvryS3naY8ekJ/wOHVKuyUhLq+zIHRMjNfHp/k4lJdL+ugDOltnfDfu/+fpKHcotcehQ5VNcN9xgvq+PNbRa5Q7LuuXPP6XzPnxYCrLOnVNOp5uo2TBvQ7pg2Zl0D3hU7QtXtUynT0udzY8elQJac1P9AFKN59690hOTfftKzZBZWdI1UF1wZPgkXtVl9GjT55CfL73XljTH6nz+uTz/sWOlv8vhw8b9oqq+V9Y2rxmeR2ysdfuTIk7y62EiIiL0/79+/bobS0IApLF0TI2BFBAAxMVJU6ZMnCiNV6PVGk9jYMmErErjzowaJY394u8vjbmiK0edOlKehlO/GJY7KEhehhdfBB57TJqKIzxcGofowAFg0CBpfJ8GDaQpanST5t58szwPQFqvU0eaFDckxHgi1tBQaawfPz/j1xITTZfXcEychg2liYaDgpQne23WTJqi4uefgdatpfGTdO+z7tiBgdJ4Q+beJ1PeeAPYulUaO6hhQ2m6kQEDLNu3fXupbDt3SuNrxcRYf3wlKpU0Jtfmzcav1a8vnau/v3RNXrkijd308MPA0qWV6bp0Uf47KF3joaGV0444S0CANL6Zj49xGVQqaWwsQBqPLTIS+L//k8awKi6WzjcionJanshIaTqgkBDpmh40SLrGY2OVzy85Gfj2W2DfPun9Cw+X3t+qE1zr3Hef6XMIDZWuU2smXh47Vhpf6+efgR49pDHMSkula9XU9RoQUDlmmKWef14+FtXUqdbtT4pqdOCUnp6OgwcPIjMzE4WFhahfvz4SExORnJwMf8P5v2qI7777Tv//1q1bu7EkVC2VSrrp6/j4SDd0W76oH3hAGliwqoEDK+exq3rjDw6WFltEREi/PX18pJvq+fPS9rAwaQ7ABQukG21EhDQvnxLdl5kSPz/pC0hJmzbS+5Wba/yaYbDp7y8tusBRSWysNNdfWZl8cEE/P2mAST8/8wN/mhMZKQU+O3dKgxGWlysHcOb2b99ePpClI9x0k3Lg1KePdM66YFYIac6755+XBpIsKZHSvfCC5ceKjJTycbaqnyFDYWHygKF1a2lA0l9+ka6nZs2kOQnPnJHmauvQQbpeKiqk868akN10kzRvm84zz0jvmVotBT4FBdKPB8P5EaOjzQ9E6esrLdYIDpaCtsxMaY6+evWk97uiwvbPtpLXX5cGt927V/qsTJ7suLxrMYcFTmfOnMGePXuwd+9e7NmzB/v370dBlckiExMTkZGR4ZBjrV69GrNmzUJKSori69HR0Rg+fDj++9//IjY21iHHdBatVousrCwcP34cX3/9NZYtWwYACAgIwBNPPOHm0pFVQkJMjw5cnXvukQdOL71k+ejM1qhag1b1Zh8UJG0fPFga7VkI8zVEtggMlGpulG7eVSc+BqQvv8BA6de8Kb6+lUGqYVDjiB9OQUFSjVF+vvTFZiogNCUy0rYg2pzu3ZW333GH/HqJjpbeuxYtpIlely4FOnaU/r6WsqYGxZkMa1kaNwb695eCjLg46bOimxRZlzYoSAq6dLWngFSLOGiQFGi3bi0FQ/7+0t81N1f6W9Wvbxw4Pfigcc2rowQFSTWSsbHS+21NcG6JyEhg1y7g+HHpGM46j9rGnna+rVu3in79+ono6Gij/jOGS2Jiot3tigUFBWLEiBHVHku31K1bV2zatMnu4zpadna22XJHRkaKn3/+2aHHZB8nD6dWS+O1tGsnjfp79qz1wyxYS9dH6+TJyg6nGo305JC5zuf20GiMJ0ytX185bV6eckfYqjIzq3/qyx65uVI/H0/pUJuVpdwHp+pQAEouXFDuR1RT5eTY1nfs9GnpgYuqw73oHqbIyFAe3mHHDseV25BGY13fKFvp5uskh7Dr59DBgwexZcsWe7KwmEajwfDhw/HLL7/ItsfFxaFz586IjIxEWloaDhw4APG/6uUrV67ggQcewG+//YZevXq5pJz2UKlUeO655/Dyyy+jrrnmEPI+vr7A9OlSH6S8PKnGwJFNPEp0/SmCgipraHx8pD5MWq1jam0M+fgAy5ZJ/TquXpW2jRihnDYysvr8LEljjzp1pGN4Su1LXJzydsOmTkNRUVKTlLOvKVcx17xnTvPmxjWZPj5SbYxGI9Uuvf++fJ+bb7a9nNVx1XXlyOY/ck4fp8DAQDRq1AhpaWkOy3PatGmyoMnf3x+zZs3C448/joAqVbmpqamYMGGCvhmvrKwMAwcOxOHDh1G/fn2zx1i/fj0KCwvtLmtycjKaNWtm8vWoqCgc/t8EpxqNBlevXkVKSgq++uorfPzxxzhx4gS+/PJLNGrUyO6yUA0TGCg1Hdja5GcNX1+pacOwOUzXJOXoZiadxERg3TppiYoCnn3W9ryc/T6ZezjAXUaMkPr16Dz4YPX7uOJ6ciV7/iaRkcZNYiqVdL0/9JA8cPriC+v7L5H3s6e6avbs2cLf31906tRJTJgwQcybN0/s27dPlJeXi61btzqsqS4tLU34+/vL8lu/fr3J9MXFxaJHjx6y9BMnTqz2OImJiRY3A5pbFi5caNN5FhQUiH79+umbGVNTU23KRwmb6mqIkhLnNJGZO54tj+vbSzd+z7FjbEKw1s6d8kfod+50d4m8y++/S2Onvf++9YN6Uq1g10/KMWPG4IknnkCQkzucvfnmm6io8ojo2LFj8cADD5hMHxwcjEWLFqF9+/YoLy8HAMyfPx8vv/yy2ZogdwsLC8OSJUvQpEkTXLlyBZMmTcK2bdvcXSxyJVd33nRXZ9HISKlJEuAvemv16CE9KbVqlfSklKkO42Sb224D2rUDrl9nZ2pSZFfgFOWMp34MlJSUYPXq1bJtUy0Yi6JVq1YYOHAgVq5cCQBQq9VYtmwZXnvtNZP7HDp0CNqqT2HYKMSOavH4+Hj06tULv/76K/78809cunSp2iZGohonIEBqpissZOBki169pKfLVCq+f87g7y81YTuruZpqNI+/KjZv3ozi4mL9eo8ePSwe32jcuHH6wAkA1q5dazZwqjr4pDtVHUIhIyODgRN5pzp1pL4mntaHqKYwN1QD2SciwvrhJ6jW8JBHRUzbtGmTbL1v374W79u7d2/4VfnFcODAAVxRGqHZw1y8eFH//3B+eMlbmRssk6oXFeWcsb5IPkYYkQGPD5yOHDkiW+/Ro4fF+4aGhqJ9+/aybUePHnVIuZwlIyND/0RgSEgImjdv7uYSEZFHsmXEaiKym8cHTseOHZOtt2jRwqr9DQOP1NRUu8tki6VLl+KqbtwaE7KzszFs2DB9R/iHHnoIwRx/g4iIyGN4dF1kbm4ucg3mtUownJqhGobpT506ZXe5bPH1119jwoQJuPfee9G3b18kJSUhKioKarUaFy9exJ9//onFixfj2rVrAKQAcebMmWbzzM7Otvj4OTk5dpWfiIiIPDxwytM9rvw/ISEhCLWyQ2R8fLxsPT8/395i2ay0tBRr1qzBmjVrzKa75557MH/+/Grn2TM8NyIiInIujw6cDEfxtqXZynCfqhMPu9LSpUuxbds2/Pnnnzh48CCuXLmC7OxsaDQaREZGokWLFrjpppswYsQI3HTTTW4pIxEREZlXowInWwbaNAycHDGlii0aNmyIhx9+GA8//LBbjk9ERET28/jO4VWpbBjvxZZ9iIiIiJR4dI1TWFiYbL2kpMTqPAz3McyzJsvKyjL5mmFn8NzcXPTq1cvZRSIiIvJqDJxqsLi4OJOvseM4ERGR43l0U11kZKRsvbi4GEVFRVblYVgrU6dOHXuLRURERLWURwdOMTExRhMJnzt3zqo8zp49K1tv2bKl3eUiIiKi2smjAycAaNOmjWz99OnTVu1/5swZs/kRERERWcrjA6d27drJ1nXzuFmiqKgIhw4dMpuft8rKypIt7ppqhoiIyJt4fOB01113yda3bdtm8b7bt2+HWq3Wr3fu3Bl169Z1VNGIiIiolvHop+oAoH///ggODtY/HZeSkoLjx4+jdevW1e67aNEi2fqgQYOcUUSPxKfqiIiIHM/ja5xCQkIwZMgQ2bb33nuv2v1OnjyJdevW6df9/PwwcuRIh5ePiIiIag+Pr3ECgOnTp2P58uWoqKgAINUkDRo0CPfff79i+tLSUowbNw7l5eX6bePHj0fz5s1dUt6awnCQTCIiIlvFxMTAx8fj62PsphJCCHsyuHDhgqwfkc6uXbvw0EMP6dcbNmyIHTt2KOYRFhaG2NhYs8d56aWX8OGHH+rX/f39MWvWLDz++OMICAjQbz927BgmTJiAnTt36rfFxMTg8OHDqF+/vsXnVdNxqhkiInKlrKwsswMzewu7A6cmTZoYjZVkrTFjxhj1RzKk0WgwYMAAbNy4UbY9Pj4eXbp0QXh4OM6cOYP9+/ej6ikFBATgt99+Q+/eve0qY02TnZ0tWz958iSnXCEiIqepLYFTjWiqAwBfX1+sXLkSEyZMwIoVK/Tbs7KysGnTJsV94uPjsXjx4loXNAHG07GwWY6IiMh+NaoxMiwsDMuXL8eqVatw8803m0wXHR2NSZMm4ciRI0bDGRARERHZyu6mOndKT0/H/v37kZmZiaKiItSrVw+JiYno2bOnrN8TAWq1GqdOnZJti46OtrojX05ODpKSkmTbUlNTq+2jRlST8Donb+eMa7y2dA6vMU11Spo2bYqmTZu6uxg1gp+fn9Omm4mNja0V7dpUu/E6J2/Ha9wy3h8aEhERETkIAyciIiIiCzFwIiIiIrIQAyciIiIiCzFwIiIiIrIQAyciIiIiCzFwIiIiIrIQAyciIiIiCzFwIiIiIrIQAyciIiIiC9XoueqIiIiIXIk1TkREREQWYuBEREREZCEGTkREREQWYuBEREREZCE/dxeAap709HQcPHgQmZmZKCwsRP369ZGYmIjk5GT4+/u7u3hEAICKigr8/fffOHfuHC5duoSwsDA0aNAAnTt3RpMmTRx6LFd9Jlx5TuRavF7t49LvJUFkoVWrVokePXoIAIpLdHS0mDRpksjOznZ3UclDvPHGGyavF0uWMWPGWH3MrKwsMWnSJBEdHW0y3+TkZLF69Wq7z89VnwlXnhNJ0tLSxPLly8WLL74o+vTpI8LDw2Xvd2JiokOOw+u1ZpxTVQycqFoFBQVixIgRFn/Z1a1bV2zatMndxSYP4OrA6ZdffhHx8fEW5//www+LwsJCq8/LlZ8JV50TCbF161bRr18/s1/4jgyceL3WjHMyxMCJzFKr1eKee+4xugjj4uJEv379xNChQ0WXLl2ESqWSvR4YGCi2b9/u7uKTm7kycNq6dasICAiQ7a9SqUTXrl3F0KFDxZ133iliY2ONjjFgwACh0WgsPo4rPxOuOieSzJ492+Jr097AiddrzTgnJQycyKwXX3xRduH5+/uLuXPnirKyMlm6o0ePGlWXxsTEiMzMTDeVnDyBYeD0/fffi/T0dIsXS6vXz58/L6KiomTH6tmzp0hNTZWlKy0tFR9//LHw9/eXpf2///s/i8/JVZ8JV54TSUwFToGBgaJ58+YOC5x4vdaMczKFgROZlJaWZnRxr1+/3mT64uJio4t04sSJLiwxeRrDwGnr1q1OOc6jjz4qO05ycrIoKSkxmX7dunVGX4wZGRnVHseVnwlXnRNVmj17tvD39xedOnUSEyZMEPPmzRP79u0T5eXlYuvWrQ4LnHi91oxzMoWBE5n0yCOPyC62sWPHVrvPiRMnZFW1fn5+Ii0tzQWlJU/kisDp5MmTwtfXV3+MgIAAcfLkyWr3GzNmjKxs48aNq3YfV30mXHlOVCk3N9fkl72jAiderzXjnMxh4ESKiouLRUhIiOwCPXbsmEX7Dhs2TLbfjBkznFxa8lSuCJymT58uO8aIESMs2i81NVW2X2hoqNlfyK78TLjqnMhyjgqceL3WjHMyhwNgkqLNmzejuLhYv96jRw+0bt3aon3HjRsnW1+7dq1Dy0ZU1bp162TrhtefKW3atMFNN92kXy8qKsKWLVtMpnflZ8JV50Sux+u1kiefkzkMnEjRpk2bZOt9+/a1eN/evXvDz69ybNUDBw7gypUrjioakd7ly5fx77//6tf9/PzQs2dPi/c3vK43btxoMq2rPhOuPCdyLV6vxjzxnKrDwIkUHTlyRLbeo0cPi/cNDQ1F+/btZduOHj3qkHIRVWV4nXbo0AGhoaEW75+cnCxbN3eduuoz4cpzItfi9WrME8+pOgycSNGxY8dk6y1atLBq/+bNm8vWU1NT7S4T1Xzz5s3DHXfcgYYNGyIoKAjh4eFo0qQJ+vTpg1dffRXbt2+3Kj/D68qZ16mrPhOuPCdyLV6vth8H8JzvJQZOZCQ3Nxe5ubmybQkJCVblYZj+1KlTdpeLar7ly5fj999/R2ZmJsrKylBYWIizZ8/ir7/+wjvvvINbbrkFN954I3777TeL8jt9+rRs3drrNDExUbZ+9epVXLt2zSidKz8Trjoncj1er8Y88Zyqw8CJjOTl5cnWQ0JCrKp6BYD4+HjZen5+vr3Folpi79696NevH1599VUIIcymNbxWDa+76oSFhSEoKEi2TeladeVnwlXnRK7H69WYJ55TdfyqT0K1TWFhoWw9ODjY6jwM9ykoKLCrTFSzNWzYEPfccw+6d++ONm3aIDo6Gj4+Prh69Sr279+Pn376CZs3b9anF0LgnXfegVarxbvvvmsyX0ddq6Wlpfp1pWvVlZ8JV50TuR6vV9PH8qRzqg4DJzJieIEa/hqwhOEFapgn1Q7du3fH5s2bceedd0KlUimmSU5OxuTJk7F3716MHDlSVn0+c+ZM3HzzzXjggQcU93XUtVq1aUDpWnXlZ8JV50Sux+vV9LE86Zyqw6Y6qpapLzxH70Pe55577kG/fv0suh66deuGXbt2oVWrVrLt06ZNg0ajseh4rrpWXfmZ4OfPe3nytefp16s7PxcMnMhIWFiYbL2kpMTqPAz3McyTSEl0dDS+//572Q3u+PHj2Lp1q2J6V12rrvxM8PPnvXi92n4sT/pcMHAiI550gVLt06VLF/Tr10+2zXDgOx1vvGnz8+e9eL3afixP+lwwcCIjkZGRsvXi4mIUFRVZlUdWVpZsvU6dOvYWi2qRu+66S7Z+6NAhxXSG12p2drZVxyksLDS6mSpdq678TLjqnMj1eL0a88Rzqg4DJzISExODqKgo2bZz585ZlcfZs2dl6y1btrS7XFR7NGnSRLZu6mZseF0ZXnfVMUwfHR1tdO0Drv1MuOqcyPV4vVZ/HE84p+owcCJFbdq0ka0bDnJWnTNnzpjNj8gcw6dfTFXLO/o6TUpKMpnWVZ8JV54TuRav1+qP4wnnVB0GTqSoXbt2svWUlBSL9y0qKjJqWjHMj8icnJwc2XpsbKxiOsPr6tChQ7LZ06vz999/m83P3GvO+ky48pzItXi9GvPEc6oOAydSZNjHZNu2bRbvu337dqjVav16586dUbduXUcVjWqB3bt3y9YbNGigmK5+/fro0KGDfl2tVmPHjh0WH8fwur777rtNpnXVZ8KV50SuxevVmCeeU3UYOJGi/v37y5pLUlJScPz4cYv2XbRokWx90KBBjiwaebnS0lKsXbtWtq1v374m0xteXwsXLrToOMePH5cFaKGhoUZP81Xlys+Eq86JXI/XayVPPiezBJEJo0ePFgD0y9ixY6vd58SJEyIgIEC/j5+fnzh9+rQLSkveYvr06bLrztfXV2RkZJhMf/LkSeHr66tPHxAQIE6ePFntccaOHSs7zrhx46rdx1WfCVeeE1lm69atsvc2MTHRpnx4vdaMczKHgROZlJaWJvz9/WUX6Q8//GAyfUlJiUhOTpalnzhxogtLTJ7k22+/FZcvX7Zqn6+++kqoVCrZNTR+/Phq93v00Udl+yQnJ4uSkhKT6devXy9LHxAQ8P/t3XtQVPX7B/D3skmssNy8JhHesRwpFJUUA4aStPKSaWgGmoU2NeaYo+no0AVxNG0ojS6TUqbSaGpTauG1vKabpqIZaJABChISuCLCLp/fH7+pr+s5wDl7lt1F3q+ZM6PP7nnOc/acZZ85l89ptDn7lzO/E85aJ1LGUY2TENxfW8o6NYSNEzVqzpw5NjtcmzZtxMqVK8XNmzdt3vfbb79Jds527dqJS5cuuahycrXo6GhhMBhEYmKi2LZtmzCbzQ2+12QyibFjx9rsPwBEUFCQuHz5cpPLKiwsFAEBATbzDh06VJw7d87mfTU1NeKDDz6Q/OGdP3++4vVy1nfCmetE/1NYWCgKCgokU1ZWlmTflHtfQUGBKCsra3IZ3F/df50awsaJGmWxWMSIESMkP2gdO3YUjz/+uBg/frwYMGCA5CiBp6en2L9/v6vLJxeKjo622Sc8PDxEaGioiI+PFxMmTBATJ04Uw4cPF506dZLsXwBEYGCgyMnJUby8ffv22RyOByB0Op2IiIgQEyZMEPHx8aJDhw6S5Tz55JPCYrEoXo4zvxPOWif6n5CQENn9Uc2UlJTU5HK4v7aMdZLDxomadO3aNfHss88q/qPRsWNH8f3337u6bHKx2xsnNVNcXJwoLCxUvczt27fL/mFuaJo4cWKjR8Ia4szvhLPWif6fsxonIbi/tpR1uh0bJ1Js06ZNIjIyssEdMzAwULz88sviypUrri6V3MCWLVvEpEmTFP8QeXt7i7Fjx4rdu3drWm5paamYMWOG5LTBrVNkZKT4+uuvNa+js74Tzlyn1s6ZjZMQ3F9byjrdSieEECBSoaCgACdOnMClS5dw/fp1dO7cGSEhIRg6dCg8PT1dXR65oX/++Qdnz55FYWEhSktLUV1djfr6evj7+yMgIAD3338/wsLCoNfrHbbM2tpaHDp0CBcvXkRJSQm8vb0RFBSE8PBwdOvWzWHLAZz3nXDmOpFzcX/Vxpm/S2yciIiIiBTiAJhERERECrFxIiIiIlKIjRMRERGRQmyciIiIiBRi40RERESkEBsnIiIiIoXYOBEREREpxMaJiIiISCE2TkREREQKsXEiIiIiUoiNExEREZFCbJyIiIiIFGLjRERERKQQGyciohbk888/h06ns5mmTJni6rKIWg02TkREREQKsXEiaiZvvvmm5MiATqdzdVlERKTBXa4ugIikiouLkZOTYxMLCgpCv379XFQRqcVtSHRnYuNE5IZ27dqFqVOn2sSSkpLw+eefu6YgUo3bkOjOxFN1RERERArxiBMRUQsyZcoU3kVH5EI84kRERESkEBsnIiIiIoXYOBEREREpxMaJiIiISCFeHE5ENsrKynDy5En8+eefqKiogMViQfv27dGxY0f06tULffv2dXpNubm5OHv2LAoLC2E2m6HX6xEYGIj4+HiEhIQoylFfX4+CggL8/vvvuHTpEqqqqlBTUwM/Pz8EBgaiU6dOGDhwIHx9fZt5bdxbXl4ezpw5g6KiIpjNZuh0OhiNRoSEhCAsLEzx5+1IJSUlOH78OPLz81FVVQUfHx+0b98evXr1QkREBDw8eAyAnEgQUbNISUkRACRTQ+Tea++kVklJiXj77bdF3759m8wdHBwsZsyYIfLy8uz+bDIzMyV5k5KSbN5TVFQk5s2bJ4KCghqsJTMzs9HlmEwmkZqaKuLi4oTBYGhy3Tw8PER4eLh45513xNWrV1WvlzO2oZLPTq1z586JV155RXTq1KnJurp16ybmz58vioqK7F5eQUGBJG9ISIjNe6xWq9iwYYOIjIwUOp2uwXoCAwNFcnKypnqI1GDjRNRMWkLjdP36dTFv3jxx9913q15GmzZtxMyZM0VlZaXqz6apH//09HTh7e3dZA0NNU6pqamiR48emj5Do9EoUlNTRX19veL1ammNU0VFhZg2bZrw8PBQXZ+np6dYsGCBqKmpUb3cphqn8+fPi8jISFX1GAwG8emnn9r1ORCpweObRK1Ubm4uIiIisHTpUty8eVP1/HV1dfjggw/wyCOPoKSkxCE1CSHw4osvYtasWbh+/brdeT766CP88ccfmmq5du0aFi5ciDFjxuDatWuacrmj3377DQMGDMDq1atRX1+vev7a2lqkpaVh2LBhuHz5ssPqOnz4MAYPHoyff/5Z1Xw3btxAcnIyVqxY4bBaiOTwGieiVujkyZN49NFHUV5e3uB7goOD0blzZxiNRpSXlyM/P1+2gTh16hSGDBmCgwcPokuXLprqWrhwIVavXi2Jd+nSBffccw+MRiNKSkpQWFhoV2Ol1+sREhICf39/+Pn5wWq1orKyEhcuXGgw37fffovExERs3bpV9fLcVW5uLmJiYlBWVib7uk6nQ9euXdGlSxdYrVYUFxejsLBQ9r0mkwmxsbE4cOAAOnTooKmuM2fOYOTIkaisrLSJBwQE4L777kNgYCCuXbuG/Px8XL16VTbH3LlzMXToUERGRmqqhaghbJyI3MSRI0f++/f27duRmppq8/rIkSOxaNEizcu5evUqRo0aJds09ezZE7Nnz8aoUaMQFBRk81pdXR3279+PtLQ07N271+a1goICJCUlYefOndDpdHbVdezYMeTm5v73f6PRiDlz5iAhIQG9e/e2ea/VasWPP/4Io9HYaE5PT0/ExMRg9OjRiIyMRN++fXH33XdL3ldfX4+cnBysX78en3zyCaqqqmxe/+abb7Bq1Sq8+uqrjS7PWdtQi5s3b2LChAmyTZPRaMSCBQswefJk3HvvvTav5eXlYc2aNXjvvfdQV1dn81pubi5eeOEFfPfdd3bXVVNTg2eeeea/pkmv12PKlClITk6WXABeX1+PAwcOYMGCBTh8+LBNnvr6ekybNg1nzpyxe18kapSrzxUS3anUXuN0q+a4APhfTz31lCS3TqcT8+fPFxaLRVGOjIwModfrJXnS09MVzS+3frdOAwcOFJcvX7Z7HSMjI8XSpUtFRUWF6nmLiopEdHS0pKZ27dqJ6upqxXmaaxtqzTt79mzZz3zAgAHir7/+anL+U6dOifvuu082R0ZGhqIa5K5xunUKCgoSx48fbzKPxWIRzz33nGyOPXv2KKqFSC1e40TUiqxfv172qMBHH32EtLQ06PV6RXlefvllZGRkSOKLFy9GTU2NphrDwsKwZ88edO7c2e4chw8fxty5c+Hv76963qCgIOzYsQODBg2yiZeXl2P9+vV21+QOCgoKkJ6eLon36dMH2dnZCA4ObjJHWFgYdu7cifbt20temz9/vqZr0wCgXbt2OHToEPr379/ke/V6PT777DP06NFD8lpmZqamOogawsaJqBVZtmyZJPbSSy9h+vTpqnMlJydj9OjRNrGysjJs2LDB7vruuusuZGZmNnkKrilaT9G0bdsWn376qSS+bt06TXldbdWqVZILwT08PLB27Vq0a9dOcZ7Q0FC8//77knhlZSW++OILTTWuXr1a1VhRXl5emDdvniR+4MABTXUQNYSNE1ErsXPnTpw+fdomZjQasXjxYrtzpqSkSGJr1qyxO9+YMWMUHWlwhgcffBAPPfSQTcxkMsFisbimII1qampkt01iYiIGDhyoOt+kSZMwZMgQSXzVqlV21QcAERERkmZcifHjx0ua5YsXLzZ68wORvdg4EbUSWVlZklhCQoKmO6HCw8PxwAMP2MRMJpPdp+teeOEFu2tpDrefrquurkZOTo6LqtHm6NGj+OeffyTx5ORku3PKHak8d+5cg3fgNeXFF1+0az5/f3/Z03V5eXl25SNqDBsnolZi//79kti4ceM05x02bJjN/2tra/HLL7+ozuPh4YFHHnlEcz2O1LFjR0lM6/hQrnLrHX//6tq1Kx5++GG7c44bN072LkW5ZSkRHR1tdy09e/aUxG4f1oDIETgcAVErcOnSJeTn50viERERmnN37dpVEjt9+jSioqJU5enTpw+8vb011yPn5s2b2LVrF0wmE06fPo28vDxUVlaiqqoKZrMZQgjFueSO2rQEcgNKDh48WFNOb29v9OvXT9IoHzlyBBMmTFCVS6/XIzQ01O5a5J4xyMaJmgMbJ6JW4MyZM5KYj48Pzp8/j/Pnz2vKLddINDQ4YWNuHzfIEf766y+kpqZi06ZNDmt4Wmrj9Oeff0pit1/DZY/w8HBJ43Tx4kXVeQICAjRd1G8wGCQxrXd4Eslh40TUCshdJGs2mzWdpmmMPY2Tn5+fQ2tYtmwZ3nrrLVRXVzs0r9bb7V2loqJCEtM60ntDOeSW1ZTmONqo5kgikVJsnIhaAWffXWTPKZK2bds6bPkzZ87EypUrHZbvVi31x1iumZE7vaWWXMNrT+NE1FKwcSJqBZx9rYfVanXq8m61cuXKBpumNm3aIDw8HIMHD0ZISAiCg4NhNBrh5eUFLy8vyamizz77TPbZeS3RjRs3JDG501tqyeVoqUfliJRg40TUCigdEbylu3Lliuyz4AwGAxYtWoTp06cjMDBQcb5t27Y5sjyX8vX1lZxClXtos1pyORx92pXInbBxImoF5EbiHjJkCA4dOuSCaprPl19+KTm6ZjAYcODAAQwYMEB1vjvplFNAQICkcbr9Ycb2kDuaqaY5JWppOI4TUSsg9wyyljoeUWO+/fZbSWzRokV2NU2A868Na04BAQGSmNwQFWrJ7UdyyyK6U7BxImoFbh/dGwBKS0tRWlrqgmqaz/HjxyWxxMREh+ZrqeT2gV9//VVzXrkccssiulOwcSJyQ1ofUnu7nj17yj5aRe4ITUtVXV0tuSjZaDQiKCjIrnylpaW4cOGC3fU4ehtqJTf0xJEjR1BbW2t3zuLiYtlxwJprmAsid8DGicgNyT3Goq6uTlPOJ554QhJbu3atppzuRG5gSh8fH7vzaXlYMdA821ALuQfyVlRUYMeOHXbnXLduHerr621iOp1O84jkRO6MjRORG5K7mFvrLd5yp6wOHjyI7du3a8rrLuTu5Pr777/tOqJSXV2NjIwMTfU0xzbUol+/fujdu7cknp6eble+Gzdu4OOPP5bEhw8fzrvq6I7GxonIDcldXCv3yAw1YmNj0b9/f0l8xowZKC4u1pTbHXh7e0sGdKyrq8OePXtU53r99ddRVFSkqZ7m2IZa6HQ6vPrqq5L4Tz/9hK+++kp1vrS0NNn1mTlzpj3lEbUYbJyI3JDcxbVnz57V/Jy0FStWSGJFRUUYMWKEXc8Xu91PP/2E3bt3a85jr2HDhkliKSkpqk6Rvf/++7JHUtRqrm2oxZQpU+Dv7y+JT58+XdWF8Js3b8aSJUsk8T59+mDEiBFaSiRye2yciNyQv78/unfvbhOzWCxYvHixprwxMTGyRwRycnIQHh6ONWvWqD61VVxcjIyMDISHhyMmJkbywFdnGjNmjCRmMpkwceJEmM3mRuetqqrCa6+9hlmzZv0X0zJwaHNtQy2MRiM+/PBDSbyqqgrx8fHYunVro/MLIbBy5UpMmjRJMjq8Xq/HmjVr3O6ieCJH4wCYRG7qmWeewbJly2xiy5cvx9GjR/H0008jNDQUvr6+sj/ukZGRDeZdvnw58vLy8MMPP9jEKyoqMG3aNCxcuBAJCQmIiopCWFgYAgMD4efnh+vXr6OyshJlZWXIycnBqVOncPDgQfzyyy9u8/y2xMREpKamSo6ebd68GUePHsUrr7yC+Ph49OjRAwaDAWVlZbhw4QK2bduGtWvX2gzP0KFDB4wbN07T0afm2oZaTJo0Cdu2bUNWVpZNvLy8HE8//TTi4uIwefJkREVF4Z577oHVakVxcTH27NmDzMxMnDhxQjbvggULeDcdtQ6CiJpFSkqKACCZlPrjjz+El5eXbI6mpqZUV1eL0aNH25W7qWnJkiVNLj8zM1MyX1JSkuLPpjHZ2dlCr9drWgdPT0+xd+9e2W2YkpKiuJbm2IaO+OzMZrOIiopy2DZPSEgQFotF8fILCgokOUJCQlStw+2SkpIkOTMzMzXlJJLDU3VEbqp79+523/HUFIPBgC1btmDJkiXw9PR0aO42bdo4NJ9aw4cPR0ZGht2n2YxGI7Zu3YrY2FjNtTTnNtTC29sb2dnZSEhI0JRHp9Nhzpw5WLduXat5HiIRGyciNzZ9+nR88803dg/i2BgPDw+88cYbOHfuHCZPnqypgfLz88PUqVOxb98+zJ4924FV2ic5ORk7d+6UfdRMY2JiYmAymTBy5EiH1dKc21CLtm3bIisrCxs3bkSPHj1Uz9+/f3/s27cP7777LpsmalV0QrjJxQlE1CCr1Yrs7Gzs2rULJ0+eRH5+PqqqqmA2m2GxWCTvt+drfeXKFWzZsgXZ2dkwmUwNDlGg1+vRrVs33H///Rg8eDDi4uIwcOBAt/zxrKurw5dffomNGzfi4MGDknGUdDodunfvjsceewzPP/+8ZJDIY8eO4dixYzaxQYMGYdCgQaprccY2tJfVasX333+PjRs3Yv/+/bJ3WOp0OvTu3RuxsbFISEhAdHS00+ojcidsnIhIVnV1NYqLi2E2m2G1WuHj4wNfX1+0b9/e4af3nMFisaC0tBTl5eWwWCzw8fFBcHAwDAaDq0tzO2azGUVFRf81mr6+vggODoaXl5eLKyNyPTZORERERArxGiciIiIihdg4ERERESnExomIiIhIITZORERERAqxcSIiIiJSiI0TERERkUJsnIiIiIgUYuNEREREpBAbJyIiIiKF2DgRERERKcTGiYiIiEghNk5ERERECrFxIiIiIlKIjRMRERGRQmyciIiIiBRi40RERESkEBsnIiIiIoXYOBEREREp9H+YGXUU53HdsQAAAABJRU5ErkJggg==",
112 |       "text/plain": [
113 |        "<Figure size 600x600 with 1 Axes>"
114 |       ]
115 |      },
116 |      "metadata": {},
117 |      "output_type": "display_data"
118 |     }
119 |    ],
120 |    "source": [
121 |     "fig_dir = 'figures' \n",
122 |     "os.makedirs(fig_dir, exist_ok=True)\n",
123 |     "\n",
124 |     "fig, ax = plt.subplots(1, 1,figsize = (6, 6))\n",
125 |     "\n",
126 |     "for (alg, toclip, lr) in [('sgd', True, 0.02),('adam', True, 0.02)]:\n",
127 |     "    losses = torch.zeros(len(seeds), int(max_iters/stride))\n",
128 |     "    for idx, sd in enumerate(seeds):\n",
129 |     "        losses[idx] = loss_plots[(alg, toclip, lr, sd)]\n",
130 |     "    losses_mean = torch.mean(losses, axis=0)\n",
131 |     "    losses_std = torch.std(losses, axis=0)\n",
132 |     "    if alg == 'sgd':\n",
133 |     "        ax.plot(range(0,max_iters,stride), losses_mean, color = 'black', lw = 3,label='SGDM')\n",
134 |     "        ax.fill_between(range(0,max_iters,stride), losses_mean-losses_std/4, losses_mean+losses_std/4, color = 'black', alpha = 0.1)\n",
135 |     "    elif alg == 'adam':\n",
136 |     "        ax.plot(range(0,max_iters,stride), losses_mean, color = 'red', lw = 3, label='Adam')\n",
137 |     "        ax.fill_between(range(0,max_iters,stride), losses_mean-losses_std/4, losses_mean+losses_std/4, color = 'red', alpha = 0.1)\n",
138 |     "\n",
139 |     "    ax.set_xlabel('Iteration',fontsize=40)\n",
140 |     "    ax.tick_params(axis='both', which='major', labelsize=30, width = 3, length = 10)\n",
141 |     "    ax.tick_params(axis='both', which='minor', labelsize=20, width = 3, length = 5)\n",
142 |     "    ax.legend(fontsize=30)\n",
143 |     "    ax.spines[['right', 'top']].set_visible(False)\n",
144 |     "    ax.spines['left'].set_linewidth(3)\n",
145 |     "    ax.spines['bottom'].set_linewidth(3)\n",
146 |     "    ax.set_yscale('log')\n",
147 |     "    \n",
148 |     "    plt.tight_layout()\n",
149 |     "    plt.savefig(fig_dir + '/loss_layer{}_N{}_{}.pdf'.format(n_layer, N, mode), dpi=600)"
150 |    ]
151 |   }
152 |  ],
153 |  "metadata": {
154 |   "kernelspec": {
155 |    "display_name": "pytorch",
156 |    "language": "python",
157 |    "name": "python3"
158 |   },
159 |   "language_info": {
160 |    "codemirror_mode": {
161 |     "name": "ipython",
162 |     "version": 3
163 |    },
164 |    "file_extension": ".py",
165 |    "mimetype": "text/x-python",
166 |    "name": "python",
167 |    "nbconvert_exporter": "python",
168 |    "pygments_lexer": "ipython3",
169 |    "version": "3.11.3"
170 |   },
171 |   "orig_nbformat": 4
172 |  },
173 |  "nbformat": 4,
174 |  "nbformat_minor": 2
175 | }
176 | 


--------------------------------------------------------------------------------