├── .gitignore ├── src ├── molecule.py ├── gnn.py ├── functions.py └── main.py ├── README.md ├── LICENSE └── dataset └── PROTEINS ├── README.txt ├── README.txt~ └── PROTEINS_graph_labels.txt /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | *.pyc -------------------------------------------------------------------------------- /src/molecule.py: -------------------------------------------------------------------------------- 1 | class Molecule: 2 | def __init__(self, graph, nodes): 3 | self.graph = graph.to('cuda') # Ajacency Matrix 4 | self.nodes = nodes.to('cuda') # means C, N, ... Br 5 | 6 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Graph Isomorphism Network 2 | 3 | This program is neural network for graph classification using 4 | Graph Isomorphism Network (GIN). 5 | GIN was proposed in `"Keyulu Xu*, Weihua Hu*, Jure Leskovec, Stefanie Jegelka. How Powerful are Graph Neural Networks? ICLR 2019"`. 6 | Dataset is PROTEINS in [Benchmark Data Sets for Graph Kernels](http://graphkernels.cs.tu-dortmund.de) . 7 | 8 | And accuracy is about 0.74. 9 | 10 | ## Requirement 11 | * python 3.6 12 | * pytorch 13 | 14 | And GPU is necessary. 15 | 16 | ## Usage 17 | ``` 18 | cd ./src 19 | python3 main.py 20 | ``` 21 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 yukiTakezawa 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /src/gnn.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import torch 3 | from molecule import * 4 | import copy 5 | 6 | class GraphIsomorphismNetwork: 7 | def __init__(self, node_dim, update_loop_size): 8 | self.node_dim = node_dim 9 | self.update_loop_size = update_loop_size 10 | self.eps = 0.0 #np.random.normal() # learnable parameter 11 | 12 | # function to update nodes 13 | def mlp(self, molecule): 14 | next_nodes = torch.zeros_like(molecule.nodes) 15 | #for i in range(next_nodes.shape[0]): 16 | # next_nodes[i] = (torch.t(molecule.graph[i]) * molecule.nodes).sum(dim=1) 17 | #return (1.0 + eps)*molecule.nodes + next_nodes 18 | return molecule.nodes + torch.mm(molecule.graph, molecule.nodes) 19 | 20 | def readout(self, molecule): 21 | return molecule.nodes.sum(dim=0) 22 | 23 | def predict(self, molecule): 24 | tmp_molecule = copy.deepcopy(molecule) 25 | # CONCAT(READOUT(molecule.nodes at k) k < update_loop_size) 26 | sum_of_nodes = torch.zeros(self.node_dim).to('cuda') 27 | for i in range(self.update_loop_size): 28 | tmp_molecule.nodes = self.mlp(tmp_molecule) 29 | sum_of_nodes += self.readout(tmp_molecule) 30 | 31 | return sum_of_nodes 32 | -------------------------------------------------------------------------------- /dataset/PROTEINS/README.txt: -------------------------------------------------------------------------------- 1 | README for dataset PROTEINS 2 | 3 | 4 | === Usage === 5 | 6 | This folder contains the following comma separated text files 7 | (replace DS by the name of the dataset): 8 | 9 | n = total number of nodes 10 | m = total number of edges 11 | N = number of graphs 12 | 13 | (1) DS_A.txt (m lines) 14 | sparse (block diagonal) adjacency matrix for all graphs, 15 | each line corresponds to (row, col) resp. (node_id, node_id) 16 | 17 | (2) DS_graph_indicator.txt (n lines) 18 | column vector of graph identifiers for all nodes of all graphs, 19 | the value in the i-th line is the graph_id of the node with node_id i 20 | 21 | (3) DS_graph_labels.txt (N lines) 22 | class labels for all graphs in the dataset, 23 | the value in the i-th line is the class label of the graph with graph_id i 24 | 25 | (4) DS_node_labels.txt (n lines) 26 | column vector of node labels, 27 | the value in the i-th line corresponds to the node with node_id i 28 | 29 | There are OPTIONAL files if the respective information is available: 30 | 31 | (5) DS_edge_labels.txt (m lines; same size as DS_A_sparse.txt) 32 | labels for the edges in DS_A_sparse.txt 33 | 34 | (6) DS_edge_attributes.txt (m lines; same size as DS_A.txt) 35 | attributes for the edges in DS_A.txt 36 | 37 | (7) DS_node_attributes.txt (n lines) 38 | matrix of node attributes, 39 | the comma seperated values in the i-th line is the attribute vector of the node with node_id i 40 | 41 | (8) DS_graph_attributes.txt (N lines) 42 | regression values for all graphs in the dataset, 43 | the value in the i-th line is the attribute of the graph with graph_id i 44 | 45 | 46 | === Previous Use of the Dataset === 47 | 48 | Feragen, A., Kasenburg, N., Petersen, J., de Bruijne, M., Borgwardt, K.M.: Scalable 49 | kernels for graphs with continuous attributes. In: C.J.C. Burges, L. Bottou, Z. Ghahra- 50 | mani, K.Q. Weinberger (eds.) NIPS, pp. 216-224 (2013) 51 | 52 | Neumann, M., Garnett R., Bauckhage Ch., Kersting K.: Propagation Kernels: Efficient Graph 53 | Kernels from Propagated Information. Under review at MLJ. 54 | 55 | 56 | === References === 57 | 58 | K. M. Borgwardt, C. S. Ong, S. Schoenauer, S. V. N. Vishwanathan, A. J. Smola, and H. P. 59 | Kriegel. Protein function prediction via graph kernels. Bioinformatics, 21(Suppl 1):i47–i56, 60 | Jun 2005. 61 | 62 | P. D. Dobson and A. J. Doig. Distinguishing enzyme structures from non-enzymes without 63 | alignments. J. Mol. Biol., 330(4):771–783, Jul 2003. 64 | 65 | 66 | -------------------------------------------------------------------------------- /dataset/PROTEINS/README.txt~: -------------------------------------------------------------------------------- 1 | README for dataset PROTEINS 2 | 3 | 4 | === Usage === 5 | 6 | This folder contains the following comma separated text files 7 | (replace DS by the name of the dataset): 8 | 9 | n = total number of nodes 10 | m = total number of edges 11 | N = number of graphs 12 | 13 | (1) DS_A.txt (m lines) 14 | sparse (block diagonal) adjacency matrix for all graphs, 15 | each line corresponds to (row, col) resp. (node_id, node_id) 16 | 17 | (2) DS_graph_indicator.txt (n lines) 18 | column vector of graph identifiers for all nodes of all graphs, 19 | the value in the i-th line is the graph_id of the node with node_id i 20 | 21 | (3) DS_graph_labels.txt (N lines) 22 | class labels for all graphs in the dataset, 23 | the value in the i-th line is the class label of the graph with graph_id i 24 | 25 | (4) DS_node_labels.txt (n lines) 26 | column vector of node labels, 27 | the value in the i-th line corresponds to the node with node_id i 28 | 29 | There are OPTIONAL files if the respective information is available: 30 | 31 | (5) DS_edge_labels.txt (m lines; same size as DS_A_sparse.txt) 32 | labels for the edges in DS_A_sparse.txt 33 | 34 | (6) DS_edge_attributes.txt (m lines; same size as DS_A.txt) 35 | attributes for the edges in DS_A.txt 36 | 37 | (7) DS_node_attributes.txt (n lines) 38 | matrix of node attributes, 39 | the comma seperated values in the i-th line is the attribute vector of the node with node_id i 40 | 41 | (8) DS_graph_attributes.txt (N lines) 42 | regression values for all graphs in the dataset, 43 | the value in the i-th line is the attribute of the graph with graph_id i 44 | 45 | 46 | === Previous Use of the Dataset === 47 | 48 | Feragen, A., Kasenburg, N., Petersen, J., de Bruijne, M., Borgwardt, K.M.: Scalable 49 | kernels for graphs with continuous attributes. In: C.J.C. Burges, L. Bottou, Z. Ghahra- 50 | mani, K.Q. Weinberger (eds.) NIPS, pp. 216-224 (2013) 51 | 52 | Neumann, M., Garnett R., Bauckhage Ch., Kersting K.: Propagation Kernels: Efficient Graph 53 | Kernels from Propagated Information. Under review at MLJ. 54 | 55 | 56 | === References === 57 | 58 | K. M. Borgwardt, C. S. Ong, S. Schoenauer, S. V. N. Vishwanathan, A. J. Smola, and H. P. 59 | Kriegel. Protein function prediction via graph kernels. Bioinformatics, 21(Suppl 1):i47–i56, 60 | Jun 2005. 61 | 62 | I. Schomburg, A. Chang, C. Ebeling, M. Gremse, C. Heldt, G. Huhn, and D. Schomburg. Brenda, 63 | the enzyme database: updates and major new developments. Nucleic Acids Research, 32D:431–433, 2004. 64 | 65 | 66 | -------------------------------------------------------------------------------- /src/functions.py: -------------------------------------------------------------------------------- 1 | import math 2 | import torch 3 | from ast import literal_eval 4 | from molecule import * 5 | 6 | def load_data(): 7 | ### load data 8 | # 9 | # Returns: 10 | # list of Molecule: 11 | # list of torch.Tensor: 12 | ### 13 | 14 | data_num = 1112 15 | 16 | graph_label_file = open('../dataset/PROTEINS/PROTEINS_graph_labels.txt') 17 | node_label_file = open('../dataset/PROTEINS/PROTEINS_node_labels.txt') 18 | node_attribute_file = open('../dataset/PROTEINS/PROTEINS_node_attributes.txt') 19 | graph_idx_file = open('../dataset/PROTEINS/PROTEINS_graph_indicator.txt') 20 | adjacency_file = open('../dataset/PROTEINS/PROTEINS_A.txt') 21 | 22 | # check graph size 23 | tmp0, tmp1 = 1, 1 24 | graph_list = [] # list of adjacency matrix 25 | graph_size_list = [] # list of total number of nodes that each graph has 26 | 27 | for graph_idx in range(data_num): 28 | graph_size = 1 # total number of nodes that graph has 29 | 30 | while True: 31 | tmp1 = int(graph_idx_file.readline()) 32 | if (tmp0 == tmp1): 33 | graph_size += 1 34 | tmp0 = tmp1 35 | else: 36 | tmp0 = tmp1 37 | break 38 | 39 | # append adjacency matrix 40 | graph_list.append(torch.zeros(graph_size, graph_size)) 41 | graph_size_list.append(graph_size) 42 | 43 | # chack adjacency matrix 44 | tmp_sum = 0 45 | tmp_sum1 = 0 46 | for graph_idx in range(data_num): 47 | tmp_sum += graph_size_list[graph_idx] 48 | 49 | while True: 50 | edge = literal_eval(adjacency_file.readline()) 51 | if ((edge[0] <= tmp_sum) & (edge[1] <= tmp_sum)): 52 | graph_list[graph_idx][edge[0] - tmp_sum1][edge[1] - tmp_sum1] = 1.0 53 | graph_list[graph_idx][edge[1] - tmp_sum1][edge[0] - tmp_sum1] = 1.0 54 | else: 55 | break 56 | tmp_sum1 = tmp_sum 57 | 58 | print(graph_list[0]) 59 | 60 | # check node feature 61 | node_dim = 4 # dimension of node 62 | node_list = [] 63 | for graph_idx in range(data_num): 64 | nodes = torch.zeros(graph_size_list[graph_idx], node_dim) 65 | 66 | for node_idx in range(graph_size_list[graph_idx]): 67 | node_label = int(node_label_file.readline()) 68 | node_attribute = float(node_attribute_file.readline()) 69 | node_val = 1.0 #0.000000000000001 70 | if (node_label == 0): 71 | nodes[node_idx] = torch.tensor([node_val, 0.0, 0.0, node_attribute]) 72 | elif (node_label == 1): 73 | nodes[node_idx] = torch.tensor([0.0, node_val, 0.0, node_attribute]) 74 | else: 75 | nodes[node_idx] = torch.tensor([0.0, 0.0, node_val, node_attribute]) 76 | node_list.append(nodes) 77 | 78 | 79 | # set molecule class 80 | molecule_list = [] 81 | label_list = [] 82 | for graph_idx in range(data_num): 83 | molecule_list.append(Molecule(graph_list[graph_idx], node_list[graph_idx])) 84 | label_list.append(torch.tensor([float(graph_label_file.readline()) - 1.0]).to('cuda')) 85 | 86 | graph_label_file.close() 87 | node_label_file.close() 88 | node_attribute_file.close() 89 | graph_idx_file.close() 90 | adjacency_file.close() 91 | 92 | return molecule_list, label_list 93 | 94 | -------------------------------------------------------------------------------- /src/main.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torch.optim as optim 4 | import torch.nn.functional as F 5 | import sys 6 | import matplotlib.pyplot as plt 7 | 8 | from functions import * 9 | from gnn import * 10 | 11 | class Model(nn.Module): 12 | def __init__(self): 13 | super(Model, self).__init__() 14 | self.fun0 = nn.Linear(4, 100) 15 | self.fun1 = nn.Linear(100, 50) 16 | self.fun2 = nn.Linear(50, 25) 17 | self.fun3 = nn.Linear(25, 1) 18 | 19 | def forward(self, x): 20 | #print(x) 21 | x = self.fun0(x) 22 | x = F.relu(x) 23 | x = self.fun1(x) 24 | x = F.relu(x) 25 | x = self.fun2(x) 26 | x = F.relu(x) 27 | x = self.fun3(x) 28 | x = F.sigmoid(x) 29 | #print(x) 30 | return x 31 | 32 | 33 | def preproceccing(data): 34 | ### processe data by GIN and normalize each value 35 | # 36 | # Args: 37 | # data(list of Molecule): 38 | # 39 | # Returns: 40 | # list of torch.Tensor : processed data by GIN 41 | ### 42 | 43 | gin = GraphIsomorphismNetwork(4, 20) 44 | processed_data = [] 45 | 46 | for i in range(len(data)): 47 | processed_data.append(gin.predict(data[i])) 48 | 49 | maximum_attribute_val = 0 # maximum value of attribute value 50 | 51 | for i in range(len(processed_data)): 52 | processed_data[i][0:3] /= processed_data[i][0:3].max() 53 | 54 | tmp = processed_data[i][3] 55 | tmp = tmp.cpu() 56 | tmp = tmp.numpy() 57 | if (tmp > maximum_attribute_val): 58 | maximum_attribute_val = tmp 59 | 60 | for i in range(len(processed_data)): 61 | processed_data[i][3] /= torch.from_numpy(maximum_attribute_val) 62 | 63 | return processed_data 64 | 65 | def shuffle_data(x, y): 66 | ### shuffle x and y 67 | # 68 | # Args: 69 | # x (list): 70 | # y (list); 71 | # 72 | # Returns: 73 | # list, list: shuffled x and y 74 | ### 75 | 76 | zipped = list(zip(x, y)) 77 | np.random.shuffle(zipped) 78 | x, y = zip(*zipped) 79 | return x, y 80 | 81 | def calc_accuracy(model, test_data, test_labels, test_size): 82 | ### calcurate accuracy 83 | # 84 | # Args: 85 | # model (Model): trained model 86 | # test_data (list): 87 | # test_labels (list): 88 | # test_size (int): size of test data 89 | # 90 | # Returns: 91 | # float: accuracy 92 | ### 93 | 94 | correct = 0 95 | error1 = 0 96 | error0 = 0 97 | count1 = 0 98 | count0 = 0 99 | with torch.no_grad(): 100 | for i in range(test_size): 101 | output = model(test_data[i]) 102 | prediction = (output.cpu()).numpy()[0] 103 | label = (test_labels[i].cpu()).numpy()[0] 104 | 105 | if ((prediction>=0.5) & (label==1)): 106 | correct += 1 107 | count1 += 1 108 | elif ((prediction<0.5) & (label==0)): 109 | correct += 1 110 | count0 += 1 111 | elif ((prediction>=0.5) & (label==0)): 112 | error1 += 1 113 | else: 114 | error0 += 1 115 | 116 | print(count0, count1, error0, error1) 117 | return correct/test_size 118 | 119 | def main(): 120 | epoch_size = 1500 121 | train_data_size = 912 122 | test_data_size = 200 123 | accuracy_list = [] 124 | 125 | device = torch.device('cuda') 126 | data, labels = load_data() 127 | processed_data = preproceccing(data) 128 | 129 | model = Model().to(device) 130 | optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9) 131 | criterion = nn.BCELoss()#nn.MSELoss() 132 | 133 | torch.autograd.set_detect_anomaly(True) 134 | 135 | # split data into train data and test data 136 | shuffled_data, shuffled_labels = shuffle_data(processed_data, labels) 137 | test_data = shuffled_data[0:200] 138 | test_labels = shuffled_labels[0:200] 139 | train_data = shuffled_data[200:1112] 140 | train_labels = shuffled_labels[200:1112] 141 | 142 | # traning 143 | for i in range(epoch_size): 144 | 145 | shuffled_data, shuffled_labels = shuffle_data(train_data, train_labels) 146 | 147 | for j in range(train_data_size): 148 | with torch.autograd.detect_anomaly(): 149 | optimizer.zero_grad() 150 | output = model(shuffled_data[j]) 151 | #print(output, shuffled_labels[j]) 152 | loss = criterion(output, shuffled_labels[j]) 153 | loss.backward() 154 | optimizer.step() 155 | 156 | accuracy = calc_accuracy(model, test_data, test_labels, test_data_size) 157 | accuracy_list.append(accuracy) 158 | print(accuracy) 159 | print("learnig %d epoch" % (i)) 160 | print("finished learning \n best accuracy is %f" % (max(accuracy_list))) 161 | 162 | 163 | plt.plot(accuracy_list) 164 | plt.show() 165 | print("end") 166 | 167 | if __name__ == '__main__': 168 | main() 169 | -------------------------------------------------------------------------------- /dataset/PROTEINS/PROTEINS_graph_labels.txt: -------------------------------------------------------------------------------- 1 | 1 2 | 1 3 | 1 4 | 1 5 | 1 6 | 1 7 | 1 8 | 1 9 | 1 10 | 1 11 | 1 12 | 1 13 | 1 14 | 1 15 | 1 16 | 1 17 | 1 18 | 1 19 | 1 20 | 1 21 | 1 22 | 1 23 | 1 24 | 1 25 | 1 26 | 1 27 | 1 28 | 1 29 | 1 30 | 1 31 | 1 32 | 1 33 | 1 34 | 1 35 | 1 36 | 1 37 | 1 38 | 1 39 | 1 40 | 1 41 | 1 42 | 1 43 | 1 44 | 1 45 | 1 46 | 1 47 | 1 48 | 1 49 | 1 50 | 1 51 | 1 52 | 1 53 | 1 54 | 1 55 | 1 56 | 1 57 | 1 58 | 1 59 | 1 60 | 1 61 | 1 62 | 1 63 | 1 64 | 1 65 | 1 66 | 1 67 | 1 68 | 1 69 | 1 70 | 1 71 | 1 72 | 1 73 | 1 74 | 1 75 | 1 76 | 1 77 | 1 78 | 1 79 | 1 80 | 1 81 | 1 82 | 1 83 | 1 84 | 1 85 | 1 86 | 1 87 | 1 88 | 1 89 | 1 90 | 1 91 | 1 92 | 1 93 | 1 94 | 1 95 | 1 96 | 1 97 | 1 98 | 1 99 | 1 100 | 1 101 | 1 102 | 1 103 | 1 104 | 1 105 | 1 106 | 1 107 | 1 108 | 1 109 | 1 110 | 1 111 | 1 112 | 1 113 | 1 114 | 1 115 | 1 116 | 1 117 | 1 118 | 1 119 | 1 120 | 1 121 | 1 122 | 1 123 | 1 124 | 1 125 | 1 126 | 1 127 | 1 128 | 1 129 | 1 130 | 1 131 | 1 132 | 1 133 | 1 134 | 1 135 | 1 136 | 1 137 | 1 138 | 1 139 | 1 140 | 1 141 | 1 142 | 1 143 | 1 144 | 1 145 | 1 146 | 1 147 | 1 148 | 1 149 | 1 150 | 1 151 | 1 152 | 1 153 | 1 154 | 1 155 | 1 156 | 1 157 | 1 158 | 1 159 | 1 160 | 1 161 | 1 162 | 1 163 | 1 164 | 1 165 | 1 166 | 1 167 | 1 168 | 1 169 | 1 170 | 1 171 | 1 172 | 1 173 | 1 174 | 1 175 | 1 176 | 1 177 | 1 178 | 1 179 | 1 180 | 1 181 | 1 182 | 1 183 | 1 184 | 1 185 | 1 186 | 1 187 | 1 188 | 1 189 | 1 190 | 1 191 | 1 192 | 1 193 | 1 194 | 1 195 | 1 196 | 1 197 | 1 198 | 1 199 | 1 200 | 1 201 | 1 202 | 1 203 | 1 204 | 1 205 | 1 206 | 1 207 | 1 208 | 1 209 | 1 210 | 1 211 | 1 212 | 1 213 | 1 214 | 1 215 | 1 216 | 1 217 | 1 218 | 1 219 | 1 220 | 1 221 | 1 222 | 1 223 | 1 224 | 1 225 | 1 226 | 1 227 | 1 228 | 1 229 | 1 230 | 1 231 | 1 232 | 1 233 | 1 234 | 1 235 | 1 236 | 1 237 | 1 238 | 1 239 | 1 240 | 1 241 | 1 242 | 1 243 | 1 244 | 1 245 | 1 246 | 1 247 | 1 248 | 1 249 | 1 250 | 1 251 | 1 252 | 1 253 | 1 254 | 1 255 | 1 256 | 1 257 | 1 258 | 1 259 | 1 260 | 1 261 | 1 262 | 1 263 | 1 264 | 1 265 | 1 266 | 1 267 | 1 268 | 1 269 | 1 270 | 1 271 | 1 272 | 1 273 | 1 274 | 1 275 | 1 276 | 1 277 | 1 278 | 1 279 | 1 280 | 1 281 | 1 282 | 1 283 | 1 284 | 1 285 | 1 286 | 1 287 | 1 288 | 1 289 | 1 290 | 1 291 | 1 292 | 1 293 | 1 294 | 1 295 | 1 296 | 1 297 | 1 298 | 1 299 | 1 300 | 1 301 | 1 302 | 1 303 | 1 304 | 1 305 | 1 306 | 1 307 | 1 308 | 1 309 | 1 310 | 1 311 | 1 312 | 1 313 | 1 314 | 1 315 | 1 316 | 1 317 | 1 318 | 1 319 | 1 320 | 1 321 | 1 322 | 1 323 | 1 324 | 1 325 | 1 326 | 1 327 | 1 328 | 1 329 | 1 330 | 1 331 | 1 332 | 1 333 | 1 334 | 1 335 | 1 336 | 1 337 | 1 338 | 1 339 | 1 340 | 1 341 | 1 342 | 1 343 | 1 344 | 1 345 | 1 346 | 1 347 | 1 348 | 1 349 | 1 350 | 1 351 | 1 352 | 1 353 | 1 354 | 1 355 | 1 356 | 1 357 | 1 358 | 1 359 | 1 360 | 1 361 | 1 362 | 1 363 | 1 364 | 1 365 | 1 366 | 1 367 | 1 368 | 1 369 | 1 370 | 1 371 | 1 372 | 1 373 | 1 374 | 1 375 | 1 376 | 1 377 | 1 378 | 1 379 | 1 380 | 1 381 | 1 382 | 1 383 | 1 384 | 1 385 | 1 386 | 1 387 | 1 388 | 1 389 | 1 390 | 1 391 | 1 392 | 1 393 | 1 394 | 1 395 | 1 396 | 1 397 | 1 398 | 1 399 | 1 400 | 1 401 | 1 402 | 1 403 | 1 404 | 1 405 | 1 406 | 1 407 | 1 408 | 1 409 | 1 410 | 1 411 | 1 412 | 1 413 | 1 414 | 1 415 | 1 416 | 1 417 | 1 418 | 1 419 | 1 420 | 1 421 | 1 422 | 1 423 | 1 424 | 1 425 | 1 426 | 1 427 | 1 428 | 1 429 | 1 430 | 1 431 | 1 432 | 1 433 | 1 434 | 1 435 | 1 436 | 1 437 | 1 438 | 1 439 | 1 440 | 1 441 | 1 442 | 1 443 | 1 444 | 1 445 | 1 446 | 1 447 | 1 448 | 1 449 | 1 450 | 1 451 | 1 452 | 1 453 | 1 454 | 1 455 | 1 456 | 1 457 | 1 458 | 1 459 | 1 460 | 1 461 | 1 462 | 1 463 | 1 464 | 1 465 | 1 466 | 1 467 | 1 468 | 1 469 | 1 470 | 1 471 | 1 472 | 1 473 | 1 474 | 1 475 | 1 476 | 1 477 | 1 478 | 1 479 | 1 480 | 1 481 | 1 482 | 1 483 | 1 484 | 1 485 | 1 486 | 1 487 | 1 488 | 1 489 | 1 490 | 1 491 | 1 492 | 1 493 | 1 494 | 1 495 | 1 496 | 1 497 | 1 498 | 1 499 | 1 500 | 1 501 | 1 502 | 1 503 | 1 504 | 1 505 | 1 506 | 1 507 | 1 508 | 1 509 | 1 510 | 1 511 | 1 512 | 1 513 | 1 514 | 1 515 | 1 516 | 1 517 | 1 518 | 1 519 | 1 520 | 1 521 | 1 522 | 1 523 | 1 524 | 1 525 | 1 526 | 1 527 | 1 528 | 1 529 | 1 530 | 1 531 | 1 532 | 1 533 | 1 534 | 1 535 | 1 536 | 1 537 | 1 538 | 1 539 | 1 540 | 1 541 | 1 542 | 1 543 | 1 544 | 1 545 | 1 546 | 1 547 | 1 548 | 1 549 | 1 550 | 1 551 | 1 552 | 1 553 | 1 554 | 1 555 | 1 556 | 1 557 | 1 558 | 1 559 | 1 560 | 1 561 | 1 562 | 1 563 | 1 564 | 1 565 | 1 566 | 1 567 | 1 568 | 1 569 | 1 570 | 1 571 | 1 572 | 1 573 | 1 574 | 1 575 | 1 576 | 1 577 | 1 578 | 1 579 | 1 580 | 1 581 | 1 582 | 1 583 | 1 584 | 1 585 | 1 586 | 1 587 | 1 588 | 1 589 | 1 590 | 1 591 | 1 592 | 1 593 | 1 594 | 1 595 | 1 596 | 1 597 | 1 598 | 1 599 | 1 600 | 1 601 | 1 602 | 1 603 | 1 604 | 1 605 | 1 606 | 1 607 | 1 608 | 1 609 | 1 610 | 1 611 | 1 612 | 1 613 | 1 614 | 1 615 | 1 616 | 1 617 | 1 618 | 1 619 | 1 620 | 1 621 | 1 622 | 1 623 | 1 624 | 1 625 | 1 626 | 1 627 | 1 628 | 1 629 | 1 630 | 1 631 | 1 632 | 1 633 | 1 634 | 1 635 | 1 636 | 1 637 | 1 638 | 1 639 | 1 640 | 1 641 | 1 642 | 1 643 | 1 644 | 1 645 | 1 646 | 1 647 | 1 648 | 1 649 | 1 650 | 1 651 | 1 652 | 1 653 | 1 654 | 1 655 | 1 656 | 1 657 | 1 658 | 1 659 | 1 660 | 1 661 | 1 662 | 1 663 | 1 664 | 2 665 | 2 666 | 2 667 | 2 668 | 2 669 | 2 670 | 2 671 | 2 672 | 2 673 | 2 674 | 2 675 | 2 676 | 2 677 | 2 678 | 2 679 | 2 680 | 2 681 | 2 682 | 2 683 | 2 684 | 2 685 | 2 686 | 2 687 | 2 688 | 2 689 | 2 690 | 2 691 | 2 692 | 2 693 | 2 694 | 2 695 | 2 696 | 2 697 | 2 698 | 2 699 | 2 700 | 2 701 | 2 702 | 2 703 | 2 704 | 2 705 | 2 706 | 2 707 | 2 708 | 2 709 | 2 710 | 2 711 | 2 712 | 2 713 | 2 714 | 2 715 | 2 716 | 2 717 | 2 718 | 2 719 | 2 720 | 2 721 | 2 722 | 2 723 | 2 724 | 2 725 | 2 726 | 2 727 | 2 728 | 2 729 | 2 730 | 2 731 | 2 732 | 2 733 | 2 734 | 2 735 | 2 736 | 2 737 | 2 738 | 2 739 | 2 740 | 2 741 | 2 742 | 2 743 | 2 744 | 2 745 | 2 746 | 2 747 | 2 748 | 2 749 | 2 750 | 2 751 | 2 752 | 2 753 | 2 754 | 2 755 | 2 756 | 2 757 | 2 758 | 2 759 | 2 760 | 2 761 | 2 762 | 2 763 | 2 764 | 2 765 | 2 766 | 2 767 | 2 768 | 2 769 | 2 770 | 2 771 | 2 772 | 2 773 | 2 774 | 2 775 | 2 776 | 2 777 | 2 778 | 2 779 | 2 780 | 2 781 | 2 782 | 2 783 | 2 784 | 2 785 | 2 786 | 2 787 | 2 788 | 2 789 | 2 790 | 2 791 | 2 792 | 2 793 | 2 794 | 2 795 | 2 796 | 2 797 | 2 798 | 2 799 | 2 800 | 2 801 | 2 802 | 2 803 | 2 804 | 2 805 | 2 806 | 2 807 | 2 808 | 2 809 | 2 810 | 2 811 | 2 812 | 2 813 | 2 814 | 2 815 | 2 816 | 2 817 | 2 818 | 2 819 | 2 820 | 2 821 | 2 822 | 2 823 | 2 824 | 2 825 | 2 826 | 2 827 | 2 828 | 2 829 | 2 830 | 2 831 | 2 832 | 2 833 | 2 834 | 2 835 | 2 836 | 2 837 | 2 838 | 2 839 | 2 840 | 2 841 | 2 842 | 2 843 | 2 844 | 2 845 | 2 846 | 2 847 | 2 848 | 2 849 | 2 850 | 2 851 | 2 852 | 2 853 | 2 854 | 2 855 | 2 856 | 2 857 | 2 858 | 2 859 | 2 860 | 2 861 | 2 862 | 2 863 | 2 864 | 2 865 | 2 866 | 2 867 | 2 868 | 2 869 | 2 870 | 2 871 | 2 872 | 2 873 | 2 874 | 2 875 | 2 876 | 2 877 | 2 878 | 2 879 | 2 880 | 2 881 | 2 882 | 2 883 | 2 884 | 2 885 | 2 886 | 2 887 | 2 888 | 2 889 | 2 890 | 2 891 | 2 892 | 2 893 | 2 894 | 2 895 | 2 896 | 2 897 | 2 898 | 2 899 | 2 900 | 2 901 | 2 902 | 2 903 | 2 904 | 2 905 | 2 906 | 2 907 | 2 908 | 2 909 | 2 910 | 2 911 | 2 912 | 2 913 | 2 914 | 2 915 | 2 916 | 2 917 | 2 918 | 2 919 | 2 920 | 2 921 | 2 922 | 2 923 | 2 924 | 2 925 | 2 926 | 2 927 | 2 928 | 2 929 | 2 930 | 2 931 | 2 932 | 2 933 | 2 934 | 2 935 | 2 936 | 2 937 | 2 938 | 2 939 | 2 940 | 2 941 | 2 942 | 2 943 | 2 944 | 2 945 | 2 946 | 2 947 | 2 948 | 2 949 | 2 950 | 2 951 | 2 952 | 2 953 | 2 954 | 2 955 | 2 956 | 2 957 | 2 958 | 2 959 | 2 960 | 2 961 | 2 962 | 2 963 | 2 964 | 2 965 | 2 966 | 2 967 | 2 968 | 2 969 | 2 970 | 2 971 | 2 972 | 2 973 | 2 974 | 2 975 | 2 976 | 2 977 | 2 978 | 2 979 | 2 980 | 2 981 | 2 982 | 2 983 | 2 984 | 2 985 | 2 986 | 2 987 | 2 988 | 2 989 | 2 990 | 2 991 | 2 992 | 2 993 | 2 994 | 2 995 | 2 996 | 2 997 | 2 998 | 2 999 | 2 1000 | 2 1001 | 2 1002 | 2 1003 | 2 1004 | 2 1005 | 2 1006 | 2 1007 | 2 1008 | 2 1009 | 2 1010 | 2 1011 | 2 1012 | 2 1013 | 2 1014 | 2 1015 | 2 1016 | 2 1017 | 2 1018 | 2 1019 | 2 1020 | 2 1021 | 2 1022 | 2 1023 | 2 1024 | 2 1025 | 2 1026 | 2 1027 | 2 1028 | 2 1029 | 2 1030 | 2 1031 | 2 1032 | 2 1033 | 2 1034 | 2 1035 | 2 1036 | 2 1037 | 2 1038 | 2 1039 | 2 1040 | 2 1041 | 2 1042 | 2 1043 | 2 1044 | 2 1045 | 2 1046 | 2 1047 | 2 1048 | 2 1049 | 2 1050 | 2 1051 | 2 1052 | 2 1053 | 2 1054 | 2 1055 | 2 1056 | 2 1057 | 2 1058 | 2 1059 | 2 1060 | 2 1061 | 2 1062 | 2 1063 | 2 1064 | 2 1065 | 2 1066 | 2 1067 | 2 1068 | 2 1069 | 2 1070 | 2 1071 | 2 1072 | 2 1073 | 2 1074 | 2 1075 | 2 1076 | 2 1077 | 2 1078 | 2 1079 | 2 1080 | 2 1081 | 2 1082 | 2 1083 | 2 1084 | 2 1085 | 2 1086 | 2 1087 | 2 1088 | 2 1089 | 2 1090 | 2 1091 | 2 1092 | 2 1093 | 2 1094 | 2 1095 | 2 1096 | 2 1097 | 2 1098 | 2 1099 | 2 1100 | 2 1101 | 2 1102 | 2 1103 | 2 1104 | 2 1105 | 2 1106 | 2 1107 | 2 1108 | 2 1109 | 2 1110 | 2 1111 | 2 1112 | 2 1113 | 2 1114 | --------------------------------------------------------------------------------