├── .ipynb_checkpoints ├── gcn-node_classification-checkpoint.ipynb └── gcn-prediction-checkpoint.ipynb ├── A gentle introduction about graph neural networks.pdf ├── GCN-Thomas Kipf.pdf ├── How_Powerful_are_GNN.pdf ├── Inductive biases, graph neural networks, attention and relational inference.pdf ├── README.md ├── figures ├── gat_attention.png ├── gcn_prediction.jpg ├── graph_conv.jpg └── gru.png ├── gnn.png └── tutorials ├── .ipynb_checkpoints ├── ggnn-checkpoint.ipynb └── graph_attn-checkpoint.ipynb ├── gat.ipynb ├── gcn_classification.ipynb ├── gcn_prediction.ipynb └── ggnn.ipynb /.ipynb_checkpoints/gcn-node_classification-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [], 3 | "metadata": {}, 4 | "nbformat": 4, 5 | "nbformat_minor": 2 6 | } 7 | -------------------------------------------------------------------------------- /.ipynb_checkpoints/gcn-prediction-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Prediction of molecular properties using GCN\n", 8 | "\n", 9 | "A tutorial for supervised learning of labels as outputs of graph inputs.
\n", 10 | "I implemented GCN and GAT to predict molecular properties and observed better results from GAT.
\n", 11 | "Please refer to the following article for the whole contents.
\n", 12 | "Ryu, Seongok, Jaechang Lim, and Woo Youn Kim. \"Deeply learning molecular structure-property relationships using graph attention neural network.\" arXiv preprint arXiv:1805.10988 (2018).\n", 13 | "\n", 14 | "There is a key difference between the node classification and prediction of labels from whole graph inputs. The later task has to satisfy permutation invariance with respect to changing node orders. Therefore, we will implement readout functions which satisfy the permutation invariance in this tutorial." 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": 1, 20 | "metadata": {}, 21 | "outputs": [ 22 | { 23 | "name": "stderr", 24 | "output_type": "stream", 25 | "text": [ 26 | "/Users/Lulu/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n", 27 | " from ._conv import register_converters as _register_converters\n" 28 | ] 29 | } 30 | ], 31 | "source": [ 32 | "import tensorflow as tf\n", 33 | "from IPython.display import Image" 34 | ] 35 | }, 36 | { 37 | "cell_type": "markdown", 38 | "metadata": {}, 39 | "source": [ 40 | "Overall architecture is as below." 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": 2, 46 | "metadata": {}, 47 | "outputs": [ 48 | { 49 | "data": { 50 | "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAMCAgICAgMCAgIDAwMDBAYEBAQEBAgGBgUGCQgKCgkICQkKDA8MCgsOCwkJDRENDg8QEBEQCgwSExIQEw8QEBD/2wBDAQMDAwQDBAgEBAgQCwkLEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBD/wAARCAGlANQDASIAAhEBAxEB/8QAHQABAAIDAQEBAQAAAAAAAAAAAAYHBAUIAgMJAf/EAE0QAAAFAwIBBwcIBwYFBAMAAAECAwQFAAYHERITCBQXIVZX1BYiMZSVltIVQVFSVZKT0yMyNmF1gbM0N0J3tbYzYnFzkRgkobRFcoL/xAAaAQEAAwEBAQAAAAAAAAAAAAAAAQIDBQYE/8QANREAAgADBgUCBQQCAgMAAAAAAAECA1IREhMUkdEEIVGh4TGxQVNx0vAFM2GBIjJCwRVi8f/aAAwDAQACEQMRAD8A/VOlKUAqH5cynamFMdTeTr2XWJEwaAKqEbkA666hjARJBIoiAGUUUMQhQ1ABMYNRAOuphVD8tW37nmsGLSlpW66uB9ak/CXSeJaE3uH7dg/RcLJJF69ygppnEC6CIiGgAIiFARZ7yq83WIwiL6zXyX1LTsGUeINXUo1upF+/hCuFipN1XzMESbSiZQm/hqHFPXQQEQ0roh5ednx0ohByF1w7aScuCNEGaz5Ii6q5yGUIkVMTbjHMQpjAUA1EpREOoBrhrlRcrjH1zWE4yDyeeUtfJ7sRh+I0s+12jdYiIJqCZd3JoqtFFmpEiHHimOomAAmUA6x1Gx07Hs1flF56zBL46bXVc9qWtBnhQOkBnRDfJzk5yNFBDciqoIATiEEDegAH06gdJQmTMb3LcL+0rcyDbUrORX9vjGUsgu7af91EhxOn/wD0AV/GmT8ayFxtrPYZDtlzPPWwvW0WjLNzvFm4CICsRED7zE1AfOANOoeuvzQxBdmO0chcmR9b124pbtGsmo2PE2dazhupBC9i3JE2chLLuFDqrqLmKThKkIdZQDKCHm18cP3ByY5zkxY6x7jyOjHWbxv9gug1YtBGbRkk5Yqjh4dQobytysiHEVDG4YEAC+kugAfoNaHKJsKcm5y3LnnIG15OPu15akYzfzSJF5dRAqQ8RBM+wxjG4unDKBxDT0jrUxyXePR3jq6L/GO+UPJqGey/NONwucc3ROrw9+htu7Zpu2jprroPorh3JuK8dO+T7yvMhvrHhnlzeVE6olLOWKarpHmqDcyHDUMAmIBDbjBtEOsRGur88Kqr8ljIK65hMopYUoc5hDrEwx6giNAbuLzRYIWBDX7ed1QFpoScKym10pWXQRKzScpgYu9RQSBt3CYoHEAARKOn0Vvpa/rEgIks9O3rAx0YdmaQK9dySKKBmpQKJlwUMYCimAHIIn12huL19YVyhijF2PMh5ust7ftkxNwHh8D24DIkozI5TQFZdwRUwJqAJQOJSgXdpqACYA6jDXPlhzWELTleSw5z38kNrZiiZDZxh5hIBYs3SMuQjQVQMG0pSFKJSmOGhTbB6hABAD9OI68bRmI6Kl4i6od8xnRAIp02fJKovxEpjhwDlMIK+aUxvNEeooj6AGotlvOePcNWs8uq65lsZJhIMY1dqi7blXIu7VTTTASqKFANCqcUQEQHhlMYAHSuIrcuTHdjP4bMMCipAYTRz6u/h5HmiqcY2ZLW4szUdJF00RaKSKihSn0Knqfq0CsnJF0YwzFYXKQyOjGtJy1mN72e9bv30XxUhbt045Nw8RKcgiKIo8fRUoaHSMYQESG6wO7JPKmL4SQh4iZyRazB9cRSnh2rqYbpKyJTabRbkMcBWAdQ02AOutSmvzF5Q0vhFG78oy9tXVjRyymbbhDksm+La0SuCNKxKLM9tv2axVykOXRMhUyCKbhMxgAobRr9BCEve4sRsPJJdvZlzPohmo3+VGp5IkYsYhDHSVTFRMywlDcTUTlER6x+igJtXNrrlO5cva5LrYcnTk9EvmCsx6vEvZuVudOFRkZJAdFmrApkVRW2G8wVTiQm8BAB0DUbrx1FZCh7ZIyydd8Xcs4Cyhjv42JGNQMmI+YUERVVEBAPSO/r+gK40xpmHGuHMS3Ryas15bmMMXhaU9KPCySRE0nMkwXklnaDyPMsism5IqQ4kEhSHUAQMG0B0GgL+Ycq63ZXk+r5tY2pKJSSb09vltZ4YqT0bgB1zMsaYwbigYzkSl3gAgBDbxANBAM9TlFEtrOwYSyfbLa1yykAM7bc6MqCzOX4BAF+385NPgqoa79NT7kvPHZ+rXL9lXtfN62rjidvefuuSjIXlDAwBW7WRG0kWPFiqSNFykRFIEzGVXbnDUhdDKkH6Kk/LOsyQ5bN5/8ApNx2MW0Tsghbiuq7HjPnJIl4dE4MY1EAMUeMuBhOqOuhEgAfOHzBA6Q5P+Z3WebSfZCa2Y4g7acSbhvbTp04Ey01HpjtK/FEUyigmoYDbCiJhEoAbUAMAVEOUtyuLd5N1z2Hbcrazua8rX+km5brCmnAxZVkUVJFfQhg4ZVXKJdDCQB1N52oAA4GA+U/ZT/GluwV/RydlXfFT7bG8pbiDFXhspwqYgkikVMpgI2UTT4iSgjwwIIBu1AapeZxflzlb3xm2+bMuCyo+0pts5xRHmuGGdPHAsWRh504aHSXSKnuenVEDGKfUW6Y6eboIHXV4ZQTsrIVl2jMRGkTeyjqPaS5Vx2oyiafGRanT26ACyJHIlU3frIgTTU4DU6rhlbIs9kzkn4fJMHL5eweTrZtWVRMJTrBMRkqRJ2If8wpIKrCP1DG+au5qAUpSgFKUoBSlKAUpSgPmm3bpKHVSQTIdQdTmKUAEw/vH56+lKUB8ubNwIYhUSFA4iYdpQDzh+f/AK/vqHYcxRBYWx1CY3gHz2QZwSSqCDt/wxcHIdY6ogYSFKXQBOIBoAdQB8/XU2pQClKUAqEz+KIK4sqWvlp4/fFkrVjJOKbNSCTmy6L4UBVFUBKJhEObk26GAOsdQHq0m1KA8CkkKfBFMopgABt0DTQP3VG7TvVldc5edvNo9Ruay5tKCXOcQErg541k+A5AD0FAj4hNB+chh9AhUnqqsN/3i52/zAZ/7VgaAtA7ZsociijdM5k/1DGIAiX/AKD81fWlKAV81G7dY5FFkEzmTHUhjFARKP0gPzV9KUBF8mY7t7K9jStg3OVcrGUTKALtjgRw1XIcFEXCJxAdiqSpCKENoOhiFHQakbVJRu1RQWcncKJplIdZQCgZQQDQTCBQAAEfT1AAfQFfWlAaW5rPt+7/AJKC4WZ3RIaTQl2qXHUInzpHUUTnIUQBQCGEDlKcBKByEPpuIUQ3VKUBEbhxpCXRf9rZAmXbxdaz03gxbATE5oR05IVMzsxdu4ViJAqmQd2hSrq9QiICEupSgFKUoBSlKAUpSgFKUoBSlKAUpSgFKUoBSlKAVVWG/wC8XO3+YDP/AGrA1OL6jrpl7LnYux7hLA3E6jnCUTJnQIuRm8FMQRVMmcpinKU+0RKJR1ABCvzz5BeeeVhm7lJ33bl4x8PbERBTJ5m/ytI39MvJJxzeJQj9yh1AR1NHlXHYACIoLgBgKJSgB+lNKUoBSlKAUpSgFKUoBSlKAUpSgFKUoCv5a6Jyak3bOAk/kxhHuDNTuE0SKLuFiCAKAXiAYhCFNuJ1lEwmKOm0ADdruFeHeZP+qxvhax7cMY7eSE5hMPy7MBqI69QSDgA/+K21d2GVBArqhWiZyYpkcbtterMHhXh3mT/qsb4WnCvDvMn/AFWN8LWdSrXYKVotiL0VT1Zg8K8O8yf9VjfC04V4d5k/6rG+FrOpS7BStFsL0VT1Zg8K8O8yf9VjfC04V4d5k/6rG+FrMOciZBUUOUpShqJjDoABRNRNUgKJHKchvQYo6gP86XYKVotheiqerMPhXh3mT/qsb4WnCvDvMn/VY3wtZ1KXYKVotheiqerMHhXh3mT/AKrG+Fpwrw7zJ/1WN8LWdSl2ClaLYXoqnqzB4V4d5k/6rG+FrR2/YZrVlJ2bty65KOf3O9LIy7hBhGlO8clSIkChx5r1jsTKH0a7h9JjCMpEQKAmMIAABqIj81eUV0HBOI3WTVJrpuIYDBr/ANQpdgpWi2F6Kp6sxOFeHeZP+qxvhacK8O8yf9VjfC1nUpdgpWi2F6Kp6sweFeHeZP8Aqsb4WnCvDvMn/VY3wtZ1KXYKVotheiqerMHhXh3mT/qsb4WnCvDvMn/VY3wtZ1KXYKVotheiqerMHhXh3mT/AKrG+Fpwrw7zJ/1WN8LWdSl2ClaLYXoqnqzB4V4d5k/6rG+Fpwrw7zJ/1WN8LWdSl2ClaLYXoqnqzB4V4d5k/wCqxvha9EC8UhEwZGmVR+YFmjAS+n5wK3KP7vSHprMpS7BStFsL0VT1Z7a5cgYgp42+JFswk0DbRFMhuG5IIAILEL5wlAdRASiIiBimDUQ0MKuZ+Us8dt77YEQdLJlGJSEQIcQDXjLdfVSvslfocmfAplrVphH+qTJUTgsTsL+tr+zSX8emf9RcVt61Ftf2aS/j0z/qLitvXyv1NV6ClKVBIpSlAc0ZOs+Oz/ynkcQZBXcOrEs21GtzrQCa5028xIuXiqSRnQFEOIkiVtuKmPUJj6j1agMZw6tifG2Yrka4eb3paDGTtd7IqWPM2w/YRa7tmqTdINVHO0pNCqESORIu0wHKbUNA1uDK+DJu7b6hcuYzyCeyr4hWSkUZ2pHEfspKOOoVQzV03ExBMUDl3EMU5RKJhHr6tI/b3JtvlTJSuW8l5rXuadXt2RtsrNvCkZRjNs5MkYot0QUMcpimTMJzHUOKm4oakAgAODhd62z4+psolZZaRiI5RWfFcDx+Z7rtLGltI3QeLGHPJ3Edu0jGjggio8kVVtmoCPDFNBHcoPEAojqAiEfjOWterixLseRMTYt83Da90QVvNndtySycPMBJqEIQUzqAcyJyiYSmATKFA3+IwBVnXTyZ5CUxfi2y4G92bWbxSowXjpCQhQeMnijdmZqYVmYql6jFOYxdFNxDaCBuqqbzhya8vQ9r3DNM8qvbjn77ui0VVXDK1wKeHdNXqZAdIJJqGDmyKYgYEj6iHDETqmERGqxYkK5fn4yYbjN1mzJmX0cc5hw/mm3bZI8d4ymbjhZe21VuaKt00+Cu1VTXEVAVTOqkIKBoRQptQKQQEoTfHGV86RGSLKsLMNsWWxi77hHLuETg3TpZ5GrM0kzqIO1FABNbcmoUQOmUgFMUxfODQw+pLkwXrfDS+H+VswITdw3TaDqy410wgeZMohk4DVVQrUXBxVVOoBDmEVS9RClDQKsWbxKExkXHd/8AlAZHyBaSTUGnNQNz3naCaW4T7w4e3h66aG1106tOu6hjtt/PUhxQ2WFacpCCPlfMGMcAz0o8b2XcDWZnbiZM3Kjc8sRkVAqTRRRMxTAiJnImOUB87YHoEAEPuvhbF3JtuxbM2PX6dlwjCAklLgtSPKPNp0iCIKpqpoioBU10QTMO8hBExTCAiAaiM1zdhA2VVbcue27wdWfe1mOlXcBPtmqbngcVPhrILIn0BZBQugGJuKI7Sjr1CAxVjyZ567rme3hygcmeWzw0I+t6MYxsSWJj41q8TBN0oRPiKqGcKFAS8QVOooiAF9Ghwu83Z/ZCiVllpqLbzdyhYh5YNz5ZsW0GtpZJkG0a2aQzhyaTt9Z0mZRqDo6v6NyBtoEOJCpbDG6gOFRwvKb5QaUE0yw/sWxG2PUbzG1XifPnZpZygaUNHA8RAC8JLaoJBFM28TgU4gJNS1MrY5NmRCytis8mZqJdNq42dFfQUcjAlZu3TlJMyTRR+54xwWFFM5tATTT3n0MbXTQdtIcmkr/CYYdG8xIAXQFyfKPydr/+a+U+DwuL+/hb93/Pt/w1Fkxr/wCE2wGgmeU5dEHZ2REXdqxpr8tS8EbSiIkqpwRklHqiXyasI67gKokuBz6ejhqaeiuiUeNwU+cbOLtDibNdu7Tr0169Nap25OTRB3HyjIDlBK3I9Q+R2IJOYIhB5rIPUiqkavFfPABURI4XKXUhh84uhi6CBphiJLIaNprhk14s5ljTEoZA6ySCSgMBeK8zAxUBEgCDfhfOJvrDu1ALw3k7IikV1rkcgcpe1neHQc39Oz+UpPIU3cqcmxvGMcvkrdtqN5+QE2zsCG5sk3Tb+acp0jmUExh9Bh0lvKStSxGVx3Ze+a8n3E8lZxui0xfbNpzT5ORQ4bcuqjZm2MUV3B3RjCKhgOmBQS3CX0BLZPkm5EeW/cGJmmeBb4suOUcvXEQa3yKSqDRysKzhik+FbaCRlDqaGMiY5Sn2gI6a1933JdyxHZhunLmPs9Q0I4uMjVqglI2KlKOI1mgiVMrVFyd0Q5UhEu8xQApRN1iGuojk4IufL85/zzNFEupcOGU74SxJZqWTDKGuwkGyLNCoYpji8BEvF3CXqE27XUQ6tdamVay2WM7G29Gx9zzqc1LN2yab2RTZg0K7WAoAdUEQMYEwMOo7QMOmumo1s6+hckYv1FKUqSBSlKA5k5Tn7esP4Ql/WWpTlOft6w/hCX9Zalen4P8AYh+hxeI/diOh4JPmx5hifUFUJuSMoUwaCXiulFydX701SGD6QMA/PW0qRz1lxk67LJFdvY58BQIZyyUKUyhAHUCnKcpiHAOvQTFEQ1HQQ1HXT9GLnvHuf8OO8JXj4eLlRK2J2M9DFw0adiVqMSlZfRi57x7n/DjvCU6MXPePc/4cd4SrZmTV2exXAmdPYxKVl9GLnvHuf8OO8JToxc949z/hx3hKZmTV2ewwJnT2MSlZfRi57x7n/DjvCU6MXPePc/4cd4SmZk1dnsMCZ09jEpWX0Yue8e5/w47wlOjFz3j3P+HHeEpmZNXZ7DAmdPYxKVl9GLnvHuf8OO8JToxc949z/hx3hKZmTV2ewwJnT2MSlZfRi57x7n/DjvCVCrAhp66rryVBSF/zqbezroQhGJkUGAHUQPCxj4TKiLYQE/FfKlASgUNhSBpqAmMzMmrs9hgTOnsSqlZfRi57x7n/AA47wlOjFz3j3P8Ahx3hKZmTV2ewwJnT2MSlZfRi57x7n/DjvCU6MXPePc/4cd4SmZk1dnsMCZ09jEpWX0Yue8e5/wAOO8JToxc949z/AIcd4SmZk1dnsMCZ09jEpWX0Yue8e5/w47wlOjFz3j3P+HHeEpmZNXZ7DAmdPYxKVl9GLnvHuf8ADjvCU6MXPePc/wCHHeEpmZNXZ7DAmdPYxKVl9GLnvHuf8OO8JXomMlAEQWv+5ViD6SmBiT5/pI2KYP5D89M1Jq7PYnAmdPY59zTYE/fV4EfW/HvHabBmmyXFuhxAIruMptEQHqHYqQdPoMH00rquIh46CYJxsW34SCeo9ZzHMYwjqJjGMImMYR6xMYRER9I0rWH9dmylcghViKP9Klxu9E+bM2lKVwTqilKUApSlAKUpQClKUApSlAKqrDf94udv8wGf+1YGrVqqsN/3i52/zAZ/7VgaAtWlKUApSlAKUpQClKUApSlAKUpQClKUApVVP+Heki/dzhAcsmj9wzZsjm3IEBBQUjHMTXac5lCHHUweaGhQ084TYXR9YXYiA9mo/DXRh4BNf5RWP6W/9o+KLi2n/jDy+vguKlU70fWF2IgPZqPw06PrC7EQHs1H4atkIK3p5IzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwWLfVmQORrLnbBuhuZeIuKOcRb0hDbTiismJDCU3+EwAbUDekBABD0V+dvIF5MOconlLX1cGe7quaYi8WyyjSG+U3rhRtKSyjQjZOQTKoYwGAkaVEoCYNwFVbgAhw9A7N6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4a9EsOyUhEyFoQyJh/wAaLFNMwdevUYoAIdYBTIQVvTyM3FT38FwUqjZfNwYtdhbEwgeTKJCuGaq7wAVIgbzQIcxtTKCBiH0MPWIbQHUQEwqr/wCL4p84IbV15Fs9IXKJ2M3Ftf2aS/j0z/qLitvWotr+zSX8emf9RcVt66D9T4l6ClKVBIpSlAQrKeZ8a4Ximkvka5k4wki5KzYNyIKuXT1cwgAJoN0SmVVHUwa7SjpqGulY2Lc7YszMaTQx9c3PXsIoVKSj3TNdk9ZmNrt4rdwQipQHQdDCXQdB0HqGq1WJDuuXkQl1FTO6Z44SVtMrkPNIqZ8sV+dDXq4uwG4GEPO2a6ebuqNhddyseUo+Y33YVjRl7OsdSzhvM27cLx07TjUHCQpJuEVEUiF3KHExDaHEOGcAEA11yvu3+DS4rDqylcSxry/7f5OWPZu486ZQuG7strQqhmkMk1VfrJC0OuoyjROKKTL9F5yrpQxhEEjD+sYKjC2R80wVpZDsVre97WmtC3zZ0ZFObllGUtLxTaSVTBYq7hMyiapNB3ARQ5zAUdDiOtQ5yXqicK34ndd23VAWLa8ted0v+ZQ0GzWkH7nhHV4LdIgnOfYQDHNoUBHQoCI/MA1sm66Tpuk6QPuSWIVQhtBDUohqA9f7q4mzgxufFrLLeKAyrcV329P4enLkM0uN5z17GvUBKgKia3UYqKwLGHhabCmSNs0DzQm9mwN6Yhzli2Hc5hvS7m+QoGULONpx6RRmVyzborJLNG5SARr/AMQ5BInoAl27txgExpxHbY0Rh8rbS/Ml5Tx/h61lr0yTdDSDiEDgnxl9xjKqCAiVNJMgCdVQQARAhCiYQAR06hrQ405Q+JstTT62LOuJx8uRyBXTiJk411GvSoG0AFgQdJpnMnqYA3lAQARABENQqvs0lhleVlgZG7QAYvm1xKRpXADzYZkqTbm//LxgSFwKe7r112+dpW15QN54xsS5mF1Ooor3J0HadxytsiTijwWyDXevzgEzAXgmMVMocQBDdrt0NqNHE02/ggoVYurL0pXIMWyvvFqOFcllz7dV2yORpuMjLhYyr8HMXJpvmyiplGTYpAI04Il3kFHYAkLofdURWcZUi8WNOUY5zxfbmUY5H+TG0Jz1IkSMYa4TR52qyAJ/p9UjGEFFBMYogTbt29bF/gYdvxO7KVxxduUci27L3zyY0b7e+Xly3iyJZr865eeI2/Ijxll0tesxWZEHpN2g7RKlr1iGvYbchU0E0iKmVBMoE3mNuMbTq1EfnH6avDGovQrFDdK8k+UdgiGv0MXy2WLaaXSKybb5MWfEKqCymmxIRHzQUNqXQgjuHcHV1hXrIPKJwdiida2xkbKNv2/KvEyrJNHrsCKAmY20pzB/gKIgOhjaAOg9fUOnMHKIg2bfk9T0/gqOs2TxcrPu7lvd28MsWcWeIynEecz4ifD4wKJmIUyo6lAgFIAgJdJtez2GWvnI9k4JxorfeQcismprpfzJiEg4BAzJNFui7X2iYQ4Q8YrRMFDm3nN5oCFZOZF6fnkvchOqmzlu8bpPGbhNdBchVElUzgYihDBqBiiHUICAgICFfSojiKxFMYYttLHKsmMie2oZpFnd7NnHMikUgnAuo6AIh1BqOgVLq2XNczJilKVIFKUoDmTlOft6w/hCX9ZalOU5+3rD+EJf1lqV6fg/2IfocXiP3YjoS2v7NJfx6Z/1FxW3rWtyFgZiSt+QMRBdWQdvmoGHTnKK6xlt5NR87aZUSGAOsBD0AAlEdlXmLbf8l6M7VlnJilKUApSlAQfKGFMaZkbx6WQLbB8vDrC4jXzdys0esVB01Mg5QMRVPXaXUCmAB2hqA6BWlsvkx4QsCUXn7asoCTTtq6Zu5Z1IOnb92i44YKkWcLKGUVDRJPbuMOzQdm3U2tpUqt2Fu2wm87LLSvrjwLiy6rEgMcStvLlhLUFqaDK1knTdzHGbp8NE6LlNQq5TFIIl3b9RAR1EdaqTK/IgxpOWU6tnHNqtmKk/Lwilxg8mH22SYtXpVlxVMJzmO5OmKocbqUMJus/z105SocuGL1RKjiXoyq4Hkw4Vt2EuiAaWq6dIXmxGLm15CYevXbplsEgN+crLGWImBTCAFIcumuodfXUzfY/tCSuO3btew5VJa1EnKEO44ygc1I4TKmsAFA20+4pChqYBENOrTUakNKsoUvREOJv4kUyXivH+YLaNaWR7ZbTUZxSuEyKiYiiCxddqqKpBKokoACIAchgNoIhroI1o8ecnjEWL38nMWvaxlJWZR5q+k5V84k3q7fQA4ArujqKAl5ofowMBR0ARARqx6Uupu2zmLzssKosjkt4Rx5c7G7bXtNwk9iAcFiE3Us8dtYkFx/TczbrKmSbbtRD9EUuhREoaAIhUhcYXxm6soMdr2yB7eCUCa5nztcP/AHgPee8TeB9/9p/Sbd23/Dpt82ptSoUMK5JBxN/Eiz/F1gymRIvLMhbTZa7YVgtGMJMxj8RFsqOpyAUDbB11HQwlEwAYwAIAY2uJiTGzfFdqL2y3eJOCuJiTlzCkkZNMhnjxVwKZSmOc2heLt1Ew66a9WugTSlTdVtotdlhTkhyQ+T9KXO8ul7Y6h1JKUCbexwSrwsW6fhoPOFWAKg2OcRABERTHUwajqIiI/O6OR5ye7xu6Yvqds2SGcn1SLyTlpc0qzBwchAIUTJoOSEDQoAAaFD5/pGrnpVcOHoTfi6mstm24ez7ejbVt9so3jIlsmzaJKOFFzESIUClAVFDGOcdA/WMYRH5xGtnSlXKilKUApSvKiiaJBVWUKQhesTGHQA/nQHM3Kc/b1h/CEv6y1KtG7MMvsxSZbqj1mabJFErNsquY4A5IURPxSbR0Em5QxQH59oiGpRARV2JH6lw0mWpccVjRz5nBT5sbjhh5M6Ck4iJmm4NJiMaPkAMBwScolVIBg9A6GAQ1rRdFuMu7q2PZDf4KlFK8VDMjgVkLaPTOCGLm0Rfotxl3dWx7Ib/BTotxl3dWx7Ib/BUopVsebU9WVwoKVoRfotxl3dWx7Ib/AAU6LcZd3VseyG/wVKKUx5tT1YwoKVoRfotxl3dWx7Ib/BTotxl3dWx7Ib/BUopTHm1PVjCgpWhF+i3GXd1bHshv8FOi3GXd1bHshv8ABUopTHm1PVjCgpWhF+i3GXd1bHshv8FOi3GXd1bHshv8FSilMebU9WMKClaEX6LcZd3VseyG/wAFVriew7Hkb8zOzkLMgnTeLvhq0YpLRyJyNUBtuFWFJIol0IQVVlVBKXQN6hzekwiN5VVWG/7xc7f5gM/9qwNMebU9WMKClaEt6LcZd3VseyG/wU6LcZd3VseyG/wVKKUx5tT1YwoKVoRfotxl3dWx7Ib/AAU6LcZd3VseyG/wVKKUx5tT1YwoKVoRfotxl3dWx7Ib/BTotxl3dWx7Ib/BUopTHm1PVjCgpWhF+i3GXd1bHshv8FOi3GXd1bHshv8ABUopTHm1PVjCgpWhF+i3GXd1bHshv8FOi3GXd1bHshv8FSilMebU9WMKClaEX6LcZd3VseyG/wAFfRDG2OmqgLNbBtxE4egycUgUQ69fSBfpAKklKY82p6k4UHRH8AAANADQApX9pWRcUrUTl1wFuHRSln/DWcaikgkkdZY4AIAJgTTAxxKAiACOmgah9NabpXs/6s/7tyX5FawyJsathhbX0ZnFNlwuyKJL+yYUqH9K9n/Vn/duS/Ip0r2f9Wf925L8irZWfQ9GRjyqlqiYUqH9K9n/AFZ/3bkvyKdK9n/Vn/duS/IplZ9D0Yx5VS1RMKVD+lez/qz/ALtyX5FOlez/AKs/7tyX5FMrPoejGPKqWqJhSof0r2f9Wf8AduS/Ip0r2f8AVn/duS/IplZ9D0Yx5VS1RMKVD+lez/qz/u3JfkU6V7P+rP8Au3JfkUys+h6MY8qpao3F4XZAWHasve11PTs4WBZLSMg5KgosKDZIgnUU2JlMcwFKURHaUR0Aa57wXyjcFzGXskW7B5QgpaTvy/EFreaRrjnir1JO1IcVFdqIGFNMvNnBTHU2lA6JyCO4olq53WTLFfNVmL5rNOGzhMySyKtsSJyKEMGhimKLfQQEBEBAeoQrjLkW8lSwOTDmvJeSXxpd20eOjRtkh5PySijWJUEFlDKaof8AF14aG7qNoiqP6qoUys+h6MY8qpao/QKlQ/pXs/6s/wC7cl+RTpXs/wCrP+7cl+RTKz6HoxjyqlqiYUqH9K9n/Vn/AHbkvyKdK9n/AFZ/3bkvyKZWfQ9GMeVUtUTClQ/pXs/6s/7tyX5FOlez/qz/ALtyX5FMrPoejGPKqWqJhSof0r2f9Wf925L8inSvZ/1Z/wB25L8imVn0PRjHlVLVEwpUP6V7P+rP+7cl+RTpXs/6s/7tyX5FMrPoejGPKqWqJhSof0r2f9Wf925L8ivRMqWccR1UmUgDrEy0A/TKHXp1mMiAB6fpplp9D0Yx5VS1JdSvgyes5Jok/j3aLlsuQFElkTgchyj6BAwdQhSsGrOTNfUrGJVF+8mZdYwncOJZ63McwdYJt3CiCZA+goFT10+kxh01MOuyrUW1/ZpL+PTP+ouK29ehas5HH9eYpSlQBSlKAUqo8w50lLGuuCxfjmwHF8X7cSJ3yEWV8Rk2ZR6ZykUeO3Jim4SYCbaXQhhOYNoBrpr8MY53uSfv+WxNlvGprFuuNi/l1twpQkjHSUbxOEddFyBE9BIcQAyZyFMG4B6wHqrfhtsLXXZaXHStAGQLDFm/kQvaAFrFJJLP1/lJHhtE1C7kzqm3aJlMXrKJtAEOsK9xN92RPwiNzQV5QcjDuVCooyDSRRWbKqGEAKUqpTCUxhEQAAAdREQqbURYzeUqurzzhZ0Hii9Mp2bLw14oWZHPXjlvGSyShDLNkjHM3OqnxASP5ug6lEQ+gakNr5Gse8HjmHt+74KQmI5FJWSjWcki4cseIXUoLJkMJk9fm3AGtLytsF12WkkpVXZyzh0SJQMFb1nvbxvW73SjK3bdZuE25nZ0096qqqynmooJl0E6ggOm4Or0iGnsfOd9jk5liTM+KUrPmJ1i4fwD6OmSykfIlbgUV0eJwkjpLEA4G2mJoJQEQN6NYcaTsJutq0umlR+GyHYFxz0hatvXzb8pNRPVIRrKTQXdNOvT9KkQwnT6/rAFeOkjHfy0xtry9tz5XkxUBjH/ACohzl0KYmA4JJbtx9olMA7QHTaOvoGptRFjJHStaFy24aLczhZ+NGNZCoVy8B2nwERTEQUA6mu0u0QEB1Hq0662CahFSFVSOU5DgBimKOoCA+gQGpIPVK52meWK3in0xMt8M3k/x9b86NuSN4oc2Bum7K4BuqcjY6gOFUE1jbDKkIIalPprp17PIHKff21eNy2nYuFbsv1Ox2yDm6H8Us1RSj+KnxiopguoUzlYEdDimmAiAGIHpHqpiQl7kRe1K0tlXfB5As+Fvm2lzrRM+wQkWShyCQxkVSAcu4o+gdBDUPprdVf1KClKUApSlAU/kvLE/iy5RhreemQbyCJZJRMECqFBU5jEMIbh6teEBhAOoREw+kRGlQTlOft6w/hCX9Zald3h+C4ebKhjjgTb/g5k3ip0Ebhhidh0JbX9mkv49M/6i4rb1qLa/s0l/Hpn/UXFbeuG/U6S9BSlKgkUpSgOZ8iXpEYJ5VxMnZIBSNsq87Na26ncJkTmaR0i1eLKgi5OUogiVUjnzTmECiJBD5hEIDZOUz3Dm2TtWxM13zkSyV7In3TmQlW7cYo8gkduBEWThFsiCwpJqm36GOAcQga6gNdqKJprEFNVMpyGDQSmDUB/lRNNNIgJpEKQheoClDQA/lWbgdvJmijXQ4Hf2vjTFnJGwaSOsSyY5vdbyAfzNwXC0VPGNXfMlFweSREDpmd6nEyaZFjgkUyhdRACgFVRc9x4+8k8xp3PcEHM245vOx5l8aFt1SHYPo3nBCO3LVpvOY6RgKYorlMbimATAIgICP6nqJpqkFNVMpyG9JTBqA/yqI5NxjCZRhI+CmHjxmlHTMbNJKMxIU4qsnJHCZBExTBsEyYAIAGugjoIems4pNq5fnItDN6nF+T3+IbrcZkujk7to5W2WuFJWOuOQgUgSiln3mmYoiBNEzuE0AcaiAbikMUoj6AC2U8c2JjLlA8n1pj6zom303NvXAxdDHsiJGcIkat1ClVOUNT6KCJ9TCI7jGHXUw69SppJJAJUkykAREwgUNNRH0jXqrqV8fz1KuZ8DnLlGzgYrzVizO1xxrlSyoRpNQM9ItkFFhiOelbmQdKkIUw8Hc2Ehj9W3eX066Di3fyj1MwHuDH3JnJ5XJpWnMuJK5Y4VObR74Wogxat19OGo5UUOA7QN5pS6+kB06WEAMAlMACAhoID89eUkUUCcNBIiZdddCFAA1/lUuB2ux+pCiVnNHCNoSHJ9ng5Olp8n+HYFvm35pqvJoRzUEJKGYJNlCyoSYgQDJ7ziCZyqiHEUOUS7qwX2NLCieS4bI8fZcQndiuWk3ozXMkzPgWLdwoFEFhDeAAlqQCgOmhjdXWOvfZG6Cah1U0EyHU/XMUoAJv+o/PX0quDy5lsXocQX9a8u0zfN8khjCPRtTMFwtL8VdpkDmraOTEFJlqYdQEBWXbIgAAAh/7w2umoAPX9j3vauQIM07Zr/nccg8dRon5uojtXbLHQWJtOUo+aomYuumg6agIhoNSCtRa9qwlnRqkTANlEW6zty/V4q51jqOHCxlllDHUETCJlDmHrHQNdA0AAALwwXWViivI4V5SOT8f5XsqSjGt43JZ2TbTnFWDTFTVQFUJ2TRe8RsZyyBADO0VygmqByiCYFUATCYS61vMwco605XKd4YRkMsW5heFZFbJXfLosxNPTrtZqnvI0OBBSRIRLRIzg+9UBKUClAoAYe4RboCsDgUE+KAaAptDdp9Gvpr6VRyon8fzUtiLoRTFCFjNcZWs1xksRW0kYhqnCKEMcxTsgTKCRgE/nDqUAHUesfnqV0pWqVisMnzFKUqQKUpQHMnKc/b1h/CEv6y1Kcpz9vWH8IS/rLUr0/B/sQ/Q4vEfuxHSkpFyVrSTwQjHz+Mfu1HaKzREy50Dqm3qJqJl1PpvMcxTFKIaDoOggAm1/lK2+w7n92pH8irapXh4OOaVkUNp6eLhU3amVL5StvsO5/dqR/Ip5StvsO5/dqR/Iq2qVbPqnv4K5R1dvJUvlK2+w7n92pH8inlK2+w7n92pH8irapTPqnv4GUdXbyVL5StvsO5/dqR/Ip5StvsO5/dqR/Iq2qUz6p7+BlHV28lS+Urb7Duf3akfyKeUrb7Duf3akfyKtqlM+qe/gZR1dvJUvlK2+w7n92pH8inlK2+w7n92pH8irapTPqnv4GUdXbyVL5StvsO5/dqR/IrFZX3AyLl+zj2066cRbgGj5JGAfnO1XFJNYElSgjqQ4pLJKAU2g7FCG9BgEbkqqsN/3i52/zAZ/7VgaZ9U9/Ayjq7eT5eUrb7Duf3akfyKeUrb7Duf3akfyKtqlM+qe/gZR1dvJUvlK2+w7n92pH8inlK2+w7n92pH8irapTPqnv4GUdXbyVL5StvsO5/dqR/Ip5StvsO5/dqR/Iq2qUz6p7+BlHV28lS+Urb7Duf3akfyKeUrb7Duf3akfyKtqlM+qe/gZR1dvJUvlK2+w7n92pH8inlK2+w7n92pH8irapTPqnv4GUdXbyVL5StvsO5/dqR/Ir0S4U1REqMDcpz/MU1vvk9evT0nSAvz/ADjVsUpn1T38E5T/ANuxVKmGGF6HGdvQHjR2poRBqiuXVugAeaRQQ3FE+4TmHaOgbgABHTcKrWpWT4/iPhE0aLhJPxhtFKUr4z6BSlKAUpSgFKUoBSlKAUpSgFVVhv8AvFzt/mAz/wBqwNTi+pqetuy524rXtg1xy8XHOHjKHI44B5BZNMTFblU2n2mOIbQHaIaiGtcU8kTl42tnXPN12RjfF9wKK3rP+U8k8fKpIJQ8a3gY1mZVTaZQVTi7acEpSgQolVRMIgImKAHedKUoBSlKAUpSgFKUoBSlKAUpSgFKUoCBXJk9wyknERadvpzS7I/CdrLveaNklNNRTBQE1DHOXq3bSaAI6a7gMAafpSyL3fW57zr+BqL2ccysCmucdTrOHSpx+kxnChjD/MREa3Vc5z5kXNOzQ9bD+m8LKVyKC1r4txf9NIzulLIvd9bnvOv4GnSlkXu+tz3nX8DWDSoxZlXtsWyPB/KWsX3Gd0pZF7vrc951/A06Usi931ue86/gawaUxZlXtsMjwfylrF9xndKWRe763PedfwNOlLIvd9bnvOv4GsGlMWZV7bDI8H8paxfcZ3SlkXu+tz3nX8DTpSyL3fW57zr+BrBpTFmVe2wyPB/KWsX3Gd0pZF7vrc951/A06Usi931ue86/gawaUxZlXtsMjwfylrF9xndKWRe763PedfwNU3hHFCGBb9yPkKy8a2+V/kaVLIuSHuRUCMU9BMZuhow81MVlFlNA0DQyZdNEyjVrUpizKvbYZHg/lLWL7jO6Usi931ue86/gadKWRe763PedfwNYNKYsyr22GR4P5S1i+4zulLIvd9bnvOv4GnSlkXu+tz3nX8DWDSmLMq9thkeD+UtYvuM7pSyL3fW57zr+Bp0pZF7vrc951/A1g0pizKvbYZHg/lLWL7jO6Usi931ue86/gadKWRe763PedfwNYNKYsyr22GR4P5S1i+4zulLIvd9bnvOv4GnSlkXu+tz3nX8DWDSmLMq9thkeD+UtYvuM7pSyL3fW57zr+Br0nlS/SnKZ1j2FFLXzubXGoopp+4p2hCj/ADMFa+lMWZV7bDI8H8paxfcWdbVyRl1xRJaLMoBBMZJVFUu1VuqXqMmoXr0MA/vEBDQQESiAirniQuCWhZ6WQjHZ0CKuE1TgUR84/N0i6/8AgoB/KlbQ8UrOa5nPmfocV5uXFy+FpvbJ/Ztv/wB1x/WPW8rR2T+zbf8A7rj+set5Xyr0O5M/3f1FKUqSgpSlAKoXlLXXkCNvHF1kWPlQlgJXbJSSEhKmjmbvRNBkZYhQK6KJA1MXTUNB6/n9FX1VMZyxAbKmSMUuZazY24rWgJGUXnUJFNBZAhFWJyICZFX/AIn6XZoBSmEB0HqANatDZbzMp6icFkPry90QzH2QcjWdnZnjG7M9QuUIOSt1/OPHZYloydwXNjIgQyxmhuHwlOIpoJygOpR0Hq65xZ3Knx1edwQsK1hrtjWV0qnRtublIRRtGzZykFTRsqPWG4hTGJxSp7wKIk3dVQe9+TDGssoNTYix1E2xbt02RcVrXI9h0WzNBuquVEWaqjYpiCqIGBUNxCHENQAdA0EIRinA9/oS+NLduPDlwRoWO5Qcys5N5DdyUUZRmkKaKkYxTeDodQ20xQVRIRJMTk2jqAVeyFq0+VRTpcV1Llb/AC+nx/Oxc6fKxx4rcZIolt3n8hqznk0ndYwpghTyfHFDgAtu4mnGDh8Xh8Ld1b9aiefuV9blnWbkdlZDe6VJa1mTqPC5WUAd3ER01wR4LdVcSmJvBQUwNqQyZTGKU4gJgAajnMOcoSbj2K1x4qvG47wg7wbTr6Yd36mEW8at5IqxCx0eDkECmMiBQAqySYEADDvE+msvvGyeULBYlyngS1MM+UhbsfTjuIuAsyyQaC1kllVzkWIqoVUrlMVTJl80SGNsETlKAjUqGFMq506KFqxr+np+alvXRym7Us6TNbYWteN2SUTHN5C4VLciOdpQyKpAMU7k24oAIl1OCaYHU2AJtmlTO1Mt2Te822hLWkTyHPLfa3M2dJp/oFmLhQ6aRimHr3apm1KIAIfP19VU+lHZrw9d9/OLVw8pe7S+gZSUc5ZSjNrzJ8kwQZnbvAcHKIJaoAoCiYKdRjBt1rSWVjjNPJ9lLRNbuOEr9Inj1na7xWPlmzIjOSQcqrbjlXEmrYwuBDemBjgCY/o+sAGt1WGymzFFzXL6P+vraSuX5U5Fr+xXDWhaFwyMDfaskm6dEhFFRT4AnSKBVCqAVMSqpmOpqBtEtDdVdBVy1Y+NsyWVA4JeSNgBJSdrSUyncjRlJticyTfisAOCGUUAqqZN4GEpTCfQQ0KI6gF6Yvv13kOJl5N3CJRgxk/JwpCpPSuyLkaODIgsChSgXU2wREga7DbiCO4o6REl8C8iOJ/7+r/j+F/2V9eWZMlQ3KNsHFaOP1I207gdP01p9y6bqhJCjHnXBNBIhxUSAp9NxlALrtEAAQ6xx8kTXKNlbwu3yHuO3bBtGy49FZKSnoYzsk45Mjxlf0gqplRbphtIY5dwgbf6dNAk2TrIue4sxYhuuHjQXi7WkZZeWX4yZBbkXj1EUh2mMBj6qGAuhQEQ11HQOuqczNaOar6zRLjeWA53IeM4gjMttwzK7I1hGPFgICi7l82VWKo4OVXQpCqfowAgjsHduGYbHYZzHHCnba+fLT+FbZaYshyob+u9njNNK/rKxE3u+yT3U8mriZ84buHRTkILNrxVkiAAAJlDbjCbYcmmug63vycsjXFljDkBfd1MmbeSfg4IqdkQ5WzoqS6iRHKIH84ElSkKoXd16HCqfvGxr5d5Qg8tXFyaELzhXFmkgi2mZ/GrrW08Ksoc4kI4OVsciqZiJmOkfUAJpoYNKs/kt4+urGmImlvXhHNol6vIv5JOGbOucIwzdw5OqkxIoGhTAkQ4F1KAF1101DrFFdu8iJOJif5W2f3/AB/XX0/stulKVmfaKUpQClKUBWty/tJJ/wDdT/op0pcv7SSf/dT/AKKdKofW/RfRexL7LKJLeRIb9Yqzkpg+gQXOAh/Ia3dbe4ccXEzkXL+xlIxVu+WM5Vj5FdRAiSxxEVDJKkIoIFOYRMJBIOhhMIDoIFLqvJDLvZ+0PeJ14GtXKjh5WHNh42ROV9RpW/Bux9zzSvXkhl3s/aHvE68DTyQy72ftD3ideBpci6PQnMSa4dVueaV68kMu9n7Q94nXgaeSGXez9oe8TrwNLkXR6DMSa4dVueaV68kMu9n7Q94nXgaeSGXez9oe8TrwNLkXR6DMSa4dVueaV68kMu9n7Q94nXgaeSGXez9oe8TrwNLkXR6DMSa4dVueaV68kMu9n7Q94nXgaeSGXez9oe8TrwNLkXR6DMSa4dVueaV68kMu9n7Q94nXgaj9srZIuqauyCj7XtpNxZ0wnCPjLT64EUXPHtHwGSEGYiJOE+SKImAo7ynDTQAMZci6PQZiTXDqtzfV/ClKQNpCgUOsdADSvfkhl3s/aHvE68DTyQy72ftD3ideBpci6PQZiTXDqtzzSvXkhl3s/aHvE68DTyQy72ftD3ideBpci6PQZiTXDqtzzSvXkhl3s/aHvE68DTyQy72ftD3ideBpci6PQZiTXDqtzzSvXkhl3s/aHvE68DTyQy72ftD3ideBpci6PQZiTXDqtzzSvXkhl3s/aHvE68DTyQy72ftD3ideBpci6PQZiTXDqtzzSvXkhl3s/aHvE68DXpOzMsqnKmrFWk2II+cqWacriUPpAnNCbv8ApuClyPoxmZC/5w6oricYvHdxSh2rc6pSrJlMJA10HgJjoP79BD/zSuhbSs1ha8WLM6nP3bhUXLx2qmAGXWEAATAXrAhQKUpSlDXQpSgIiOoiraHhXZzZ8Mz9cSiaghtS+JIaUpX2nmxSlKAUpSgFKUoBSlKAUpSgFVVhv+8XO3+YDP8A2rA1atVVhv8AvFzt/mAz/wBqwNAWrSlKAUpSgFKUoBSlKAUpSgFKUoBSlKAUqv5y7ZmQknLGBfhHtGK5m53BUiKKrql6jgXeAlKUo6l6yiIiA+gADdque3f29l/VmPh6+V8VCnYk3pufPFxMKdljZatKqrnt39vZf1Zj4enPbv7ey/qzHw9Rm1S+25XNQ9H23LVpVVc9u/t7L+rMfD057d/b2X9WY+Hpm1S+24zUPR9ty1aVVXPbv7ey/qzHw9Oe3f29l/VmPh6ZtUvtuM1D0fbctWlVVz27+3sv6sx8PTnt39vZf1Zj4embVL7bjNQ9H23LVpVVc9u/t7L+rMfD057d/b2X9WY+Hpm1S+24zUPR9tycX1HXTL2XOxdj3CWBuJ1HOEomTOgRcjN4KYgiqZM5TFOUp9oiUSjqACFfntyDs/crbOXKRve2b1ZxNtQ1uS55a/CNY79KrIpx6EUjHAZU6gIAJ2HODcMCiJkXAAYAMUodo89u/t7L+rMfD1o7ftUbVlJ2btyccxz+53pZGXcIMWBTvHJUiJAoceb9Y7Eyh9Gu4fSYwizapfbcZqHo+25eNKqrnt39vZf1Zj4enPbv7ey/qzHw9M2qX23Gah6PtuWrSqq57d/b2X9WY+Hpz27+3sv6sx8PTNql9txmoej7blq0qque3f29l/VmPh6c9u/t7L+rMfD0zapfbcZqHo+25atKqrnt39vZf1Zj4enPbv7ey/qzHw9M2qX23Gah6PtuWrSqq57d/b2X9WY+Hpz27+3sv6sx8PTNql9txmoej7blq0qque3f29l/VmPh69EkLvTHcF8SSg/MCjVmJfT8+1AB/wDmmbVL7bjNQ9H23LTpUFYZUhGKIs7vfIMpBE20dhTbFy6AIKEDrEoDqIbRERASj1iGgiq64qT8Ykv7NVPlv/kiNxxjGPJCYwiPyxJh1j8wPVtKy6w4z9aS/jMp/wDdWrMr4pf+iOcKUpViRSlKAUpSgFK4/wCWFD4xneUDiJjluxJi74EIS5FPk2LjHb9bjALLYoKLX9IJQ69R0EA1DX5hCOWJNPcCxmYcm4WxPdtv44iYmNND29dKbtkm9mecH5yq0buBFZFIyaqRRNoUBOHUAgXQNFBarTRS7Vadx0rlS+OVBlGwZmAxre7rEdnXlMMXNwO305MuCQ8dHAuCbdsUxuGd07P5wG2iQheGYwAIaBX1tflbXrk+3LUhMYWtazy+rgm5aHdqqSp3EGyRjdouHxFkigo4SOVVuKRS7REVg1N5oiMXGRhxep1NSuLbeuC5+kG7VMrWPD/LxstWkxUQYyTkWaSvyckVJ6gcATOcNoFOCagCXUwlMA6VYsZmjlF3k6d37j7GtsyOP2NyKwIRq7lZOefN0HYtXMgifUG5ClOU5gRMXcYiZh3gOgCcDDgZ0bSuObGzDygrSSvKKk422riuS5snurYthEZN6Zqzc83FZYVjKlExGaSKInKRINwm3gAF11qWXHyjMwYxgciw+Q7PtuSu+y4VlcMa4hTOEo2YZuXJkAJw1RMogqU6ZiiG84DqBg6uqlxjDZ01SotjZzkp7a6bzK8Zb8dOrrqqczhHCq6DduJtUkzKKlKJ1Sl0A5gKBRMA7Q0qpeXI2ycHJ9uyZx9kVK120XDvHEqQsZx3L5ECl/RIr8QvNtQA4CcCHN5wabRDUYStdhVQ2uw6DpWDAmMeDjjnMJjGaIiIiOoiOwKzqqQKUpQClKUApSlAUlm5y5RutoVFwoQox6Y6FOIBrxFKV885ftY0/hyf9VWleO49vMx/U5U5vEZcsZ+tJfxmU/8AurVmVitExbu5ZocBBRKXfGOUQ0EOI4OqX/yRQgh+4QGsqvXwf6pHUFKUqxIpSlAKUpQEFn8YBOZgtHK/y4KHkrFSsZzDm27nPPRbjv4m4Nmzm/o2m3b/AEhp15eW8fhlPHkxYJpYYwJYiROdghxuFsVIp+puLrrs09IempfSptZNrKiyfhK4blyDEZZxve7G2bpj4tWCdmkoUJRm/jzqAqCZ0uKkYhyKgBynKcPSYogID1R9fk03e3hLPlYjNMgfIlpSL6RC45OPBy1e8+6nbY7EFCFTbCUCAmmRQOECZNphEBEb9pU3mSo2jn22uS9crN+/nryy4a4ZiUvaJvN05CEK2IBmSBUwaJkBYdqY7dCm1ESlAoDvNqcf4hyZb7jHrm2bdzm9iscvbmNcy0I2igJJFFRxzlZknIlWAStlHAiYQ4Qn2mMTfoIjXQdKX2L7Oe5vku3Q+k7hlojLoxq6t4Fvm1zlhCHPDyRkDIuCLCKujtBRMxi7BKmJQMPnCPXX0c8mK4rqjr7kcj5ORlbsvlkwihfsoUGrKMYNFhWTQQbCsY5txzKGMY6oiInDTQC6D0BSl9i+yEkQyJ0zmcA+V8hC2wUhmx0kAJ8rC6EQOQ4arGHggYDAbaQv6PbvExtn1zHjwMtYsunGgy/yV5Sxi0dz3m/H5vxC6b+HuLv0+jcGv0hUxpUW/Ei3nafCPa8xYNmO/fzdEiW7TTdtKAa6fN6K+9KVBApSlAKUpQClKUBR2cv2safw5P8Aqq0rf5Jsqcu+4ivIRg6dJtGxGqooo7wKpqY+0R16h2qFHT94UrynG8NOj4iOKGFtWnNmyo4o20joees2LnnJX5l3TJ4UoEFw0OBTHKHoKcpgMQ2nzCJRENR0ENR10/RgXtvcP3WXh6m1K9xFIlxO1r3R6OKTBE7WiE9GBe29w/dZeHp0YF7b3D91l4eptSoy0vo9XuVwJfQhPRgXtvcP3WXh6dGBe29w/dZeHqbUplpfR6vcYEvoQnowL23uH7rLw9OjAvbe4fusvD1NqUy0vo9XuMCX0IT0YF7b3D91l4enRgXtvcP3WXh6m1KZaX0er3GBL6EJ6MC9t7h+6y8PTowL23uH7rLw9TalMtL6PV7jAl9CE9GBe29w/dZeHqFWBDSd1XXkqCkLvlk29nXQhCMTIpNAOogeFjHwmVEUBAT8V8qUBKBQ2FIGmoCY111VWG/7xc7f5gM/9qwNMtL6PV7jAl9Df9GBe29w/dZeHp0YF7b3D91l4eptSmWl9Hq9xgS+hCejAvbe4fusvD06MC9t7h+6y8PU2pTLS+j1e4wJfQhPRgXtvcP3WXh6dGBe29w/dZeHqbUplpfR6vcYEvoQnowL23uH7rLw9OjAvbe4fusvD1NqUy0vo9XuMCX0IT0YF7b3D91l4enRgXtvcP3WXh6m1KZaX0er3GBL6EJ6MC9t7h+6y8PXomMUQH9LeE+qUfSUeaF+f6SoAIf+amlKZaX07vcYEvoYcVFMIVknHxqHCQT1EAEwmMYR6xMYwiImMI9YiIiI0rMpWyShViNUklYhSlKkkUpSgFKUoBSlKAUpSgFKUoBVVYb/ALxc7f5gM/8AasDU4vqzIHI1lztg3Q3MvEXFHOIt6QhtpxRWTEhhKb/CYANqBvSAgAh6K/O3kC8mHOUTylr6uDPd1XNMReLZZRpDfKb1wo2lJZRoRsnIJlUMYDASNKiUBMG4CqtwAQ4egAfpfSlKAUpSgFKUoBSlKAUpSgFKUoBSlKAUpSgFKUoBSlKAUpSgFKUoCCzOTvkjM1r4i+ROL5SwMvN/KHOdvN+YrM0+Fwtg79/PNd24NvD00Nu1CS3PdlrWVDrXFeVyxUDFNtOM+k3ibVul/wDsooIFL/Ma5x5QmXsbYU5VOKrvyndrO3YdWzbqYkduinEgrncxJik8wojqIEOPo06qrTlF8pXBeWZvFcjZ12Y5fwLSclQNed4M3juChZFFkmciBmoKIJLOlE19yRljAUmw4kHf1UB1Le+cbatdpYkxDLxk9DXvcSMEnJtpRPmrdM7ZwsLkFCgYihQ5uIaalDztdwaaDK7Wv2xr5hD3LZV6QVwRCZzkPIRcii7bFMT9cBVTMYoCX5w16vnr8uY6exQpjJ5E5EOyuK1o3lHM5OXYMbbUi0DRq8YY6TkIoROdJooZNRXZqbiJkOPXuEKl2bEbSvzpsuzkstGymPjRdnsbsfW9GqKxL1ZGTUVfCVFsBOeAmxOlzgETaimIkEwCJqA71uDNlkN8W3hlCxrgg7ya2jFv366cTLIrJnVaonUMgZVLeCZhEglHUBEPoHTSs3pfx3HR9srXfelu23IXW1QXjo+Tl0EFnB1SFMCaJVDFMqICbTzQ6/orgmNStadfZcuXHV/WJcMexwrNsJU+ObIPDwfnJALNJ0uL1Yp3ZCEU4aZSbip7wMJeoBw84u8eQtw3VJS942C0lnePLfbv7VyfboqtZ1omzEyYwj9BYrhMwmMchipkE5VyAfTqKNAfptSopiZ85k8WWfIvLccW+u6gWCykS4XUWVYGM3IItzqKeecxNdomP5wiGo9etSugFKUoBSlKAUpSgFKUoBSlKAUpSgFKUoBSlKAUpSgFKUoBXhVFJcuxZIihdddDFAQ1/nSlAegKUBEwFABN6R09NeU0kkSAmimUhA9BShoAfypSgCSKSBdiKREy666FKAB/8V/FG7dY5DqoJnMmOpBMUBEo/u+j0BSlAfSlKUApSlAKUpQClKUApSlAKUpQH//Z\n", 51 | "text/plain": [ 52 | "" 53 | ] 54 | }, 55 | "execution_count": 2, 56 | "metadata": {}, 57 | "output_type": "execute_result" 58 | } 59 | ], 60 | "source": [ 61 | "Image('./figures/gcn_prediction.jpg')" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": {}, 67 | "source": [ 68 | "Let's assume that shapes of the adjacency matrix, feature matrix are same as the shapes of the matrix used in a 'gcn-node_classification' tutorial." 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": 3, 74 | "metadata": {}, 75 | "outputs": [], 76 | "source": [ 77 | "num_nodes = 50\n", 78 | "num_features = 50\n", 79 | "X = tf.placeholder(tf.float64, [None, num_nodes, num_features])\n", 80 | "A = tf.placeholder(tf.float64, [None, num_nodes, num_nodes])\n", 81 | "Y_truth = tf.placeholder(tf.float64, [None,])" 82 | ] 83 | }, 84 | { 85 | "cell_type": "markdown", 86 | "metadata": {}, 87 | "source": [ 88 | "An implementation of graph convolution layer is same also." 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": 4, 94 | "metadata": {}, 95 | "outputs": [], 96 | "source": [ 97 | "def graph_conv(_X, _A, output_dim):\n", 98 | " output = tf.layers.dense(_X, units=output_dim, use_bias=True)\n", 99 | " output = tf.matmul(_A, output)\n", 100 | " output = tf.nn.relu(output)\n", 101 | " return output" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "We have to implement the readout function as described above.
\n", 109 | "There are two types of the implementation: node-wise summation (nw) and graph gathering (gg).
\n", 110 | "Equations are as follow, respectively.\n", 111 | "\n", 112 | "$$ R_{nw} = \\tau(\\sum_{i \\in G} MLP(H_{i}^{L})) $$\n", 113 | "$$ R_{gg} = \\tau(\\sum_{i \\in G} \\sigma(MLP_1(H_{i}^{L} | H_{i}^{0})) \\odot MLP_2(H_{i}^{L}))$$\n", 114 | "\n", 115 | "Notations :\n", 116 | "* $ \\tau $ : ReLU activation (or other non-linear activations)\n", 117 | "* $ \\sigma $ : sigmoid activation\n", 118 | "* $ \\odot $ : elementwise-multiplication - Hadamard product \n", 119 | "* $ (\\cdot|\\cdot) $ : concatenation\n", 120 | " \n", 121 | "Please refer to the following article for the more detail.
\n", 122 | "Gilmer, Justin, et al. \"Neural message passing for quantum chemistry.\" arXiv preprint arXiv:1704.01212 (2017)." 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "execution_count": 5, 128 | "metadata": {}, 129 | "outputs": [], 130 | "source": [ 131 | "def readout_nw(_X, output_dim):\n", 132 | " # _X : final node embeddings\n", 133 | " output = tf.layers.dense(_X, output_dim, use_bias=True)\n", 134 | " output = tf.reduce_sum(output, axis=1)\n", 135 | " output = tf.nn.relu(output)\n", 136 | " \n", 137 | " return output" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 6, 143 | "metadata": {}, 144 | "outputs": [], 145 | "source": [ 146 | "def readout_gg(_X, X, output_dim):\n", 147 | " # _X : final node embeddings\n", 148 | " # X : initial node features\n", 149 | " val1 = tf.layers.dense(tf.concat([_X, X], axis=2), output_dim, use_bias=True)\n", 150 | " val1 = tf.nn.sigmoid(val1)\n", 151 | " val2 = tf.layers.dense(_X, output_dim, use_bias=True)\n", 152 | " output = tf.multiply(val1, val2)\n", 153 | " output = tf.reduce_sum(output, axis=1)\n", 154 | " output = tf.nn.relu(output)\n", 155 | " \n", 156 | " return output" 157 | ] 158 | }, 159 | { 160 | "cell_type": "markdown", 161 | "metadata": {}, 162 | "source": [ 163 | "We finished preparing necessary functions in the architecture.
\n", 164 | "Therefore, the implementation of the overall architecture is as below." 165 | ] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "execution_count": 7, 170 | "metadata": {}, 171 | "outputs": [ 172 | { 173 | "data": { 174 | "text/plain": [ 175 | "" 176 | ] 177 | }, 178 | "execution_count": 7, 179 | "metadata": {}, 180 | "output_type": "execute_result" 181 | } 182 | ], 183 | "source": [ 184 | "gconv1 = graph_conv(X, A, 32)\n", 185 | "gconv2 = graph_conv(gconv1, A, 32)\n", 186 | "gconv3 = graph_conv(gconv2, A, 32)\n", 187 | "graph_feature = readout_gg(gconv3, gconv1, 128)\n", 188 | "graph_feature" 189 | ] 190 | }, 191 | { 192 | "cell_type": "code", 193 | "execution_count": 8, 194 | "metadata": {}, 195 | "outputs": [ 196 | { 197 | "data": { 198 | "text/plain": [ 199 | "" 200 | ] 201 | }, 202 | "execution_count": 8, 203 | "metadata": {}, 204 | "output_type": "execute_result" 205 | } 206 | ], 207 | "source": [ 208 | "Y_pred = tf.layers.dense(graph_feature, 128, use_bias=True, activation=tf.nn.relu)\n", 209 | "Y_pred = tf.layers.dense(Y_pred, 128, use_bias=True, activation=tf.nn.tanh)\n", 210 | "Y_pred = tf.layers.dense(Y_pred, 1, use_bias=True, activation=None)\n", 211 | "Y_pred" 212 | ] 213 | }, 214 | { 215 | "cell_type": "markdown", 216 | "metadata": {}, 217 | "source": [ 218 | "A loss function have to be minimized in this task is l2-norm." 219 | ] 220 | }, 221 | { 222 | "cell_type": "code", 223 | "execution_count": 9, 224 | "metadata": {}, 225 | "outputs": [ 226 | { 227 | "data": { 228 | "text/plain": [ 229 | "" 230 | ] 231 | }, 232 | "execution_count": 9, 233 | "metadata": {}, 234 | "output_type": "execute_result" 235 | } 236 | ], 237 | "source": [ 238 | "Y_pred = tf.reshape(Y_pred, shape=[-1])\n", 239 | "Y_truth = tf.reshape(Y_truth, shape=[-1])\n", 240 | "loss = tf.reduce_mean(tf.pow(Y_truth - Y_pred,2))\n", 241 | "loss" 242 | ] 243 | }, 244 | { 245 | "cell_type": "markdown", 246 | "metadata": {}, 247 | "source": [ 248 | "Yes, we have completed all necessary preparations for training.\n", 249 | "\n", 250 | "I upload all codes which implement the supervised learning of prediction molecular properties at the 'gnn-molecule' folder.
\n", 251 | "Scripts for preprocessing also exist. Hope you enjoy the graph neural networks from this moment!\n" 252 | ] 253 | } 254 | ], 255 | "metadata": { 256 | "kernelspec": { 257 | "display_name": "Python 3", 258 | "language": "python", 259 | "name": "python3" 260 | }, 261 | "language_info": { 262 | "codemirror_mode": { 263 | "name": "ipython", 264 | "version": 3 265 | }, 266 | "file_extension": ".py", 267 | "mimetype": "text/x-python", 268 | "name": "python", 269 | "nbconvert_exporter": "python", 270 | "pygments_lexer": "ipython3", 271 | "version": "3.6.5" 272 | } 273 | }, 274 | "nbformat": 4, 275 | "nbformat_minor": 2 276 | } 277 | -------------------------------------------------------------------------------- /A gentle introduction about graph neural networks.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SeongokRyu/Graph-neural-networks/e76cb1ad187d19fccad233b42b0806575f8e05e5/A gentle introduction about graph neural networks.pdf -------------------------------------------------------------------------------- /GCN-Thomas Kipf.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SeongokRyu/Graph-neural-networks/e76cb1ad187d19fccad233b42b0806575f8e05e5/GCN-Thomas Kipf.pdf -------------------------------------------------------------------------------- /How_Powerful_are_GNN.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SeongokRyu/Graph-neural-networks/e76cb1ad187d19fccad233b42b0806575f8e05e5/How_Powerful_are_GNN.pdf -------------------------------------------------------------------------------- /Inductive biases, graph neural networks, attention and relational inference.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SeongokRyu/Graph-neural-networks/e76cb1ad187d19fccad233b42b0806575f8e05e5/Inductive biases, graph neural networks, attention and relational inference.pdf -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Graph-neural-networks 2 | ![](gnn.png) 3 | 4 | Image source : https://arxiv.org/abs/1705.07664 5 | 6 | I've been using graph neural networks (GNN) mainly for molecular applications because molecular structures can be represented in graph structures. GNN is interesting in that it can effectively model relationships or interactions between objects in a system. There are various applications of GNN such as molecular applications, network analysis, and physics modeling. 7 | 8 | I will introduce GNN in this repository: from theoretical backgrounds to implementations using TensorFlow. I hope you enjoy GNN from this moment. 9 | 10 | ## References (continually updated) : 11 | ### Geometric Deep Learning and Surveys on Graph Neural Networks 12 | * Bronstein, Michael M., et al. "Geometric deep learning: going beyond euclidean data." IEEE Signal Processing Magazine 34.4 (2017): 18-42. 13 | * [NIPS 2017] Tutorial - Geometric deep learning on graphs and manifolds, https://nips.cc/Conferences/2017/Schedule?showEvent=8735 14 | * Goyal, Palash, and Emilio Ferrara. "Graph embedding techniques, applications, and performance: A survey." Knowledge-Based Systems 151 (2018): 78-94., https://github.com/palash1992/GEM 15 | * Awesome Graph Embedding And Representation Learning Papers, https://github.com/benedekrozemberczki/awesome-graph-embedding 16 | * Battaglia, Peter W., et al. "Relational inductive biases, deep learning, and graph networks." arXiv preprint arXiv:1806.01261 (2018). 17 | 18 | ### Graph Convolution Network (GCN) 19 | * Defferrard, Michaël, Xavier Bresson, and Pierre Vandergheynst. "Convolutional neural networks on graphs with fast localized spectral filtering." Advances in Neural Information Processing Systems. 2016. 20 | * Kipf, Thomas N., and Max Welling. "Semi-supervised classification with graph convolutional networks." arXiv preprint arXiv:1609.02907 (2016). 21 | * van den Berg, Rianne, Thomas N. Kipf, and Max Welling. "Graph Convolutional Matrix Completion." stat 1050 (2017): 7. 22 | * Schlichtkrull, Michael, et al. "Modeling relational data with graph convolutional networks." European Semantic Web Conference. Springer, Cham, 2018. 23 | * Levie, Ron, et al. "Cayleynets: Graph convolutional neural networks with complex rational spectral filters." arXiv preprint arXiv:1705.07664 (2017). 24 | 25 | ### Attention mechanism in GNN 26 | * Velickovic, Petar, et al. "Graph attention networks." arXiv preprint arXiv:1710.10903 (2017). 27 | * GRAM: Graph-based Attention Model for Healthcare Representation Learning 28 | * Lee, John Boaz, et al. "Attention Models in Graphs: A Survey." arXiv preprint arXiv:1807.07984 (2018). 29 | 30 | ### Message Passing Neural Network (MPNN) 31 | * Li, Yujia, et al. "Gated graph sequence neural networks." arXiv preprint arXiv:1511.05493 (2015). 32 | * Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., & Dahl, G. E. (2017). Neural message passing for quantum chemistry. arXiv preprint arXiv:1704.01212. 33 | 34 | ### Graph Autoencoder and Graph Generative Models 35 | * Kipf, Thomas N., and Max Welling. "Variational graph auto-encoders." arXiv preprint arXiv:1611.07308 (2016). 36 | * Simonovsky, Martin, and Nikos Komodakis. "GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders." arXiv preprint arXiv:1802.03480 (2018). 37 | * Liu, Qi, et al. "Constrained Graph Variational Autoencoders for Molecule Design." arXiv preprint arXiv:1805.09076 (2018). 38 | * Pan, Shirui, et al. "Adversarially Regularized Graph Autoencoder." arXiv preprint arXiv:1802.04407 (2018). 39 | * Li, Y., Vinyals, O., Dyer, C., Pascanu, R., & Battaglia, P. (2018). Learning deep generative models of graphs. arXiv preprint arXiv:1803.03324. 40 | 41 | 42 | ### Applications of GNN 43 | * Duvenaud, David K., et al. "Convolutional networks on graphs for learning molecular fingerprints." Advances in neural information processing systems. 2015. 44 | * Kearnes, Steven, et al. "Molecular graph convolutions: moving beyond fingerprints." Journal of computer-aided molecular design 30.8 (2016): 595-608. 45 | * Schütt, K. T., Arbabzadah, F., Chmiela, S., Müller, K. R., & Tkatchenko, A. (2017). Quantum-chemical insights from deep tensor neural networks. Nature communications, 8, 13890. 46 | * Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., ... & Pande, V. (2018). MoleculeNet: a benchmark for molecular machine learning. Chemical Science, 9(2), 513-530. 47 | * Shang, C., Liu, Q., Chen, K. S., Sun, J., Lu, J., Yi, J., & Bi, J. (2018). Edge Attention-based Multi-Relational Graph Convolutional Networks. arXiv preprint arXiv:1802.04944. 48 | * Feinberg, Evan N., et al. "Spatial Graph Convolutions for Drug Discovery." arXiv preprint arXiv:1803.04465 (2018). 49 | * Jin, Wengong, Regina Barzilay, and Tommi Jaakkola. "Junction Tree Variational Autoencoder for Molecular Graph Generation." arXiv preprint arXiv:1802.04364 (2018). 50 | * Liu, Qi, et al. "Constrained Graph Variational Autoencoders for Molecule Design." arXiv preprint arXiv:1805.09076 (2018). 51 | * De Cao, Nicola, and Thomas Kipf. "MolGAN: An implicit generative model for small molecular graphs." arXiv preprint arXiv:1805.11973 (2018). 52 | * Selvan, Raghavendra, et al. "Extraction of Airways using Graph Neural Networks." arXiv preprint arXiv:1804.04436 (2018). 53 | * Kipf, Thomas, et al. "Neural relational inference for interacting systems." arXiv preprint arXiv:1802.04687 (2018). 54 | -------------------------------------------------------------------------------- /figures/gat_attention.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SeongokRyu/Graph-neural-networks/e76cb1ad187d19fccad233b42b0806575f8e05e5/figures/gat_attention.png -------------------------------------------------------------------------------- /figures/gcn_prediction.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SeongokRyu/Graph-neural-networks/e76cb1ad187d19fccad233b42b0806575f8e05e5/figures/gcn_prediction.jpg -------------------------------------------------------------------------------- /figures/graph_conv.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SeongokRyu/Graph-neural-networks/e76cb1ad187d19fccad233b42b0806575f8e05e5/figures/graph_conv.jpg -------------------------------------------------------------------------------- /figures/gru.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SeongokRyu/Graph-neural-networks/e76cb1ad187d19fccad233b42b0806575f8e05e5/figures/gru.png -------------------------------------------------------------------------------- /gnn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SeongokRyu/Graph-neural-networks/e76cb1ad187d19fccad233b42b0806575f8e05e5/gnn.png -------------------------------------------------------------------------------- /tutorials/.ipynb_checkpoints/ggnn-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Gated Graph Neural Network\n", 8 | "\n", 9 | "We have found that GCN and GAT are CNN-like versions of graph neural networks. GGNN, on the other hand, is the RNN-like version of the node updating method. \n", 10 | "\n", 11 | "First, let's look at the message passing neural network (MPNN) framework. The MPNN framework updates the route node with the following formula.
\n", 12 | "\n", 13 | "$$ H_{i}^{(l+1)} = U(H_{i}^{(l)}, m^{(l+1)}) $$\n", 14 | "\n", 15 | "The i-th node, which is a route node, is newly updated through the message state, $m^{(i+1)}$ from the neighboring nodes and previous node state, $H^{(l)}$.
\n", 16 | "\n", 17 | "Updating message state can be written as a general formulation as follow.\n", 18 | "\n", 19 | "$$ m^{(l+1)} = \\sum_{j \\in N_{i}} M(H_i^{(l)}, H_j^{(l)}, e_{ij}) $$\n", 20 | "\n", 21 | "If we know the initial edge information - $e_{ij}$, we can update the message states differently for different relations, for example a single bond, a double bond and an aromatic bond will transfer a different message to the route node.
\n", 22 | "For simpliticy, we will only consider just connectivity between the node pairs, i.e.) $A_{ij} =1$ for connected node pairs, and zero otherwise.\n", 23 | "\n", 24 | "In GGNN framework, message function is defined as simple summation of the neighbor node states.\n", 25 | "\n", 26 | "$$ m^{(l+1)} = \\sum_{j \\in N_{i}} H_j^{(l)} $$\n", 27 | "\n", 28 | "And the gated recurrent unit (GRU) is used for the node updating. Finally, the node updating is re-written as follow.\n", 29 | "\n", 30 | "$$ H_i^{(l+1)} = GRU(H_i^{(l)}, \\sum_{j \\in N_i} H_i^{(l)}) $$\n", 31 | "\n", 32 | "We will implement the updating function in the GGNN framework." 33 | ] 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": 1, 38 | "metadata": {}, 39 | "outputs": [ 40 | { 41 | "name": "stderr", 42 | "output_type": "stream", 43 | "text": [ 44 | "/Users/Lulu/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n", 45 | " from ._conv import register_converters as _register_converters\n" 46 | ] 47 | } 48 | ], 49 | "source": [ 50 | "import tensorflow as tf" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": 2, 56 | "metadata": {}, 57 | "outputs": [], 58 | "source": [ 59 | "def ggnn(_X, _A, output_dim, num_layer):\n", 60 | " num_nodes = int(_X.get_shape()[1])\n", 61 | " input_dim = int(_X.get_shape()[2])\n", 62 | " \n", 63 | " if( input_dim != output_dim ):\n", 64 | " _X = tf.layers.dense(_X, units=output_dim, use_bias=False)\n", 65 | " \n", 66 | " # Message state\n", 67 | " _m = tf.matmul(_A, _X)\n", 68 | " \n", 69 | " # Update node state using GRU cell\n", 70 | " X_total = []\n", 71 | " cell = tf.contrib.rnn.GRUCell(output_dim, name='GRUcell'+str(num_layer))\n", 72 | " \n", 73 | " for i in range(num_nodes):\n", 74 | " mi = tf.expand_dims(_m[:,i,:],1)\n", 75 | " hi = _X[:,i,:]\n", 76 | " \n", 77 | " _, _h = tf.nn.dynamic_rnn(cell, mi, initial_state=hi)\n", 78 | " X_total.append(tf.expand_dims(_h, 1))\n", 79 | " \n", 80 | " output = tf.concat(X_total, 1)\n", 81 | " \n", 82 | " return output" 83 | ] 84 | }, 85 | { 86 | "cell_type": "markdown", 87 | "metadata": {}, 88 | "source": [ 89 | "Let's check if our code is correct." 90 | ] 91 | }, 92 | { 93 | "cell_type": "code", 94 | "execution_count": 3, 95 | "metadata": {}, 96 | "outputs": [ 97 | { 98 | "data": { 99 | "text/plain": [ 100 | "" 101 | ] 102 | }, 103 | "execution_count": 3, 104 | "metadata": {}, 105 | "output_type": "execute_result" 106 | } 107 | ], 108 | "source": [ 109 | "X = tf.placeholder(tf.float64, [None, 50, 58])\n", 110 | "A = tf.placeholder(tf.float64, [None, 50, 50])\n", 111 | "\n", 112 | "ggnn1 = ggnn(X, A, 32, 1)\n", 113 | "ggnn1" 114 | ] 115 | }, 116 | { 117 | "cell_type": "markdown", 118 | "metadata": {}, 119 | "source": [ 120 | "That's right. We implemented a GGNN node updating method simply by using GRU cell.\n", 121 | "\n", 122 | "However, I have implemented it using the for statement here. I wonder if it can be implemented better by using tensorflow. In particular, when a graph neural network is applied to a molecule, a dynamic computational graph should be used when the number of atoms varies. If you know about this, please comment." 123 | ] 124 | } 125 | ], 126 | "metadata": { 127 | "kernelspec": { 128 | "display_name": "Python 3", 129 | "language": "python", 130 | "name": "python3" 131 | }, 132 | "language_info": { 133 | "codemirror_mode": { 134 | "name": "ipython", 135 | "version": 3 136 | }, 137 | "file_extension": ".py", 138 | "mimetype": "text/x-python", 139 | "name": "python", 140 | "nbconvert_exporter": "python", 141 | "pygments_lexer": "ipython3", 142 | "version": "3.6.5" 143 | } 144 | }, 145 | "nbformat": 4, 146 | "nbformat_minor": 2 147 | } 148 | -------------------------------------------------------------------------------- /tutorials/gcn_classification.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Graph Convolution Network\n", 8 | "\n", 9 | "A tutorial for exercising implement graph convolution network (GCN).
\n", 10 | "We will implement the graph convolution to classify nodes in some network graph.
\n", 11 | "It is almost similar to the implementation in T. Kipf's paper \"Semi-supervised classification with graph convolutional networks\".
\n", 12 | "Refer T. Kipf's github - https://github.com/tkipf/gcn" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": 1, 18 | "metadata": {}, 19 | "outputs": [ 20 | { 21 | "name": "stderr", 22 | "output_type": "stream", 23 | "text": [ 24 | "/Users/Lulu/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n", 25 | " from ._conv import register_converters as _register_converters\n" 26 | ] 27 | } 28 | ], 29 | "source": [ 30 | "import tensorflow as tf\n", 31 | "from IPython.display import Image" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": {}, 37 | "source": [ 38 | "Assume that graph inputs - adjacency matrix and node features - are given, and number of nodes and features are 50.
\n", 39 | "Therefore, the shape of node feature matrix is (batch_size, num_nodes, num_features) and adjacency matrix is (batch_size, num_nodes, num_nodes).
\n", 40 | "And the number of labels are 10." 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": 2, 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "num_nodes = 50\n", 50 | "num_features = 50\n", 51 | "num_labels = 10\n", 52 | "X = tf.placeholder(tf.float64, shape=(None, num_nodes, num_features))\n", 53 | "A = tf.placeholder(tf.float64, shape=(None, num_nodes, num_nodes))\n", 54 | "Y_truth = tf.placeholder(tf.float64, shape=(None, num_labels))" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "The equation of graph convolution is as below.
\n", 62 | "$$ H^{l+1} = \\sigma(AH^{l}W^{l}) $$\n", 63 | "We will implement this equation in function named graph_conv. The function will receive l-th node features, adjacency matrix, and the output dimension of node features as inputs. \n", 64 | "\n", 65 | "Actually, original equation is introduced as above, which does not use bias term, in T.Kipf's paper. However, I think using bias is necessary, as below, because the bias term shifts the decision boundary. \n", 66 | "$$ H^{l+1} = \\sigma(A(H^{l}W^{l}+b^{l})) $$\n", 67 | "Therefore, I set 'use_bias=True' at a dense layer in the graph convolution." 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": 3, 73 | "metadata": {}, 74 | "outputs": [], 75 | "source": [ 76 | "def graph_conv(_X, _A, output_dim):\n", 77 | " output = tf.layers.dense(_X, units=output_dim, use_bias=True)\n", 78 | " output = tf.matmul(_A, output)\n", 79 | " output = tf.nn.relu(output)\n", 80 | " return output" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": 4, 86 | "metadata": {}, 87 | "outputs": [ 88 | { 89 | "data": { 90 | "text/plain": [ 91 | "" 92 | ] 93 | }, 94 | "execution_count": 4, 95 | "metadata": {}, 96 | "output_type": "execute_result" 97 | } 98 | ], 99 | "source": [ 100 | "X_new = graph_conv(X, A, 32)\n", 101 | "X_new" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "After single graph convolution, we can check that the dimension of node features is transformed from 50 to 32.
\n", 109 | "We want to build the graph convolution network with three graph convolution layers and softmax classifier, as below." 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": 5, 115 | "metadata": {}, 116 | "outputs": [ 117 | { 118 | "data": { 119 | "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAMCAgICAgMCAgIDAwMDBAYEBAQEBAgGBgUGCQgKCgkICQkKDA8MCgsOCwkJDRENDg8QEBEQCgwSExIQEw8QEBD/2wBDAQMDAwQDBAgEBAgQCwkLEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBD/wAARCAFfANQDASIAAhEBAxEB/8QAHQABAAIDAAMBAAAAAAAAAAAAAAYHBAUIAgMJAf/EAFIQAAAFAwIBBgoFCQUHAQkAAAECAwQFAAYHERITCBQXIZXUFiIxQVVWV5bS0xVRUpOUIzI0Njd1d7O2QmFzgbUkMzVicZGhGCU4Q0VydoKStP/EABoBAQADAQEBAAAAAAAAAAAAAAABAgMFBgT/xAAyEQACAAMHAwMDBAIDAQAAAAAAAQIDUhESExSRodEhUeEEMbFBcfAFIjNhMoEVYsHx/9oADAMBAAIRAxEAPwD6p0pSgFQ/LmU7Uwpjqbydey6xImDQBVQjcgHXXUMYCJIJFEQAyiihiEKGoAJjBqIB11MKoflq2/c81gxaUtK3XVwPrUn4S6TxLQm9w/bsH6LhZJIvXuUFNM4gXQREQ0ABEQoCLPeVXm6xGERfWa+S+padgyjxBq6lGt1Iv38IVwsVJuq+ZgiTaUTKE38NQ4p66CAiGldEPLzs+OlEIOQuuHbSTlwRogzWfJEXVXOQyhEipibcY5iFMYCgGolKIh1ANcNcqLlcY+uawnGQeTzylr5PdiMPxGln2u0brERBNQTLu5NFVoos1IkQ48Ux1EwAEygHWOo2OnY9mr8ovPWYJfHTa6rntS1oM8KB0gM6Ib6OcnORooIbkVVBACcQggbyAA+XUDpKEyZje5bhf2lbmQbalZyK/T4xlLILu2n+KiQ4nT//ACAK/GmT8ayFxtrPYZDtlzPPWwvW0WjLNzvFm4CICsRED7zE1AfGANOoeuvmhiC7Mdo5C5Mj63rtxS3aNZNRseJs61nDdSCF7FuSJs5CWXcKHVXUXMUnCVIQ6ygGUEPFr04fuDkxznJix1j3HkdGOs3jf7BdBqxaCM2jJJyxVHDw6hQ3lblZEOIqGNwwIAF8pdAA+g1ocomwpybnLcuecgbXk4+7XlqRjN/NIkXl1ECpDxEEz7DGMbi6cMoHENPKOtTHJd49HeOrov8AGO+kPBqGey/NONwucc3ROrw9+htu7Zpu2jprroPkrh3JuK8dO+T7yvMhvrHhnlzeFE6olLOWKarpHmqDcyHDUMAmIBDbjBtEOsRGur88Kqr8ljIK65hMopYUoc5hDrEwx6giNAbuLzRYIWBDX7ed1QFpoScKym10pWXQRKzScpgYu9RQSBt3CYoHEAARKOn1Vvpa/rEgIks9O3rAx0YdmaQK9dySKKBmpQKJlwUMYCimAHIIn12huL19YVyhijF2PMh5ust7ftkxNwHh8D24DIkozI5TQFZdwRUwJqAJQOJSgXdpqACYA6jDXPlhzWELTleSw5z39ENrZiiZDZxh5hIBYs3SMuQjQVQMG0pSFKJSmOGhTbB6hABAD6cR142jMR0VLxF1Q75jOiARTps+SVRfiJTHDgHKYQV8UpjeKI9RRHyANRbLec8e4atZ5dV1zLYyTCQYxq7VF23KuRd2qmmmAlUUKAaFU4ogIgPDKYwAOlcRW5cmO7Gfw2YYFFSAwmjn1d/DyPNFU4xsyWtxZmo6SLpoi0UkVFClPoVPU/VoFZOSLoxhmKwuUhkdGNaTlrMb3s963fvovipC3bpxybh4iU5BEURR4+ipQ0OkYwgIkN1gd2SeVMXwkhDxEzki1mD64ilPDtXUw3SVkSm02i3IY4CsA6hpsAddalNfMXlDS+EUbvyjL21dWNHLKZtuEOSyb4trRK4I0rEosz22/ZrFXKQ5dEyFTIIpuEzGAChtGvoIQl73FiNh4JLt7MuZ9EM1G/0o1PJEjFjEIY6SqYqJmWEobiaicoiPWP1UBNq5tdcp3Ll7XJdbDk6cnol8wVmPV4l7NytzpwqMjJIDos1YFMiqK2w3iCqcSE3gIAOgajdeOorIUPbJGWTrvi7lnAWUMd/GxIxqBkxHxCgiKqogIB5R39f1BXGmNMw41w5iW6OTVmvLcxhi8LSnpR4WSSImk5kmC8ks7QeR5lkVk3JFSHEgkKQ6gCBg2gOg0BfzDlXW7K8n1fNrG1JRKSTent8trPDFSejcAOuZljTGDcUDGciUu8AEAIbeIBoIBnqcooltZ2DCWT7ZbWuWUgBnbbnRlQWZy/AIAv2/jJp8FVDXfpqfcl447Pza5fsq9r5vW1ccTt7z91yUZC8oYGAK3ayI2kix4sVSRouUiIpAmYyq7c4akLoZUg/VUn5Z1mSHLZvP/wBJuOxi2idkELcV1XY8Z85JEvDonBjGogBijxlwMJ1R10IkAD4w+IIHSHJ/zO6zzaT7ITWzHEHbTiTcN7adOnAmWmo9MdpX4oimUUE1DAbYURMIlADagBgCohyluVxbvJuuew7blbWdzXha/wBJNy3WFNOBiyrIoqSK+hDBwyquUS6GEgDqbxtQABwMB8p+yn+NLdgr+jk7Ku+Kn22N5S3EGKvDZThUxBJFIqZTARsomnxElBHhgQQDdqA1S8zi/LnK3vjNt82ZcFlR9pTbZziiPNcMM6eOBYsjDzpw0OkukVPc9OqIGMU+ot0x08XQQOurwygnZWQrLtGYiNIm9lHUe0lyrjtRlE0+Mi1Ont0AFkSORKpu/ORAmmpwGp1XDK2RZ7JnJPw+SYOXw9g8nWzasqiYSnWCYjJUiTsQ/wCYUkFVhH7Bjeau5qAUpSgFKUoBSlKAUpSgPWm3bpKHVSQTIdQdTmKUAEw/3j569lKUB6ubNwIYhUSFA4iYdpQDxh8//X++odhzFEFhbHUJjeAfPZBnBJKoIO3/AAxcHIdY6ogYSFKXQBOIBoAdQB5+uptSgFKUoBUJn8UQVxZUtfLTx++LJWrGScU2akEnNl0XwoCqKoCUTCIc3Jt0MAdY6gPVpNqUB4CkkKfBFMopgABt0DTQP7qjdp3qyuucvO3m0eo3NZc2lBLnOICVwc8ayfAcgB5CgR8Qmg+chh8ghUnqqsN/tFzt/EBn/SsDQFoHbNlDkUUbpnMn+YYxAES/9B81e2lKAV61G7dY5FFkEzmTHUhjFARKP1gPmr2UoCL5Mx3b2V7GlbBucq5WMomUAXbHAjhquQ4KIuETiA7FUlSEUIbQdDEKOg1I2qSjdqigs5O4UTTKQ6ygFAyggGgmECgAAI+XqAA+oK9tKA0tzWfb93/RQXCzO6JDSaEu1S46hE+dI6iic5CiAKAQwgcpTgJQOQh9NxCiG6pSgIjcONIS6L/tbIEy7eLrWem8GLYCYnNCOnJCpmdmLt3CsRIFUyDu0KVdXqERAQl1KUApSlAKUpQClKUApSlAKUpQClKUApSlAKUpQCqqw3+0XO38QGf9KwNTi+o66Zey52Lse4SwNxOo5wlEyZ0CLkZvBTEEVTJnKYpylPtESiUdQAQr558gvPPKwzdyk77ty8Y+HtiIgpk8zf5Wkb+WXkk45vEoR+5Q6gI6mjyrjsABEUFwAwFEpQA+lNKUoBSlKAUpSgFKUoBSlKAUpSgFKUoCv5a6Jyak3bOAk/oxhHuDNTuE0SKLuFiCAKAXiAYhCFNuJ1lEwmKOm0ADdruFeHtMn/wsb3Wse3DGO3khOYTD9OzAaiOvUEg4AP8AxW2ruwyoIFdUK0TOTFMjjdtr1Zg8K8PaZP8A4WN7rThXh7TJ/wDCxvdazqVa7BStFwReiqerMHhXh7TJ/wDCxvdacK8PaZP/AIWN7rWdSl2ClaLgXoqnqzB4V4e0yf8Awsb3WnCvD2mT/wCFje61mHORMgqKHKUpQ1Exh0AAomomqQFEjlOQ3kMUdQH/ADpdgpWi4F6Kp6sw+FeHtMn/AMLG91pwrw9pk/8AhY3utZ1KXYKVouBeiqerMHhXh7TJ/wDCxvdacK8PaZP/AIWN7rWdSl2ClaLgXoqnqzB4V4e0yf8Awsb3WtHb9hmtWUnZu3Lrko5/c70sjLuEGEaU7xyVIiQKHHmvWOxMofVruHymMIykRAoCYwgAAGoiPmrxRXQcE4jdZNUmum4hgMGv/UKXYKVouBeiqerMThXh7TJ/8LG91pwrw9pk/wDhY3utZ1KXYKVouBeiqerMHhXh7TJ/8LG91pwrw9pk/wDhY3utZ1KXYKVouBeiqerMHhXh7TJ/8LG91pwrw9pk/wDhY3utZ1KXYKVouBeiqerMHhXh7TJ/8LG91pwrw9pk/wDhY3utZ1KXYKVouBeiqerMHhXh7TJ/8LG91pwrw9pk/wDhY3utZ1KXYKVouBeiqerMHhXh7TJ/8LG91ryIF4pCJgyNMqj5gWaMBL5fOBW5R/u8oeWsylLsFK0XAvRVPVnm1y5AxBTxt8SLZhJoG2iKZDcNyQQAQWIXxhKA6iAlEREDFMGohoYVcz8pZ47b32wIg6WTKMSkIgQ4gGvGW6+qlfZK/Q5M+BTLWrTCP9UmSonBYnYX9bX6NJfv6Z/1FxW3rUW1+jSX7+mf9RcVt6+V+5qvYUpSoJFKUoDmjJ1nx2f+U8jiDIK7h1Ylm2o1udaATXOm3mJFy8VSSM6AohxEkSttxUx6hMfUerUBjOHVsT42zFcjXDze9LQYydrvZFSx5m2H7CLXds1SbpBqo52lJoVQiRyJF2mA5TahoGtwZXwZN3bfULlzGeQT2VfEKyUijO1I4j9lJRx1CqGaum4mIJigcu4hinKJRMI9fVpH7e5Nt8qZKVy3kvNa9zTq9uyNtlZt4UjKMZtnJkjFFuiChjlMUyZhOY6hxU3FDUgEABwcLvW2fX3NlErLLSMRHKKz4rgePzPddpY0tpG6DxYw55O4jt2kY0cEEVHkiqts1AR4YpoI7lB4gFEdQEQj8Zy1r1cWJdjyJibFvm4bXuiCt5s7tuSWTh5gJNQhCCmdQDmROUTCUwCZQoG/tGAKs66eTPISmL8W2XA3uzazeKVGC8dISEKDxk8UbszNTCszFUvUYpzGLopuIbQQN1VTecOTXl6Hte4ZpnlV7cc/fd0Wiqq4ZWuBTw7pq9TIDpBJNQwc2RTEDAkfUQ4YidUwiI1WLEhXT8/GTDcZus2ZMy+jjnMOH8027bJHjvGUzccLL22qtzRVumnwV2qqa4ioCqZ1UhBQNCKFNqBSCAlCb44yvnSIyRZVhZhtiy2MXfcI5dwicG6dLPI1ZmkmdRB2ooAJrbk1CiB0ykApimL4waGHykuTBet8NL4f5WzAhN3DdNoOrLjXTCB5kyiGTgNVVCtRcHFVU6gEOYRVL1EKUNAqxZvEoTGRcd3/AOEBkfAFpJNQac1A3PedoJpbhPvDh7eHrpobXXTq067qGO2389yHFDZYVpykII+V8wYxwDPSjxvZdwNZmduJkzcqNzyxGRUCpNFFEzFMCImciY5QHxtgeQQAQ96+FsXcm27FszY9fp2XCMICSUuC1I8o82nSIIgqmqmiKgFTXRBMw7yEETFMICIBqIzXN2EDZVVty57bvB1Z97WY6VdwE+2apueBxU+GsgsifQFkFC6AYm4ojtKOvUIDFWPJnnruuZ7eHKByZ4bPDQj63oxjGxJYmPjWrxME3ShE+IqoZwoUBLxBU6iiIAXyaHC7zdn+yFErLLTUW3m7lCxDywbnyzYtoNbSyTINo1s0hnDk0nb6zpMyjUHR1fybkDbQIcSFS2GN1AcKjheU3yg0oJplh/YtiNseo3mNqvE+fOzSzlA0oaOB4iAF4SW1QSCKZt4nApxASalqZWxybMiFlbFZ5MzUS6bVxs6K+go5GBKzdunKSZkmij9zxjgsKKZzaAmmnvPoY2umg7aQ5NJX+Eww6N5iQAugLk+kfo7X/wCdfSfB4XF/v4W/d/z7f7NRZMa/+E2wGgmeU5dEHZ2REXdqxpr8tS8EbSiIkqpwRklHqiX0asI67gKokuBz6eThqaeSuiUeNwU+cbOLtDibNdu7Tr0169Nap25OTRB3HyjIDlBK3I9Q+h2IJOYIhB5rIPUiqkavFfHABURI4XKXUhh8Yuhi6CBphiJLIaNprhk14s5ljTEoZA6ySCSgMBeK8zAxUBEgCDfhecTfaHdqAXhvJ2RFIrrXQ5A5S9rO8Og5v6dn8pSeQpu5U5NjeMY5fJW7bUbz8gJtnYENzZJum38U5TpHMoJjD5DDpLeUlaliMrjuy9815PuJ5KzjdFpi+2bTmnycihw25dVGzNsYoruDujGEVDAdMCgluEvkCWyfJNyI8t+4MTNM8C3xZcco5euIg1vkUlUGjlYVnDFJ8K20EjKHU0MZExylPtAR01r3vuS7liOzDdOXMfZ6hoRxcZGrVBKRsVKUcRrNBEqZWqLk7ohypCJd5igBSibrENdRHJwRden51/vqaKJdy4cMp3wliSzUsmGUNdhINkWaFQxTHF4CJeLuEvUJt2uoh1a61Mq1lssZ2Nt6Nj7nnU5qWbtk03simzBoV2sBQA6oIgYwJgYdR2gYdNdNRrZ19C6IxfuKUpUkClKUBzJynP19YfuhL+ctSnKc/X1h+6Ev5y1K9P6P+CH7HF9R/LEdDwSfNjzDE+oKoTckZQpg0EvFdKLk6v701SGD6wMA+etpUjnrLjJ12WSK7exz4CgQzlkoUplCAOoFOU5TEOAdegmKIhqOghqOun6MXPtHuf7uO7pXj4fVyolbE7Gehi9NGnYlajEpWX0YufaPc/3cd3SnRi59o9z/AHcd3SrZmTVs+CuBM7fBiUrL6MXPtHuf7uO7pToxc+0e5/u47ulMzJq2fAwJnb4MSlZfRi59o9z/AHcd3SnRi59o9z/dx3dKZmTVs+BgTO3wYlKy+jFz7R7n+7ju6U6MXPtHuf7uO7pTMyatnwMCZ2+DEpWX0YufaPc/3cd3SnRi59o9z/dx3dKZmTVs+BgTO3wYlKy+jFz7R7n+7ju6VCrAhp66rryVBSF/zqbezroQhGJkUGAHUQPCxj4TKiLYQE/FfKlASgUNhSBpqAmMzMmrZ8DAmdvglVKy+jFz7R7n+7ju6U6MXPtHuf7uO7pTMyatnwMCZ2+DEpWX0YufaPc/3cd3SnRi59o9z/dx3dKZmTVs+BgTO3wYlKy+jFz7R7n+7ju6U6MXPtHuf7uO7pTMyatnwMCZ2+DEpWX0YufaPc/3cd3SnRi59o9z/dx3dKZmTVs+BgTO3wYlKy+jFz7R7n+7ju6U6MXPtHuf7uO7pTMyatnwMCZ2+DEpWX0YufaPc/3cd3SvImMlAEQWv+5ViD5SmBiTz/WRsUwf5D56ZqTVs+CcCZ2+Dn3NNgT99XgR9b8e8dpsGabJcW6HEAiu4ym0RAeodipB0+owfXSuq4iHjoJgnGxbfhIJ6j1nMcxjCOomMYwiYxhHrExhERHyjStYf12bKVyCFWIo/wBKlxu9E+rM2lKVwTqilKUApSlAKUpQClKUApSlAKqrDf7Rc7fxAZ/0rA1atVVhv9oudv4gM/6VgaAtWlKUApSlAKUpQClKUApSlAKUpQClKUApVVP+Heki/dzhAcsmj9wzZsjm3IEBBQUjHMTXac5lCHHUweKGhQ08YTYXR9YXqRAdmo/DXRh9Amv3RWP7W/8AqPii9W0/2w9Pv4LipVO9H1hepEB2aj8NOj6wvUiA7NR+GrZCCt6eSM3FTv4LipVO9H1hepEB2aj8NOj6wvUiA7NR+GmQgrenkZuKnfwXFSqd6PrC9SIDs1H4adH1hepEB2aj8NMhBW9PIzcVO/guKlU70fWF6kQHZqPw06PrC9SIDs1H4aZCCt6eRm4qd/BcVKp3o+sL1IgOzUfhp0fWF6kQHZqPw0yEFb08jNxU7+Cxb6syByNZc7YN0NzLxFxRziLekIbacUVkxIYSm/smADagbygIAIeSvnbyBeTDnKJ5S19XBnu6rmmIvFsso0hvpN64UbSkso0I2TkEyqGMBgJGlRKAmDcBVW4AIcPQOzej6wvUiA7NR+GnR9YXqRAdmo/DTIQVvTyM3FTv4LipVO9H1hepEB2aj8NOj6wvUiA7NR+GmQgrenkZuKnfwXFSqd6PrC9SIDs1H4adH1hepEB2aj8NMhBW9PIzcVO/guKlU70fWF6kQHZqPw06PrC9SIDs1H4aZCCt6eRm4qd/BcVKp3o+sL1IgOzUfhp0fWF6kQHZqPw0yEFb08jNxU7+C4qVTvR9YXqRAdmo/DTo+sL1IgOzUfhpkIK3p5Gbip38FxUqnej6wvUiA7NR+GvIlh2SkImQtCGRMP8AbRYppmDr16jFABDrAKZCCt6eRm4qd/BcFKo2XzcGLXYWxMIHkyiQrhmqu8AFSIG8UCHMbUyggYh9DD1iG0B1EBMKq/8AF+qfWCG1d+hbPSF0idjNxbX6NJfv6Z/1FxW3rUW1+jSX7+mf9RcVt66D9z4l7ClKVBIpSlAQrKeZ8a4Ximkvka5k4wki5KzYNyIKuXT1cwgAJoN0SmVVHUwa7SjpqGulY2Lc7YszMaTQx9c3PXsIoVKSj3TNdk9ZmNrt4rdwQipQHQdDCXQdB0HqGq1WJDuuXkQl1FTO6Z44SVtMrkPFIqZ8sV+dDXq4uwG4GEPG2a6eLuqNhddyseUo+Y33YVjRl7OsdSzhvM27cLx07TjUHCQpJuEVEUiF3KHExDaHEOGcAEA11yvu3+jS4rDqylcSxry/7f5OWPZu486ZQuG7strQqhmkMk1VfrJC0OuoyjROKKTL8l4yrpQxhEEjD+cYKjC2R80wVpZDsVre97WmtC3zZ0ZFObllGUtLxTaSVTBYq7hMyiapNB3ARQ5zAUdDiOtQ5yXuicK36ndd23VAWLa8ted0v+ZQ0GzWkH7nhHV4LdIgnOfYQDHNoUBHQoCI+YBrZN10nTdJ0gfcksQqhDaCGpRDUB6/7q4mzgxufFrLLeKAyrcV329P4enLkM0uN5z17GvUBKgKia3UYqKwLGHhabCmSNs0DxQm9mwN6Yhzli2Hc5hvS7m+QoGULONpx6RRmVyzborJLNG5SARr/vDkEiegCXbu3GATGnEdtjRGH0ttL8yXlPH+HrWWvTJN0NIOIQOCfGX3GMqoICJU0kyAJ1VBABECEKJhABHTqGtDjTlD4my1NPrYs64nH05HIFdOImTjXUa9KgbQAWBB0mmcyepgDeUBABEAEQ1Cq+zSWGV5WWBkbtABi+bXEpGlcAPNhmSpNub/APLxgSFwKe7r112+NpW15QN54xsS5mF1Ooor3J0HadxytsiTijwWyDXevzgEzAXgmMVMocQBDdrt0NqNHE02/ogoVYu7L0pXIMWyvvFqOFcllz7dV2yORpuMjLhYyr8HMXJpvmyiplGTYpAI04Il3kFHYAkLofdURWcZUi8WNOUY5zxfbmUY5H+jG0Jz1IkSMYa4TR52qyAJ/l9UjGEFFBMYogTbt29bF/oYdv1O7KVxxduUci27L3zyY0b7e+Hly3iyJZr865eeI2/Ijxll0tesxWZEHpN2g7RKlr1iGvYbchU0E0iKmVBMoE3mNuMbTq1EfOP11eGNRexWKG6V5J8o7BENfoYvlssW00ukVk230Ys+IVUFlNNiQiPigobUuhBHcO4OrrCvLIPKJwdiida2xkbKNv2/KvEyrJNHrsCKAmY20pzB/YKIgOhjaAOg9fUOnMHKIg2bfk9T0/gqOs2TxcrPu7lvd28MsWcWeIynEecz4ifD4wKJmIUyo6lAgFIAgJdJtez2GWvnI9k4JxorfeQcismprpfzJiEg4BAzJNFui7X2iYQ4Q8YrRMFDm3nN4oCFZOZF7fnkvchOqmzlu8bpPGbhNdBchVElUzgYihDBqBiiHUICAgICFeyojiKxFMYYttLHKsmMie2oZpFnd7NnHMikUgnAuo6AIh1BqOgVLq2XVdTJilKVIFKUoDmTlOfr6w/dCX85alOU5+vrD90JfzlqV6f0f8EP2OL6j+WI6Etr9Gkv39M/6i4rb1rW5CwMxJW/IGIgurIO3zUDDpzlFdYy28mo+NtMqJDAHWAh5AASiOyrzFtv7l7M7VlnRilKUApSlAQfKGFMaZkbx6WQLbB8vDrC4jXzdys0esVB01Mg5QMRVPXaXUCmAB2hqA6BWlsvkx4QsCUXn7asoCTTtq6Zu5Z1IOnb92i44YKkWcLKGUVDRJPbuMOzQdm3U2tpUqt2Fu2wm87LLSvrjwLiy6rEgMcStvLlhLUFqaDK1knTdzHGbp8NE6LlNQq5TFIIl3b9RAR1EdaqTK/IgxpOWU6tnHNqtmKk/Lwilxg8mH22SYtXpVlxVMJzmO5OmKocbqUMJus/nrpylQ5cMXuiVHEvZlVwPJhwrbsJdEA0tV06QvNiMXNryEw9eu3TLYJAb85WWMsRMCmEAKQ5dNdQ6+upm+x/aElcdu3a9hyqS1qJOUIdxxlA5qRwmVNYAKBtp9xSFDUwCIadWmo1IaVZQpeyIcTf1IpkvFeP8wW0a0sj2y2mozilcJkVExFEFi67VUVSCVRJQAEQA5DAbQRDXQRrR485PGIsXv5OYte1jKSsyjzV9Jyr5xJvV2+gBwBXdHUUBLxQ/JgYCjoAiAjVj0pdTdtnUXnZYVRZHJbwjjy52N22vabhJ7EA4LEJupZ47axILj+W5m3WVMk23aiH5IpdCiJQ0ARCpC4wvjN1ZQY7XtkD28EoE1zPna4f7YD3nvE3gff+k/lNu7b/AGdNvi1NqVChhXRIOJv6kWf4usGUyJF5ZkLabLXbCsFoxhJmMfiItlR1OQCgbYOuo6GEomADGABADG1xMSY2b4rtRe2W7xJwVxMScuYUkjJpkM8eKuBTKUxzm0LxduomHXTXq10CaUqbqttFrssKckOSHyfpS53l0vbHUOpJSgTb2OCVeFi3T8NB5wqwBUGxziIAIiKY6mDUdRERH13RyPOT3eN3TF9TtmyQzk+qReSctLmlWYODkIBCiZNByQgaFAADQoef6xq56VXDh7E34u5rLZtuHs+3o21bfbKN4yJbJs2iSjhRcxEiFApQFRQxjnHQPzjGER84jWzpSrlRSlKAUpXioomiQVVlCkIXrExh0AP86A5m5Tn6+sP3Ql/OWpVo3Zhl9mKTLdUeszTZIolZtlVzHAHJCiJ+KTaOgk3KGKA+faIhqUQEVdiR+pemky1LjisaOfM9FPmxuOGHozoKTiImabg0mIxo+QAwHBJyiVUgGDyDoYBDWtF0W4y9nVsdkN/gqUUrxUMyOBWQto9M4IYurRF+i3GXs6tjshv8FOi3GXs6tjshv8FSilWx5tT1ZXCgpWhF+i3GXs6tjshv8FOi3GXs6tjshv8ABUopTHm1PVjCgpWhF+i3GXs6tjshv8FOi3GXs6tjshv8FSilMebU9WMKClaEX6LcZezq2OyG/wAFOi3GXs6tjshv8FSilMebU9WMKClaEX6LcZezq2OyG/wU6LcZezq2OyG/wVKKUx5tT1YwoKVoRfotxl7OrY7Ib/BVa4nsOx5G/Mzs5CzIJ03i74atGKS0cicjVAbbhVhSSKJdCEFVZVQSl0Deoc3lMIjeVVVhv9oudv4gM/6VgaY82p6sYUFK0Jb0W4y9nVsdkN/gp0W4y9nVsdkN/gqUUpjzanqxhQUrQi/RbjL2dWx2Q3+CnRbjL2dWx2Q3+CpRSmPNqerGFBStCL9FuMvZ1bHZDf4KdFuMvZ1bHZDf4KlFKY82p6sYUFK0Iv0W4y9nVsdkN/gp0W4y9nVsdkN/gqUUpjzanqxhQUrQi/RbjL2dWx2Q3+CnRbjL2dWx2Q3+CpRSmPNqerGFBStCL9FuMvZ1bHZDf4K9iGNsdNVAWa2DbiJw8hk4pAoh16+UC/WAVJKUx5tT1JwoOyPwAAA0ANAClftKyLilaicuuAtw6KUs/wCGs41FJBJI6yxwAQATAmmBjiUBEAEdNA1D6603SvZ/2Z/3bkvkVrDImxq2GFtfZmcU2XC7Iokv9kwpUP6V7P8Asz/u3JfIp0r2f9mf925L5FWys+h6MjHlVLVEwpUP6V7P+zP+7cl8inSvZ/2Z/wB25L5FMrPoejGPKqWqJhSof0r2f9mf925L5FOlez/sz/u3JfIplZ9D0Yx5VS1RMKVD+lez/sz/ALtyXyKdK9n/AGZ/3bkvkUys+h6MY8qpaomFKh/SvZ/2Z/3bkvkU6V7P+zP+7cl8imVn0PRjHlVLVG4vC7ICw7Vl72up6dnCwLJaRkHJUFFhQbJEE6imxMpjmApSiI7SiOgDXPeC+UbguYy9ki3YPKEFLSd+X4gtbzSNcc8VepJ2pDiortRAwppl5s4KY6m0oHROQR3FEtXO6yZYr5qsxfNZpw2cJmSWRVtiRORQhg0MUxRb6CAgIgID1CFcZci3kqWByYc15LyS+NLu2jx0aNskPB+SUUaxKggsoZTVD/e68NDd1G0RVH81UKZWfQ9GMeVUtUfQKlQ/pXs/7M/7tyXyKdK9n/Zn/duS+RTKz6HoxjyqlqiYUqH9K9n/AGZ/3bkvkU6V7P8Asz/u3JfIplZ9D0Yx5VS1RMKVD+lez/sz/u3JfIp0r2f9mf8AduS+RTKz6HoxjyqlqiYUqH9K9n/Zn/duS+RTpXs/7M/7tyXyKZWfQ9GMeVUtUTClQ/pXs/7M/wC7cl8inSvZ/wBmf925L5FMrPoejGPKqWqJhSof0r2f9mf925L5FeRMqWccR1UmUgDrEy0A/TKHXp1mMiAB5frplp9D0Yx5VS1JdSvQyes5Jok/j3aLlsuQFElkTgchyj5BAwdQhSsGrOjNfcrGJVF+8mZdYwncOJZ63McwdYJt3CiCZA+ooFT10+sxh01MOuyrUW1+jSX7+mf9RcVt69C1Z0OP79RSlKgClKUApVR5hzpKWNdcFi/HNgOL4v24kTvkIsr4jJsyj0zlIo8duTFNwkwE20uhDCcwbQDXTX0Yxzvck/f8tibLeNTWLdcbF/TrbhShJGOko3icI66LkCJ6CQ4gBkzkKYNwD1gPVW/DbYWuuy0uOlaAMgWGLN/Ihe0ALWKSSWfr/SSPDaJqF3JnVNu0TKYvWUTaAIdYV5xN92RPwiNzQV5QcjDuVCooyDSRRWbKqGEAKUqpTCUxhEQAAAdREQqbURYzeUqurzzhZ0Hii9Mp2bLw14oWZHPXjlvGSyShDLNkjHM3OqnxASP4ug6lEQ+oakNr5Gse8HjmHt+74KQmI5FJWSjWcki4cseIXUoLJkMJk9fNuANaXlbYLrstJJSquzlnDokSgYK3rPe3jet3ulGVu26zcJtzOzpp71VVVlPFRQTLoJ1BAdNwdXlENPY+c77HJzLEmZ8UpWfMTrFw/gH0dMllI+RK3Aoro8ThJHSWIBwNtMTQSgIgbyaw40nYTdbVpdNKj8NkOwLjnpC1bevm35SaieqQjWUmgu6aden5VIhhOn1/aAK8OkjHf00xtrw9tz6XkxUBjH/SiHOXQpiYDgklu3H2iUwDtAdNo6+Qam1EWMkdK1oXLbhotzOFn40Y1kKhXLwHafARFMRBQDqa7S7RAQHUerTrrYJqEVIVVI5TkOAGKYo6gID5BAakg8qVztM8sVvFPpiZb4ZvJ/j6350bckbxQ5sDdN2VwDdU5Gx1AcKoJrG2GVIQQ1KfTXTr2eQOU+/tq8bltOxcK3ZfqdjtkHN0P4pZqilH8VPjFRTBdQpnKwI6HFNMBEAMQPKPVTEhL3Ii9qVpbKu+DyBZ8LfNtLnWiZ9ghIslDkEhjIqkA5dxR8g6CGofXW6q/uUFKUoBSlKAp/JeWJ/FlyjDW89Mg3kESySiYIFUKCpzGIYQ3D1a8IDCAdQiJh8oiNKgnKc/X1h+6Ev5y1K7vp/RenmyoY44E2/6OZN9VOgjcMMTsOhLa/RpL9/TP+ouK29ai2v0aS/f0z/qLitvXDfudJewpSlQSKUpQHM+RL0iME8q4mTskApG2VedmtbdTuEyJzNI6RavFlQRcnKUQRKqRz4pzCBREgh5hEIDZOUz3Dm2TtWxM13zkSyV7In3TmQlW7cYo8gkduBEWThFsiCwpJqm36GOAcQga6gNdqKJprEFNVMpyGDQSmDUB/yommmkQE0iFIQvUBShoAf5Vm4Hb0Zoo12OB39r40xZyRsGkjrEsmOb3W8gH8zcFwtFTxjV3zJRcHkkRA6ZnepxMmmRY4JFMoXUQAoBVUXPcePvBPMadz3BBzNuObzseZfGhbdUh2D6N5wQjty1abzmOkYCmKK5TG4pgEwCICAj9T1E01SCmqmU5DeUpg1Af8qiOTcYwmUYSPgph48ZpR0zGzSSjMSFOKrJyRwmQRMUwbBMmACABroI6CHlrOKTaun50LQze5xfk9/iG63GZLo5O7aOVtlrhSVjrjkIFIEopZ94pmKIgTRM7hNAHGogG4pDFKI+QAtlPHNiYy5QPJ9aY+s6Jt9Nzb1wMXQx7IiRnCJGrdQpVTlDU+igifUwiO4xh11MOvUqaSSQCVJMpAERMIFDTUR8o15VdSvr+e5VzPoc5co2cDFeasWZ2uONcqWVCNJqBnpFsgosMRz0rcyDpUhCmHg7mwkMfq27y+XXQcW7+UepmA9wY+5M5PC5NK05lxJXLHCpzaPfC1EGLVuvpw1HKihwHaBvFKXXygOnSwgBgEpgAQENBAfPXikiigThoJETLrroQoAGv+VS4Ha7H7kKJWdUcI2hIcn2eDk6Wnyf4dgW+bfmmq8mhHNQQkoZgk2ULKhJiBAMnvOIJnKqIcRQ5RLurBfY0sKJ5Lhsjx9lxCd2K5aTejNcyTM+BYt3CgUQWEN4ACWpAKA6aGN1dY699kboJqHVTQTIdT88xSgAm/6j569lVwenUti9jiC/rXl2mb5vkkMYR6NqZguFpfirtMgc1bRyYgpMtTDqAgKy7ZEAAAEP9sNrpqAD1/Y972rkCDNO2a/53HIPHUaJ+bqI7V2yx0FibTlKPiqJmLrpoOmoCIaDUgrUWvasJZ0apEwDZRFus7cv1eKudY6jhwsZZZQx1BEwiZQ5h6x0DXQNAAAC8MF1lYoryOFeUjk/H+V7KkoxreNyWdk205xVg0xU1UBVCdk0XvEbGcsgQAztFcoJqgcogmBVAEwmEutbzMHKOtOVyneGEZDLFuYXhWRWyV3y6LMTT067Wap7yNDgQUkSES0SM4PvVASlApQKAGHuEW6ArA4FBPigGgKbQ3afVr5a9lUcqJ/X81LYi7EUxQhYzXGVrNcZLEVtJGIapwihDHMU7IEygkYBP4w6lAB1HrHz1K6UrVKxWGT6ilKVIFKUoDmTlOfr6w/dCX85alOU5+vrD90JfzlqV6f0f8EP2OL6j+WI6UlIuStaSeCEY+fxj92o7RWaImXOgdU29RNRMup9N5jmKYpRDQdB0EAE2v8ACVt6Duf3akfkVbVK8PB65pWRQ2np4vSpu1MqXwlbeg7n92pH5FPCVt6Duf3akfkVbVKtn1Tv4K5R1beSpfCVt6Duf3akfkU8JW3oO5/dqR+RVtUpn1Tv4GUdW3kqXwlbeg7n92pH5FPCVt6Duf3akfkVbVKZ9U7+BlHVt5Kl8JW3oO5/dqR+RTwlbeg7n92pH5FW1SmfVO/gZR1beSpfCVt6Duf3akfkU8JW3oO5/dqR+RVtUpn1Tv4GUdW3kqXwlbeg7n92pH5FYrK+4GRcv2ce2nXTiLcA0fJIwD852q4pJrAkqUEdSHFJZJQCm0HYoQ3kMAjclVVhv9oudv4gM/6VgaZ9U7+BlHVt5PV4StvQdz+7Uj8inhK29B3P7tSPyKtqlM+qd/Ayjq28lS+Erb0Hc/u1I/Ip4StvQdz+7Uj8irapTPqnfwMo6tvJUvhK29B3P7tSPyKeErb0Hc/u1I/Iq2qUz6p38DKOrbyVL4StvQdz+7Uj8inhK29B3P7tSPyKtqlM+qd/Ayjq28lS+Erb0Hc/u1I/Ip4StvQdz+7Uj8irapTPqnfwMo6tvJUvhK29B3P7tSPyK8iXCmqIlRgblOfzFNb75PXr08p0gL5/ONWxSmfVO/gnKf8AbYqlTDDC9DjO3oDxo7U0Ig1RXLq3QAPFIoIbiifcJzDtHQNwAAjpuFVrUrJ+v9R9ImjRekk/WG0UpSvjPoFKUoBSlKAUpSgFKUoBSlKAVVWG/wBoudv4gM/6VganF9TU9bdlztxWvbBrjl4uOcPGUORxwDyCyaYmK3KptPtMcQ2gO0Q1ENa4p5InLxtbOuebrsjG+L7gUVvWf8J5J4+VSQSh41vAxrMyqm0ygqnF204JSlAhRKqiYRARMUAO86UpQClKUApSlAKUpQClKUApSlAKUpQERnL9Oyeqx0DEkklmxtjhRVzwEUz+cgHAhxMYPPoXQNdNddQDW9IV2eqMR22r3Wo5AnFSNBU3WZRZc5h+swqnER/7iNbCuapsyNXlFZb9uDnOfMi6p2aGz6Qrs9UYjttXutOkK7PVGI7bV7rWspS/Nre3BGNMq+ODZ9IV2eqMR22r3WnSFdnqjEdtq91rWUpfm1vbgY0yr44Nn0hXZ6oxHbavdadIV2eqMR22r3WtZSl+bW9uBjTKvjg2fSFdnqjEdtq91p0hXZ6oxHbavda1lKX5tb24GNMq+ODZ9IV2eqMR22r3WnSFdnqjEdtq91rWUpfm1vbgY0yr44Nn0hXZ6oxHbavdaprCOGIDAt+5HyFZdjxZX+RpUsi5IeYOBGKegmM3Q0a+KmKyiymgaBoZMumiZRq0qUvza3twMaZV8cGz6Qrs9UYjttXutOkK7PVGI7bV7rWspS/Nre3AxplXxwbPpCuz1RiO21e606Qrs9UYjttXutaylL82t7cDGmVfHBs+kK7PVGI7bV7rTpCuz1RiO21e61rKUvza3twMaZV8cGz6Qrs9UYjttXutOkK7PVGI7bV7rWspS/Nre3AxplXxwbPpCuz1RiO21e606Qrs9UYjttXutaylL82t7cDGmVfHBs+kK7PVGI7bV7rX6TIdzlMAr2hH8PXxuDMHMfT+4DNygP8AmIVq6Uvza3twMaZV8cFiQk2xuCPJIsDH2iIkOmoG1RJQPziHDzGD/sPUICICAiqnFJeQjZF+kycGSKosVQwAPlNwkw1/7AH/AGpUr9QUPSJdTeH1Ss6rqZtvf8JT/wARX+YatlWtt7/hKf8AiK/zDVsqylf4L7Hxw+yFKUq5YUpSgFKV+GARAQAdBEOofqoDiaEyzlrKfhdJ2zyqIC1sgxc1ItojGjyOjk0iptHB00mrnnH+0qHXKQomXTUApBV8UPF0q+rm5RLWx/oW2puwLrn74dwqMxLW9arEkgtFpCXRRRUwqFTAnEKchAA5jnEo7Cmqm7vtDLl22jKY5zNyUInKlzpGcMou9Cu4pq0doKCYEXShjCRwzOQpgAxEkzD+T1KI6hW9s6yc3cnu4Y+ab2C5yiSTsSEt6Tcxcm2QeN5ONKsACbniifEQV5wP5QDCYok6yDqFbNJmzULPzKfKeezlw4aRw49ute3bzfun0nIQltg/VWbNSmA7ASKlEUlOKAlWDaB0yFMbUunXsLA5aEM5xfGXxkm0Lrj3cxdD+2mDZpbbjiO1iLORbppIAY6hz8JuCZ9NdFtxRAoAOntxjhPI1qPcTSFwRzMzqNl7qnrjBo4TFGOWleMqREmolFUCnVBPUhRDUuugB11q7ExflluOPoWbx8rHo2Rk+cmnD00k0VSdRrtKVMk6TKVTeAb3iJBIYoHAR126AIg/bZYQ7lln59Tdjy6cXJoulndhZNa/QjoGtzgraqulsiJgADPzgYSAUQEDBwTKjtHdpp11LL/5T9k2BcEtb5rVvO4fBtsk8uF7AwxnbWFRUJxCGcn3FH/d+OJUwOYCeMIAGo1BLkw3kh9YPKThWdvAo9v+XWdW6lzxAOepGjWiIDqJ9E/yiShdFBKPi6+QQEYdmnEObr4unIsO/sa67qaz7NJraLltfQxFvxzfmYEOR20RXIoqqC/EMO5NUFQMQoiQoCIQlCwoYGX1McpbEtuyzmMuCbXjUkrY8L2z5y3ErR/GAACc7dT/AOIcu4uqegH8YuhRAQGp1Z9zNrzteLuxnHSLBvLNiO0W8ghwHJEzhqXiJ6iJBEBAdo9Ya6CADqFcgX/ye8154tqwrYlLOZ2Y3xbbDORilpRVk+PKXKmmmUrRQqRlNjIoI6KCOm8VC6FMBNa6Gs3K15Sl6W1YF5Y5C35SVtBW4ZMoSqLk0e7ScJIGbCVHeUSGFUTEVE4bgIYADUpwLDhVnQrFCrOg5R+Sr9xTi2Zu7HmPFrqkWTJyubR6g3RjyJpGOLlbimAyhC7deGmBjG006tdQiN5ZcyaexsPwFiLQyV75TBqQ8lINTKNGCJWAunjkECmDiGAC6ETEwAInDUdAqz8xwEtdWJL1tiAac6k5e3pFizQ4hScVdVuchC7jCBS6mMAaiIAHnGqou/H+T4Oz8KXlaVr/AE1cWMk26cpbpX6SCjxstH81dESVMYEjKpiIHKBjAQ2wQ16wpDYIbCUYOv2/5C6b4xLlN9FylxWQuyVJLxrQzRGSYPEjKIKGQE5+EqApqlMUDCHigIeXUbfql8EWpfi18ZDzLkS1fBV9erhg0YQZ3aTldrHsUjJpKOFETGT4qhlFDiUhjAUuwNRHWroqsXuVist6ClKVUgUpSgFKUoCKSf8AxZ7/AIhP5ZKUk/8Aiz3/ABCfyyUrmR/5v7lDdW+AliyFHylVWAQ+oQVNqFbGtzNWPLovV3lrqMjpOlBWUaPFDpFTUMIicxFClOIAYR12iXqER0HQdA13gpkH0Zb3a6/da6Sgilq60+n9GrlRw9LDHpWR4KZB9GW92uv3WngpkH0Zb3a6/danr2ej4FyLszHpWR4KZB9GW92uv3WngpkH0Zb3a6/dadez0fAuRdmY9KyPBTIPoy3u11+608FMg+jLe7XX7rTr2ej4FyLszHpWR4KZB9GW92uv3WngpkH0Zb3a6/dadez0fAuRdmY9KyPBTIPoy3u11+608FMg+jLe7XX7rTr2ej4FyLszHpWR4KZB9GW92uv3Wo/bLu7Lqmrsgo+EiU3FnTCcI+MtKKgRRc8e0fAZIQbiIk4T5IoiYCjvKcNNAAxnXs9HwLkXZm4r82lAwnAobhAAEdOsQDzf+RrJ8FMg+jLe7XX7rTwUyD6Mt7tdfutOvZ6PgXIuzMelZHgpkH0Zb3a6/daeCmQfRlvdrr91p17PR8C5F2Zj0rI8FMg+jLe7XX7rTwUyD6Mt7tdfutOvZ6PgXIuzMelZHgpkH0Zb3a6/daeCmQfRlvdrr91p17PR8C5F2Zj0rI8FMg+jLe7XX7rTwUyD6Mt7tdfutOvZ6PgXIuzMelZHgpkH0Zb3a6/da8iWlfxzAVRnb6JRHrOEisqIB/8ATwC6/wD7BTr2ejFyPsyHvWrhxKPjIImUAFSlEShroPCJ1f8AkKVcNvW0zgY/mom50uqcVnLhQgaqqiAAI6f2QAAAADzAUOsR1EVQv0+9+6J2Nm0PpXZ1ZuKUpXUPtFKUoBSlKAUpSgFKUoBSlKAVVWG/2i52/iAz/pWBq1aqrDf7Rc7fxAZ/0rA0BatKUoBSlKAUpSgFKUoBSlKAUpSgFKUoBSlKAUpSgFKUoBSlKAUpSgFK0T297Xj7zi8evJPh3BNMHcmxZ8FQeM2anRIufeBdhdpnKIaGMAjv6gHQdN7QGivqOumXsudi7HuEsDcTqOcJRMmdAi5GbwUxBFUyZymKcpT7REolHUAEK+e3IOz9yts5cpG97ZvVnE21DW5Lnlr8I1jvyqsinHoRSMcBlTqAgAnYc4NwwKImRcABgAxSh9CrlvW2bQcwjS4pPmitxyZIaMLwVFOcPDpqKFT8Qo7dSJKDuNoXxdNdRABW/ZFo2rKTs3blvMo5/c70sjLuEEgKd45KkRIFDj5x2JlD6tdw+UxhEDeUrWXNccNZ9uSt23G95nEwrJaQfOOGdTgt0SCdQ+0gCY2hSiOhQER06gEayo2RZy8c1lo5bjNHqJHCCm0S70zlAxTaCACGoCA6CGtAZNKUoBSlKAUpSgFKUoBSlKAUpSgFKUoBSlKAUpSgFKUoBSlKAoy8v/fNxd/9h3b/AP1w9aPlc3PKhL4/xrb1xZEI+uR4+dOIKwCIIS8q0bIgJv8A2guskRggRRRITnA28+4pSCUdRqwsv8m3Ded38PK5Qth3JvIBJwjHLtZt/HHRIuKYql1aLJbgMKSf52um3q066ipORDybkIhjDsrPmmgRkitKsniF2S5X7dwsiVBUU3fOuOUh0iEKZMDgQdoDt166A5VC4cmX1j63bNujIlzWvJW9yiEbWjpeZesZGZjGgxpzAgq5S3t1nOq6iRTjv0McoGEwlqR5fv8Ay9yYX+RMW2DlK7LubOYy1nUO5nnacpLQbmUllWLgqbhxtKfcQgGSK4PsIcS+QgCA9FjyI+TAMDIWr0XphCyco2m3EcEq+K35+ggdFNwVMFtCKbDm3GLoJzaHPuMUDBu4DksYIt607qstKxCSUdfBiGuM8y+cybqUEhQKnxnLlRRY3DAA4Ybw2D1l2jqNAcrP3+XLUisp4+muk0lpSuJLilFGuSrkhpKVTkEUgJx2nNHKqwtzlVOByiAJEPsAm0BAtMpyuXefry7O48oKWfbFmwZo5zi+cYqrW1Ic04yysvEnUIq8AxRSUBM24oo66AAm3B1PanJWwdZzC5GMZar10a7YoYKXdys2/kXi8cJDE5qVy4WOskloY3ipmKGvX5QAQxbz5IuB78lVJmctiUQcuoxvCvvou4pGOJIsUC7UkHRGy5CuClKIlDiAYdo6a6aUBYeO7ia3fYFtXWxmizDeZiGb9KRK0FqDwqqJTgsCJhEUt+7dsEREuunmqQ1ixUXGwcY0hYdigyYR6CbVq2QIBE0USFApCFKHUBQKAAAB5ACsqgFKUoBSlKAUpSgFKUoBSlKA/9k=\n", 120 | "text/plain": [ 121 | "" 122 | ] 123 | }, 124 | "execution_count": 5, 125 | "metadata": {}, 126 | "output_type": "execute_result" 127 | } 128 | ], 129 | "source": [ 130 | "Image('./figures/graph_conv.jpg')" 131 | ] 132 | }, 133 | { 134 | "cell_type": "code", 135 | "execution_count": 6, 136 | "metadata": {}, 137 | "outputs": [ 138 | { 139 | "data": { 140 | "text/plain": [ 141 | "" 142 | ] 143 | }, 144 | "execution_count": 6, 145 | "metadata": {}, 146 | "output_type": "execute_result" 147 | } 148 | ], 149 | "source": [ 150 | "gconv1 = graph_conv(X, A, 32)\n", 151 | "gconv2 = graph_conv(gconv1, A, 32)\n", 152 | "gconv3 = graph_conv(gconv2, A, 32)\n", 153 | "Y_pred = tf.nn.softmax(tf.layers.dense(gconv3, units=num_labels, use_bias=True), axis=2)\n", 154 | "Y_pred" 155 | ] 156 | }, 157 | { 158 | "cell_type": "markdown", 159 | "metadata": {}, 160 | "source": [ 161 | "A shape of the final output is [batch_size, num_nodes=50, num_labels=10].
\n", 162 | "Finally, we have to set the loss function as a cross entropy loss, because it is the classification task." 163 | ] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": 7, 168 | "metadata": {}, 169 | "outputs": [], 170 | "source": [ 171 | "Y_pred = tf.reshape(Y_pred, [-1])\n", 172 | "loss = tf.reduce_mean(Y_truth*tf.log(Y_pred+1.**-5))" 173 | ] 174 | }, 175 | { 176 | "cell_type": "markdown", 177 | "metadata": {}, 178 | "source": [ 179 | "Or you can use cross entropy loss variants already implemented in TensorFlow.
\n", 180 | "As usually done in supervised learning, we have to minimize the loss function.
\n", 181 | "We do not have data to train in this tutorial, so we will skip the training. " 182 | ] 183 | } 184 | ], 185 | "metadata": { 186 | "kernelspec": { 187 | "display_name": "Python 3", 188 | "language": "python", 189 | "name": "python3" 190 | }, 191 | "language_info": { 192 | "codemirror_mode": { 193 | "name": "ipython", 194 | "version": 3 195 | }, 196 | "file_extension": ".py", 197 | "mimetype": "text/x-python", 198 | "name": "python", 199 | "nbconvert_exporter": "python", 200 | "pygments_lexer": "ipython3", 201 | "version": "3.6.5" 202 | } 203 | }, 204 | "nbformat": 4, 205 | "nbformat_minor": 2 206 | } 207 | -------------------------------------------------------------------------------- /tutorials/gcn_prediction.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Prediction of molecular properties using GCN\n", 8 | "\n", 9 | "A tutorial for supervised learning of labels as outputs of graph inputs.
\n", 10 | "I implemented GCN and GAT to predict molecular properties and observed better results from GAT.
\n", 11 | "Please refer to the following article for the whole contents.
\n", 12 | "Ryu, Seongok, Jaechang Lim, and Woo Youn Kim. \"Deeply learning molecular structure-property relationships using graph attention neural network.\" arXiv preprint arXiv:1805.10988 (2018).\n", 13 | "\n", 14 | "There is a key difference between the node classification and prediction of labels from whole graph inputs. The later task has to satisfy permutation invariance with respect to changing node orders. Therefore, we will implement readout functions which satisfy the permutation invariance in this tutorial." 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": 1, 20 | "metadata": {}, 21 | "outputs": [ 22 | { 23 | "name": "stderr", 24 | "output_type": "stream", 25 | "text": [ 26 | "/Users/Lulu/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n", 27 | " from ._conv import register_converters as _register_converters\n" 28 | ] 29 | } 30 | ], 31 | "source": [ 32 | "import tensorflow as tf\n", 33 | "from IPython.display import Image" 34 | ] 35 | }, 36 | { 37 | "cell_type": "markdown", 38 | "metadata": {}, 39 | "source": [ 40 | "Overall architecture is as below." 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": 2, 46 | "metadata": {}, 47 | "outputs": [ 48 | { 49 | "data": { 50 | "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAMCAgICAgMCAgIDAwMDBAYEBAQEBAgGBgUGCQgKCgkICQkKDA8MCgsOCwkJDRENDg8QEBEQCgwSExIQEw8QEBD/2wBDAQMDAwQDBAgEBAgQCwkLEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBD/wAARCAGlANQDASIAAhEBAxEB/8QAHQABAAIDAQEBAQAAAAAAAAAAAAYHBAUIAgMJAf/EAE0QAAAFAwIBBwcIBwYFBAMAAAECAwQFAAYHERITCBQXIVZX1BYiMZSVltIVQVFSVZKT0yMyNmF1gbM0N0J3tbYzYnFzkRgkobRFcoL/xAAaAQEAAwEBAQAAAAAAAAAAAAAAAQIDBQYE/8QANREAAgADBgUCBQQCAgMAAAAAAAECA1IREhMUkdEEIVGh4TGxQVNx0vAFM2GBIjJCwRVi8f/aAAwDAQACEQMRAD8A/VOlKUAqH5cynamFMdTeTr2XWJEwaAKqEbkA666hjARJBIoiAGUUUMQhQ1ABMYNRAOuphVD8tW37nmsGLSlpW66uB9ak/CXSeJaE3uH7dg/RcLJJF69ygppnEC6CIiGgAIiFARZ7yq83WIwiL6zXyX1LTsGUeINXUo1upF+/hCuFipN1XzMESbSiZQm/hqHFPXQQEQ0roh5ednx0ohByF1w7aScuCNEGaz5Ii6q5yGUIkVMTbjHMQpjAUA1EpREOoBrhrlRcrjH1zWE4yDyeeUtfJ7sRh+I0s+12jdYiIJqCZd3JoqtFFmpEiHHimOomAAmUA6x1Gx07Hs1flF56zBL46bXVc9qWtBnhQOkBnRDfJzk5yNFBDciqoIATiEEDegAH06gdJQmTMb3LcL+0rcyDbUrORX9vjGUsgu7af91EhxOn/wD0AV/GmT8ayFxtrPYZDtlzPPWwvW0WjLNzvFm4CICsRED7zE1AfOANOoeuvzQxBdmO0chcmR9b124pbtGsmo2PE2dazhupBC9i3JE2chLLuFDqrqLmKThKkIdZQDKCHm18cP3ByY5zkxY6x7jyOjHWbxv9gug1YtBGbRkk5Yqjh4dQobytysiHEVDG4YEAC+kugAfoNaHKJsKcm5y3LnnIG15OPu15akYzfzSJF5dRAqQ8RBM+wxjG4unDKBxDT0jrUxyXePR3jq6L/GO+UPJqGey/NONwucc3ROrw9+htu7Zpu2jprroPorh3JuK8dO+T7yvMhvrHhnlzeVE6olLOWKarpHmqDcyHDUMAmIBDbjBtEOsRGur88Kqr8ljIK65hMopYUoc5hDrEwx6giNAbuLzRYIWBDX7ed1QFpoScKym10pWXQRKzScpgYu9RQSBt3CYoHEAARKOn0Vvpa/rEgIks9O3rAx0YdmaQK9dySKKBmpQKJlwUMYCimAHIIn12huL19YVyhijF2PMh5ust7ftkxNwHh8D24DIkozI5TQFZdwRUwJqAJQOJSgXdpqACYA6jDXPlhzWELTleSw5z38kNrZiiZDZxh5hIBYs3SMuQjQVQMG0pSFKJSmOGhTbB6hABAD9OI68bRmI6Kl4i6od8xnRAIp02fJKovxEpjhwDlMIK+aUxvNEeooj6AGotlvOePcNWs8uq65lsZJhIMY1dqi7blXIu7VTTTASqKFANCqcUQEQHhlMYAHSuIrcuTHdjP4bMMCipAYTRz6u/h5HmiqcY2ZLW4szUdJF00RaKSKihSn0Knqfq0CsnJF0YwzFYXKQyOjGtJy1mN72e9bv30XxUhbt045Nw8RKcgiKIo8fRUoaHSMYQESG6wO7JPKmL4SQh4iZyRazB9cRSnh2rqYbpKyJTabRbkMcBWAdQ02AOutSmvzF5Q0vhFG78oy9tXVjRyymbbhDksm+La0SuCNKxKLM9tv2axVykOXRMhUyCKbhMxgAobRr9BCEve4sRsPJJdvZlzPohmo3+VGp5IkYsYhDHSVTFRMywlDcTUTlER6x+igJtXNrrlO5cva5LrYcnTk9EvmCsx6vEvZuVudOFRkZJAdFmrApkVRW2G8wVTiQm8BAB0DUbrx1FZCh7ZIyydd8Xcs4Cyhjv42JGNQMmI+YUERVVEBAPSO/r+gK40xpmHGuHMS3Ryas15bmMMXhaU9KPCySRE0nMkwXklnaDyPMsism5IqQ4kEhSHUAQMG0B0GgL+Ycq63ZXk+r5tY2pKJSSb09vltZ4YqT0bgB1zMsaYwbigYzkSl3gAgBDbxANBAM9TlFEtrOwYSyfbLa1yykAM7bc6MqCzOX4BAF+385NPgqoa79NT7kvPHZ+rXL9lXtfN62rjidvefuuSjIXlDAwBW7WRG0kWPFiqSNFykRFIEzGVXbnDUhdDKkH6Kk/LOsyQ5bN5/8ApNx2MW0Tsghbiuq7HjPnJIl4dE4MY1EAMUeMuBhOqOuhEgAfOHzBA6Q5P+Z3WebSfZCa2Y4g7acSbhvbTp04Ey01HpjtK/FEUyigmoYDbCiJhEoAbUAMAVEOUtyuLd5N1z2Hbcrazua8rX+km5brCmnAxZVkUVJFfQhg4ZVXKJdDCQB1N52oAA4GA+U/ZT/GluwV/RydlXfFT7bG8pbiDFXhspwqYgkikVMpgI2UTT4iSgjwwIIBu1AapeZxflzlb3xm2+bMuCyo+0pts5xRHmuGGdPHAsWRh504aHSXSKnuenVEDGKfUW6Y6eboIHXV4ZQTsrIVl2jMRGkTeyjqPaS5Vx2oyiafGRanT26ACyJHIlU3frIgTTU4DU6rhlbIs9kzkn4fJMHL5eweTrZtWVRMJTrBMRkqRJ2If8wpIKrCP1DG+au5qAUpSgFKUoBSlKAUpSgPmm3bpKHVSQTIdQdTmKUAEw/vH56+lKUB8ubNwIYhUSFA4iYdpQDzh+f/AK/vqHYcxRBYWx1CY3gHz2QZwSSqCDt/wxcHIdY6ogYSFKXQBOIBoAdQB8/XU2pQClKUAqEz+KIK4sqWvlp4/fFkrVjJOKbNSCTmy6L4UBVFUBKJhEObk26GAOsdQHq0m1KA8CkkKfBFMopgABt0DTQP3VG7TvVldc5edvNo9Ruay5tKCXOcQErg541k+A5AD0FAj4hNB+chh9AhUnqqsN/3i52/zAZ/7VgaAtA7ZsociijdM5k/1DGIAiX/AKD81fWlKAV81G7dY5FFkEzmTHUhjFARKP0gPzV9KUBF8mY7t7K9jStg3OVcrGUTKALtjgRw1XIcFEXCJxAdiqSpCKENoOhiFHQakbVJRu1RQWcncKJplIdZQCgZQQDQTCBQAAEfT1AAfQFfWlAaW5rPt+7/AJKC4WZ3RIaTQl2qXHUInzpHUUTnIUQBQCGEDlKcBKByEPpuIUQ3VKUBEbhxpCXRf9rZAmXbxdaz03gxbATE5oR05IVMzsxdu4ViJAqmQd2hSrq9QiICEupSgFKUoBSlKAUpSgFKUoBSlKAUpSgFKUoBSlKAVVWG/wC8XO3+YDP/AGrA1OL6jrpl7LnYux7hLA3E6jnCUTJnQIuRm8FMQRVMmcpinKU+0RKJR1ABCvzz5BeeeVhm7lJ33bl4x8PbERBTJ5m/ytI39MvJJxzeJQj9yh1AR1NHlXHYACIoLgBgKJSgB+lNKUoBSlKAUpSgFKUoBSlKAUpSgFKUoCv5a6Jyak3bOAk/kxhHuDNTuE0SKLuFiCAKAXiAYhCFNuJ1lEwmKOm0ADdruFeHeZP+qxvhax7cMY7eSE5hMPy7MBqI69QSDgA/+K21d2GVBArqhWiZyYpkcbtterMHhXh3mT/qsb4WnCvDvMn/AFWN8LWdSrXYKVotiL0VT1Zg8K8O8yf9VjfC04V4d5k/6rG+FrOpS7BStFsL0VT1Zg8K8O8yf9VjfC04V4d5k/6rG+FrMOciZBUUOUpShqJjDoABRNRNUgKJHKchvQYo6gP86XYKVotheiqerMPhXh3mT/qsb4WnCvDvMn/VY3wtZ1KXYKVotheiqerMHhXh3mT/AKrG+Fpwrw7zJ/1WN8LWdSl2ClaLYXoqnqzB4V4d5k/6rG+FrR2/YZrVlJ2bty65KOf3O9LIy7hBhGlO8clSIkChx5r1jsTKH0a7h9JjCMpEQKAmMIAABqIj81eUV0HBOI3WTVJrpuIYDBr/ANQpdgpWi2F6Kp6sxOFeHeZP+qxvhacK8O8yf9VjfC1nUpdgpWi2F6Kp6sweFeHeZP8Aqsb4WnCvDvMn/VY3wtZ1KXYKVotheiqerMHhXh3mT/qsb4WnCvDvMn/VY3wtZ1KXYKVotheiqerMHhXh3mT/AKrG+Fpwrw7zJ/1WN8LWdSl2ClaLYXoqnqzB4V4d5k/6rG+Fpwrw7zJ/1WN8LWdSl2ClaLYXoqnqzB4V4d5k/wCqxvha9EC8UhEwZGmVR+YFmjAS+n5wK3KP7vSHprMpS7BStFsL0VT1Z7a5cgYgp42+JFswk0DbRFMhuG5IIAILEL5wlAdRASiIiBimDUQ0MKuZ+Us8dt77YEQdLJlGJSEQIcQDXjLdfVSvslfocmfAplrVphH+qTJUTgsTsL+tr+zSX8emf9RcVt61Ftf2aS/j0z/qLitvXyv1NV6ClKVBIpSlAc0ZOs+Oz/ynkcQZBXcOrEs21GtzrQCa5028xIuXiqSRnQFEOIkiVtuKmPUJj6j1agMZw6tifG2Yrka4eb3paDGTtd7IqWPM2w/YRa7tmqTdINVHO0pNCqESORIu0wHKbUNA1uDK+DJu7b6hcuYzyCeyr4hWSkUZ2pHEfspKOOoVQzV03ExBMUDl3EMU5RKJhHr6tI/b3JtvlTJSuW8l5rXuadXt2RtsrNvCkZRjNs5MkYot0QUMcpimTMJzHUOKm4oakAgAODhd62z4+psolZZaRiI5RWfFcDx+Z7rtLGltI3QeLGHPJ3Edu0jGjggio8kVVtmoCPDFNBHcoPEAojqAiEfjOWterixLseRMTYt83Da90QVvNndtySycPMBJqEIQUzqAcyJyiYSmATKFA3+IwBVnXTyZ5CUxfi2y4G92bWbxSowXjpCQhQeMnijdmZqYVmYql6jFOYxdFNxDaCBuqqbzhya8vQ9r3DNM8qvbjn77ui0VVXDK1wKeHdNXqZAdIJJqGDmyKYgYEj6iHDETqmERGqxYkK5fn4yYbjN1mzJmX0cc5hw/mm3bZI8d4ymbjhZe21VuaKt00+Cu1VTXEVAVTOqkIKBoRQptQKQQEoTfHGV86RGSLKsLMNsWWxi77hHLuETg3TpZ5GrM0kzqIO1FABNbcmoUQOmUgFMUxfODQw+pLkwXrfDS+H+VswITdw3TaDqy410wgeZMohk4DVVQrUXBxVVOoBDmEVS9RClDQKsWbxKExkXHd/8AlAZHyBaSTUGnNQNz3naCaW4T7w4e3h66aG1106tOu6hjtt/PUhxQ2WFacpCCPlfMGMcAz0o8b2XcDWZnbiZM3Kjc8sRkVAqTRRRMxTAiJnImOUB87YHoEAEPuvhbF3JtuxbM2PX6dlwjCAklLgtSPKPNp0iCIKpqpoioBU10QTMO8hBExTCAiAaiM1zdhA2VVbcue27wdWfe1mOlXcBPtmqbngcVPhrILIn0BZBQugGJuKI7Sjr1CAxVjyZ567rme3hygcmeWzw0I+t6MYxsSWJj41q8TBN0oRPiKqGcKFAS8QVOooiAF9Ghwu83Z/ZCiVllpqLbzdyhYh5YNz5ZsW0GtpZJkG0a2aQzhyaTt9Z0mZRqDo6v6NyBtoEOJCpbDG6gOFRwvKb5QaUE0yw/sWxG2PUbzG1XifPnZpZygaUNHA8RAC8JLaoJBFM28TgU4gJNS1MrY5NmRCytis8mZqJdNq42dFfQUcjAlZu3TlJMyTRR+54xwWFFM5tATTT3n0MbXTQdtIcmkr/CYYdG8xIAXQFyfKPydr/+a+U+DwuL+/hb93/Pt/w1Fkxr/wCE2wGgmeU5dEHZ2REXdqxpr8tS8EbSiIkqpwRklHqiXyasI67gKokuBz6ejhqaeiuiUeNwU+cbOLtDibNdu7Tr0169Nap25OTRB3HyjIDlBK3I9Q+R2IJOYIhB5rIPUiqkavFfPABURI4XKXUhh84uhi6CBphiJLIaNprhk14s5ljTEoZA6ySCSgMBeK8zAxUBEgCDfhfOJvrDu1ALw3k7IikV1rkcgcpe1neHQc39Oz+UpPIU3cqcmxvGMcvkrdtqN5+QE2zsCG5sk3Tb+acp0jmUExh9Bh0lvKStSxGVx3Ze+a8n3E8lZxui0xfbNpzT5ORQ4bcuqjZm2MUV3B3RjCKhgOmBQS3CX0BLZPkm5EeW/cGJmmeBb4suOUcvXEQa3yKSqDRysKzhik+FbaCRlDqaGMiY5Sn2gI6a1933JdyxHZhunLmPs9Q0I4uMjVqglI2KlKOI1mgiVMrVFyd0Q5UhEu8xQApRN1iGuojk4IufL85/zzNFEupcOGU74SxJZqWTDKGuwkGyLNCoYpji8BEvF3CXqE27XUQ6tdamVay2WM7G29Gx9zzqc1LN2yab2RTZg0K7WAoAdUEQMYEwMOo7QMOmumo1s6+hckYv1FKUqSBSlKA5k5Tn7esP4Ql/WWpTlOft6w/hCX9Zalen4P8AYh+hxeI/diOh4JPmx5hifUFUJuSMoUwaCXiulFydX701SGD6QMA/PW0qRz1lxk67LJFdvY58BQIZyyUKUyhAHUCnKcpiHAOvQTFEQ1HQQ1HXT9GLnvHuf8OO8JXj4eLlRK2J2M9DFw0adiVqMSlZfRi57x7n/DjvCU6MXPePc/4cd4SrZmTV2exXAmdPYxKVl9GLnvHuf8OO8JToxc949z/hx3hKZmTV2ewwJnT2MSlZfRi57x7n/DjvCU6MXPePc/4cd4SmZk1dnsMCZ09jEpWX0Yue8e5/w47wlOjFz3j3P+HHeEpmZNXZ7DAmdPYxKVl9GLnvHuf8OO8JToxc949z/hx3hKZmTV2ewwJnT2MSlZfRi57x7n/DjvCVCrAhp66rryVBSF/zqbezroQhGJkUGAHUQPCxj4TKiLYQE/FfKlASgUNhSBpqAmMzMmrs9hgTOnsSqlZfRi57x7n/AA47wlOjFz3j3P8Ahx3hKZmTV2ewwJnT2MSlZfRi57x7n/DjvCU6MXPePc/4cd4SmZk1dnsMCZ09jEpWX0Yue8e5/wAOO8JToxc949z/AIcd4SmZk1dnsMCZ09jEpWX0Yue8e5/w47wlOjFz3j3P+HHeEpmZNXZ7DAmdPYxKVl9GLnvHuf8ADjvCU6MXPePc/wCHHeEpmZNXZ7DAmdPYxKVl9GLnvHuf8OO8JXomMlAEQWv+5ViD6SmBiT5/pI2KYP5D89M1Jq7PYnAmdPY59zTYE/fV4EfW/HvHabBmmyXFuhxAIruMptEQHqHYqQdPoMH00rquIh46CYJxsW34SCeo9ZzHMYwjqJjGMImMYR6xMYRER9I0rWH9dmylcghViKP9Klxu9E+bM2lKVwTqilKUApSlAKUpQClKUApSlAKqrDf94udv8wGf+1YGrVqqsN/3i52/zAZ/7VgaAtWlKUApSlAKUpQClKUApSlAKUpQClKUApVVP+Heki/dzhAcsmj9wzZsjm3IEBBQUjHMTXac5lCHHUweaGhQ084TYXR9YXYiA9mo/DXRh4BNf5RWP6W/9o+KLi2n/jDy+vguKlU70fWF2IgPZqPw06PrC7EQHs1H4atkIK3p5IzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwWLfVmQORrLnbBuhuZeIuKOcRb0hDbTiismJDCU3+EwAbUDekBABD0V+dvIF5MOconlLX1cGe7quaYi8WyyjSG+U3rhRtKSyjQjZOQTKoYwGAkaVEoCYNwFVbgAhw9A7N6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4adH1hdiID2aj8NMhBW9PIzcVPfwXFSqd6PrC7EQHs1H4a9EsOyUhEyFoQyJh/wAaLFNMwdevUYoAIdYBTIQVvTyM3FT38FwUqjZfNwYtdhbEwgeTKJCuGaq7wAVIgbzQIcxtTKCBiH0MPWIbQHUQEwqr/wCL4p84IbV15Fs9IXKJ2M3Ftf2aS/j0z/qLitvWotr+zSX8emf9RcVt66D9T4l6ClKVBIpSlAQrKeZ8a4Ximkvka5k4wki5KzYNyIKuXT1cwgAJoN0SmVVHUwa7SjpqGulY2Lc7YszMaTQx9c3PXsIoVKSj3TNdk9ZmNrt4rdwQipQHQdDCXQdB0HqGq1WJDuuXkQl1FTO6Z44SVtMrkPNIqZ8sV+dDXq4uwG4GEPO2a6ebuqNhddyseUo+Y33YVjRl7OsdSzhvM27cLx07TjUHCQpJuEVEUiF3KHExDaHEOGcAEA11yvu3+DS4rDqylcSxry/7f5OWPZu486ZQuG7strQqhmkMk1VfrJC0OuoyjROKKTL9F5yrpQxhEEjD+sYKjC2R80wVpZDsVre97WmtC3zZ0ZFObllGUtLxTaSVTBYq7hMyiapNB3ARQ5zAUdDiOtQ5yXqicK34ndd23VAWLa8ted0v+ZQ0GzWkH7nhHV4LdIgnOfYQDHNoUBHQoCI/MA1sm66Tpuk6QPuSWIVQhtBDUohqA9f7q4mzgxufFrLLeKAyrcV329P4enLkM0uN5z17GvUBKgKia3UYqKwLGHhabCmSNs0DzQm9mwN6Yhzli2Hc5hvS7m+QoGULONpx6RRmVyzborJLNG5SARr/AMQ5BInoAl27txgExpxHbY0Rh8rbS/Ml5Tx/h61lr0yTdDSDiEDgnxl9xjKqCAiVNJMgCdVQQARAhCiYQAR06hrQ405Q+JstTT62LOuJx8uRyBXTiJk411GvSoG0AFgQdJpnMnqYA3lAQARABENQqvs0lhleVlgZG7QAYvm1xKRpXADzYZkqTbm//LxgSFwKe7r112+dpW15QN54xsS5mF1Ooor3J0HadxytsiTijwWyDXevzgEzAXgmMVMocQBDdrt0NqNHE02/ggoVYurL0pXIMWyvvFqOFcllz7dV2yORpuMjLhYyr8HMXJpvmyiplGTYpAI04Il3kFHYAkLofdURWcZUi8WNOUY5zxfbmUY5H+TG0Jz1IkSMYa4TR52qyAJ/p9UjGEFFBMYogTbt29bF/gYdvxO7KVxxduUci27L3zyY0b7e+Xly3iyJZr865eeI2/Ijxll0tesxWZEHpN2g7RKlr1iGvYbchU0E0iKmVBMoE3mNuMbTq1EfnH6avDGovQrFDdK8k+UdgiGv0MXy2WLaaXSKybb5MWfEKqCymmxIRHzQUNqXQgjuHcHV1hXrIPKJwdiida2xkbKNv2/KvEyrJNHrsCKAmY20pzB/gKIgOhjaAOg9fUOnMHKIg2bfk9T0/gqOs2TxcrPu7lvd28MsWcWeIynEecz4ifD4wKJmIUyo6lAgFIAgJdJtez2GWvnI9k4JxorfeQcismprpfzJiEg4BAzJNFui7X2iYQ4Q8YrRMFDm3nN5oCFZOZF6fnkvchOqmzlu8bpPGbhNdBchVElUzgYihDBqBiiHUICAgICFfSojiKxFMYYttLHKsmMie2oZpFnd7NnHMikUgnAuo6AIh1BqOgVLq2XNczJilKVIFKUoDmTlOft6w/hCX9ZalOU5+3rD+EJf1lqV6fg/2IfocXiP3YjoS2v7NJfx6Z/1FxW3rWtyFgZiSt+QMRBdWQdvmoGHTnKK6xlt5NR87aZUSGAOsBD0AAlEdlXmLbf8l6M7VlnJilKUApSlAQfKGFMaZkbx6WQLbB8vDrC4jXzdys0esVB01Mg5QMRVPXaXUCmAB2hqA6BWlsvkx4QsCUXn7asoCTTtq6Zu5Z1IOnb92i44YKkWcLKGUVDRJPbuMOzQdm3U2tpUqt2Fu2wm87LLSvrjwLiy6rEgMcStvLlhLUFqaDK1knTdzHGbp8NE6LlNQq5TFIIl3b9RAR1EdaqTK/IgxpOWU6tnHNqtmKk/Lwilxg8mH22SYtXpVlxVMJzmO5OmKocbqUMJus/z105SocuGL1RKjiXoyq4Hkw4Vt2EuiAaWq6dIXmxGLm15CYevXbplsEgN+crLGWImBTCAFIcumuodfXUzfY/tCSuO3btew5VJa1EnKEO44ygc1I4TKmsAFA20+4pChqYBENOrTUakNKsoUvREOJv4kUyXivH+YLaNaWR7ZbTUZxSuEyKiYiiCxddqqKpBKokoACIAchgNoIhroI1o8ecnjEWL38nMWvaxlJWZR5q+k5V84k3q7fQA4ArujqKAl5ofowMBR0ARARqx6Uupu2zmLzssKosjkt4Rx5c7G7bXtNwk9iAcFiE3Us8dtYkFx/TczbrKmSbbtRD9EUuhREoaAIhUhcYXxm6soMdr2yB7eCUCa5nztcP/AHgPee8TeB9/9p/Sbd23/Dpt82ptSoUMK5JBxN/Eiz/F1gymRIvLMhbTZa7YVgtGMJMxj8RFsqOpyAUDbB11HQwlEwAYwAIAY2uJiTGzfFdqL2y3eJOCuJiTlzCkkZNMhnjxVwKZSmOc2heLt1Ew66a9WugTSlTdVtotdlhTkhyQ+T9KXO8ul7Y6h1JKUCbexwSrwsW6fhoPOFWAKg2OcRABERTHUwajqIiI/O6OR5ye7xu6Yvqds2SGcn1SLyTlpc0qzBwchAIUTJoOSEDQoAAaFD5/pGrnpVcOHoTfi6mstm24ez7ejbVt9so3jIlsmzaJKOFFzESIUClAVFDGOcdA/WMYRH5xGtnSlXKilKUApSvKiiaJBVWUKQhesTGHQA/nQHM3Kc/b1h/CEv6y1KtG7MMvsxSZbqj1mabJFErNsquY4A5IURPxSbR0Em5QxQH59oiGpRARV2JH6lw0mWpccVjRz5nBT5sbjhh5M6Ck4iJmm4NJiMaPkAMBwScolVIBg9A6GAQ1rRdFuMu7q2PZDf4KlFK8VDMjgVkLaPTOCGLm0Rfotxl3dWx7Ib/BTotxl3dWx7Ib/BUopVsebU9WVwoKVoRfotxl3dWx7Ib/AAU6LcZd3VseyG/wVKKUx5tT1YwoKVoRfotxl3dWx7Ib/BTotxl3dWx7Ib/BUopTHm1PVjCgpWhF+i3GXd1bHshv8FOi3GXd1bHshv8ABUopTHm1PVjCgpWhF+i3GXd1bHshv8FOi3GXd1bHshv8FSilMebU9WMKClaEX6LcZd3VseyG/wAFVriew7Hkb8zOzkLMgnTeLvhq0YpLRyJyNUBtuFWFJIol0IQVVlVBKXQN6hzekwiN5VVWG/7xc7f5gM/9qwNMebU9WMKClaEt6LcZd3VseyG/wU6LcZd3VseyG/wVKKUx5tT1YwoKVoRfotxl3dWx7Ib/AAU6LcZd3VseyG/wVKKUx5tT1YwoKVoRfotxl3dWx7Ib/BTotxl3dWx7Ib/BUopTHm1PVjCgpWhF+i3GXd1bHshv8FOi3GXd1bHshv8ABUopTHm1PVjCgpWhF+i3GXd1bHshv8FOi3GXd1bHshv8FSilMebU9WMKClaEX6LcZd3VseyG/wAFfRDG2OmqgLNbBtxE4egycUgUQ69fSBfpAKklKY82p6k4UHRH8AAANADQApX9pWRcUrUTl1wFuHRSln/DWcaikgkkdZY4AIAJgTTAxxKAiACOmgah9NabpXs/6s/7tyX5FawyJsathhbX0ZnFNlwuyKJL+yYUqH9K9n/Vn/duS/Ip0r2f9Wf925L8irZWfQ9GRjyqlqiYUqH9K9n/AFZ/3bkvyKdK9n/Vn/duS/IplZ9D0Yx5VS1RMKVD+lez/qz/ALtyX5FOlez/AKs/7tyX5FMrPoejGPKqWqJhSof0r2f9Wf8AduS/Ip0r2f8AVn/duS/IplZ9D0Yx5VS1RMKVD+lez/qz/u3JfkU6V7P+rP8Au3JfkUys+h6MY8qpao3F4XZAWHasve11PTs4WBZLSMg5KgosKDZIgnUU2JlMcwFKURHaUR0Aa57wXyjcFzGXskW7B5QgpaTvy/EFreaRrjnir1JO1IcVFdqIGFNMvNnBTHU2lA6JyCO4olq53WTLFfNVmL5rNOGzhMySyKtsSJyKEMGhimKLfQQEBEBAeoQrjLkW8lSwOTDmvJeSXxpd20eOjRtkh5PySijWJUEFlDKaof8AF14aG7qNoiqP6qoUys+h6MY8qpao/QKlQ/pXs/6s/wC7cl+RTpXs/wCrP+7cl+RTKz6HoxjyqlqiYUqH9K9n/Vn/AHbkvyKdK9n/AFZ/3bkvyKZWfQ9GMeVUtUTClQ/pXs/6s/7tyX5FOlez/qz/ALtyX5FMrPoejGPKqWqJhSof0r2f9Wf925L8inSvZ/1Z/wB25L8imVn0PRjHlVLVEwpUP6V7P+rP+7cl+RTpXs/6s/7tyX5FMrPoejGPKqWqJhSof0r2f9Wf925L8ivRMqWccR1UmUgDrEy0A/TKHXp1mMiAB6fpplp9D0Yx5VS1JdSvgyes5Jok/j3aLlsuQFElkTgchyj6BAwdQhSsGrOTNfUrGJVF+8mZdYwncOJZ63McwdYJt3CiCZA+goFT10+kxh01MOuyrUW1/ZpL+PTP+ouK29ehas5HH9eYpSlQBSlKAUqo8w50lLGuuCxfjmwHF8X7cSJ3yEWV8Rk2ZR6ZykUeO3Jim4SYCbaXQhhOYNoBrpr8MY53uSfv+WxNlvGprFuuNi/l1twpQkjHSUbxOEddFyBE9BIcQAyZyFMG4B6wHqrfhtsLXXZaXHStAGQLDFm/kQvaAFrFJJLP1/lJHhtE1C7kzqm3aJlMXrKJtAEOsK9xN92RPwiNzQV5QcjDuVCooyDSRRWbKqGEAKUqpTCUxhEQAAAdREQqbURYzeUqurzzhZ0Hii9Mp2bLw14oWZHPXjlvGSyShDLNkjHM3OqnxASP5ug6lEQ+gakNr5Gse8HjmHt+74KQmI5FJWSjWcki4cseIXUoLJkMJk9fm3AGtLytsF12WkkpVXZyzh0SJQMFb1nvbxvW73SjK3bdZuE25nZ0096qqqynmooJl0E6ggOm4Or0iGnsfOd9jk5liTM+KUrPmJ1i4fwD6OmSykfIlbgUV0eJwkjpLEA4G2mJoJQEQN6NYcaTsJutq0umlR+GyHYFxz0hatvXzb8pNRPVIRrKTQXdNOvT9KkQwnT6/rAFeOkjHfy0xtry9tz5XkxUBjH/ACohzl0KYmA4JJbtx9olMA7QHTaOvoGptRFjJHStaFy24aLczhZ+NGNZCoVy8B2nwERTEQUA6mu0u0QEB1Hq0662CahFSFVSOU5DgBimKOoCA+gQGpIPVK52meWK3in0xMt8M3k/x9b86NuSN4oc2Bum7K4BuqcjY6gOFUE1jbDKkIIalPprp17PIHKff21eNy2nYuFbsv1Ox2yDm6H8Us1RSj+KnxiopguoUzlYEdDimmAiAGIHpHqpiQl7kRe1K0tlXfB5As+Fvm2lzrRM+wQkWShyCQxkVSAcu4o+gdBDUPprdVf1KClKUApSlAU/kvLE/iy5RhreemQbyCJZJRMECqFBU5jEMIbh6teEBhAOoREw+kRGlQTlOft6w/hCX9Zald3h+C4ebKhjjgTb/g5k3ip0Ebhhidh0JbX9mkv49M/6i4rb1qLa/s0l/Hpn/UXFbeuG/U6S9BSlKgkUpSgOZ8iXpEYJ5VxMnZIBSNsq87Na26ncJkTmaR0i1eLKgi5OUogiVUjnzTmECiJBD5hEIDZOUz3Dm2TtWxM13zkSyV7In3TmQlW7cYo8gkduBEWThFsiCwpJqm36GOAcQga6gNdqKJprEFNVMpyGDQSmDUB/lRNNNIgJpEKQheoClDQA/lWbgdvJmijXQ4Hf2vjTFnJGwaSOsSyY5vdbyAfzNwXC0VPGNXfMlFweSREDpmd6nEyaZFjgkUyhdRACgFVRc9x4+8k8xp3PcEHM245vOx5l8aFt1SHYPo3nBCO3LVpvOY6RgKYorlMbimATAIgICP6nqJpqkFNVMpyG9JTBqA/yqI5NxjCZRhI+CmHjxmlHTMbNJKMxIU4qsnJHCZBExTBsEyYAIAGugjoIems4pNq5fnItDN6nF+T3+IbrcZkujk7to5W2WuFJWOuOQgUgSiln3mmYoiBNEzuE0AcaiAbikMUoj6AC2U8c2JjLlA8n1pj6zom303NvXAxdDHsiJGcIkat1ClVOUNT6KCJ9TCI7jGHXUw69SppJJAJUkykAREwgUNNRH0jXqrqV8fz1KuZ8DnLlGzgYrzVizO1xxrlSyoRpNQM9ItkFFhiOelbmQdKkIUw8Hc2Ehj9W3eX066Di3fyj1MwHuDH3JnJ5XJpWnMuJK5Y4VObR74Wogxat19OGo5UUOA7QN5pS6+kB06WEAMAlMACAhoID89eUkUUCcNBIiZdddCFAA1/lUuB2ux+pCiVnNHCNoSHJ9ng5Olp8n+HYFvm35pqvJoRzUEJKGYJNlCyoSYgQDJ7ziCZyqiHEUOUS7qwX2NLCieS4bI8fZcQndiuWk3ozXMkzPgWLdwoFEFhDeAAlqQCgOmhjdXWOvfZG6Cah1U0EyHU/XMUoAJv+o/PX0quDy5lsXocQX9a8u0zfN8khjCPRtTMFwtL8VdpkDmraOTEFJlqYdQEBWXbIgAAAh/7w2umoAPX9j3vauQIM07Zr/nccg8dRon5uojtXbLHQWJtOUo+aomYuumg6agIhoNSCtRa9qwlnRqkTANlEW6zty/V4q51jqOHCxlllDHUETCJlDmHrHQNdA0AAALwwXWViivI4V5SOT8f5XsqSjGt43JZ2TbTnFWDTFTVQFUJ2TRe8RsZyyBADO0VygmqByiCYFUATCYS61vMwco605XKd4YRkMsW5heFZFbJXfLosxNPTrtZqnvI0OBBSRIRLRIzg+9UBKUClAoAYe4RboCsDgUE+KAaAptDdp9Gvpr6VRyon8fzUtiLoRTFCFjNcZWs1xksRW0kYhqnCKEMcxTsgTKCRgE/nDqUAHUesfnqV0pWqVisMnzFKUqQKUpQHMnKc/b1h/CEv6y1Kcpz9vWH8IS/rLUr0/B/sQ/Q4vEfuxHSkpFyVrSTwQjHz+Mfu1HaKzREy50Dqm3qJqJl1PpvMcxTFKIaDoOggAm1/lK2+w7n92pH8irapXh4OOaVkUNp6eLhU3amVL5StvsO5/dqR/Ip5StvsO5/dqR/Iq2qVbPqnv4K5R1dvJUvlK2+w7n92pH8inlK2+w7n92pH8irapTPqnv4GUdXbyVL5StvsO5/dqR/Ip5StvsO5/dqR/Iq2qUz6p7+BlHV28lS+Urb7Duf3akfyKeUrb7Duf3akfyKtqlM+qe/gZR1dvJUvlK2+w7n92pH8inlK2+w7n92pH8irapTPqnv4GUdXbyVL5StvsO5/dqR/IrFZX3AyLl+zj2066cRbgGj5JGAfnO1XFJNYElSgjqQ4pLJKAU2g7FCG9BgEbkqqsN/3i52/zAZ/7VgaZ9U9/Ayjq7eT5eUrb7Duf3akfyKeUrb7Duf3akfyKtqlM+qe/gZR1dvJUvlK2+w7n92pH8inlK2+w7n92pH8irapTPqnv4GUdXbyVL5StvsO5/dqR/Ip5StvsO5/dqR/Iq2qUz6p7+BlHV28lS+Urb7Duf3akfyKeUrb7Duf3akfyKtqlM+qe/gZR1dvJUvlK2+w7n92pH8inlK2+w7n92pH8irapTPqnv4GUdXbyVL5StvsO5/dqR/Ir0S4U1REqMDcpz/MU1vvk9evT0nSAvz/ADjVsUpn1T38E5T/ANuxVKmGGF6HGdvQHjR2poRBqiuXVugAeaRQQ3FE+4TmHaOgbgABHTcKrWpWT4/iPhE0aLhJPxhtFKUr4z6BSlKAUpSgFKUoBSlKAUpSgFVVhv8AvFzt/mAz/wBqwNTi+pqetuy524rXtg1xy8XHOHjKHI44B5BZNMTFblU2n2mOIbQHaIaiGtcU8kTl42tnXPN12RjfF9wKK3rP+U8k8fKpIJQ8a3gY1mZVTaZQVTi7acEpSgQolVRMIgImKAHedKUoBSlKAUpSgFKUoBSlKAUpSgFKUoCBXJk9wyknERadvpzS7I/CdrLveaNklNNRTBQE1DHOXq3bSaAI6a7gMAafpSyL3fW57zr+BqL2ccysCmucdTrOHSpx+kxnChjD/MREa3Vc5z5kXNOzQ9bD+m8LKVyKC1r4txf9NIzulLIvd9bnvOv4GnSlkXu+tz3nX8DWDSoxZlXtsWyPB/KWsX3Gd0pZF7vrc951/A06Usi931ue86/gawaUxZlXtsMjwfylrF9xndKWRe763PedfwNOlLIvd9bnvOv4GsGlMWZV7bDI8H8paxfcZ3SlkXu+tz3nX8DTpSyL3fW57zr+BrBpTFmVe2wyPB/KWsX3Gd0pZF7vrc951/A06Usi931ue86/gawaUxZlXtsMjwfylrF9xndKWRe763PedfwNU3hHFCGBb9yPkKy8a2+V/kaVLIuSHuRUCMU9BMZuhow81MVlFlNA0DQyZdNEyjVrUpizKvbYZHg/lLWL7jO6Usi931ue86/gadKWRe763PedfwNYNKYsyr22GR4P5S1i+4zulLIvd9bnvOv4GnSlkXu+tz3nX8DWDSmLMq9thkeD+UtYvuM7pSyL3fW57zr+Bp0pZF7vrc951/A1g0pizKvbYZHg/lLWL7jO6Usi931ue86/gadKWRe763PedfwNYNKYsyr22GR4P5S1i+4zulLIvd9bnvOv4GnSlkXu+tz3nX8DWDSmLMq9thkeD+UtYvuM7pSyL3fW57zr+Br0nlS/SnKZ1j2FFLXzubXGoopp+4p2hCj/ADMFa+lMWZV7bDI8H8paxfcWdbVyRl1xRJaLMoBBMZJVFUu1VuqXqMmoXr0MA/vEBDQQESiAirniQuCWhZ6WQjHZ0CKuE1TgUR84/N0i6/8AgoB/KlbQ8UrOa5nPmfocV5uXFy+FpvbJ/Ztv/wB1x/WPW8rR2T+zbf8A7rj+set5Xyr0O5M/3f1FKUqSgpSlAKoXlLXXkCNvHF1kWPlQlgJXbJSSEhKmjmbvRNBkZYhQK6KJA1MXTUNB6/n9FX1VMZyxAbKmSMUuZazY24rWgJGUXnUJFNBZAhFWJyICZFX/AIn6XZoBSmEB0HqANatDZbzMp6icFkPry90QzH2QcjWdnZnjG7M9QuUIOSt1/OPHZYloydwXNjIgQyxmhuHwlOIpoJygOpR0Hq65xZ3Knx1edwQsK1hrtjWV0qnRtublIRRtGzZykFTRsqPWG4hTGJxSp7wKIk3dVQe9+TDGssoNTYix1E2xbt02RcVrXI9h0WzNBuquVEWaqjYpiCqIGBUNxCHENQAdA0EIRinA9/oS+NLduPDlwRoWO5Qcys5N5DdyUUZRmkKaKkYxTeDodQ20xQVRIRJMTk2jqAVeyFq0+VRTpcV1Llb/AC+nx/Oxc6fKxx4rcZIolt3n8hqznk0ndYwpghTyfHFDgAtu4mnGDh8Xh8Ld1b9aiefuV9blnWbkdlZDe6VJa1mTqPC5WUAd3ER01wR4LdVcSmJvBQUwNqQyZTGKU4gJgAajnMOcoSbj2K1x4qvG47wg7wbTr6Yd36mEW8at5IqxCx0eDkECmMiBQAqySYEADDvE+msvvGyeULBYlyngS1MM+UhbsfTjuIuAsyyQaC1kllVzkWIqoVUrlMVTJl80SGNsETlKAjUqGFMq506KFqxr+np+alvXRym7Us6TNbYWteN2SUTHN5C4VLciOdpQyKpAMU7k24oAIl1OCaYHU2AJtmlTO1Mt2Te822hLWkTyHPLfa3M2dJp/oFmLhQ6aRimHr3apm1KIAIfP19VU+lHZrw9d9/OLVw8pe7S+gZSUc5ZSjNrzJ8kwQZnbvAcHKIJaoAoCiYKdRjBt1rSWVjjNPJ9lLRNbuOEr9Inj1na7xWPlmzIjOSQcqrbjlXEmrYwuBDemBjgCY/o+sAGt1WGymzFFzXL6P+vraSuX5U5Fr+xXDWhaFwyMDfaskm6dEhFFRT4AnSKBVCqAVMSqpmOpqBtEtDdVdBVy1Y+NsyWVA4JeSNgBJSdrSUyncjRlJticyTfisAOCGUUAqqZN4GEpTCfQQ0KI6gF6Yvv13kOJl5N3CJRgxk/JwpCpPSuyLkaODIgsChSgXU2wREga7DbiCO4o6REl8C8iOJ/7+r/j+F/2V9eWZMlQ3KNsHFaOP1I207gdP01p9y6bqhJCjHnXBNBIhxUSAp9NxlALrtEAAQ6xx8kTXKNlbwu3yHuO3bBtGy49FZKSnoYzsk45Mjxlf0gqplRbphtIY5dwgbf6dNAk2TrIue4sxYhuuHjQXi7WkZZeWX4yZBbkXj1EUh2mMBj6qGAuhQEQ11HQOuqczNaOar6zRLjeWA53IeM4gjMttwzK7I1hGPFgICi7l82VWKo4OVXQpCqfowAgjsHduGYbHYZzHHCnba+fLT+FbZaYshyob+u9njNNK/rKxE3u+yT3U8mriZ84buHRTkILNrxVkiAAAJlDbjCbYcmmug63vycsjXFljDkBfd1MmbeSfg4IqdkQ5WzoqS6iRHKIH84ElSkKoXd16HCqfvGxr5d5Qg8tXFyaELzhXFmkgi2mZ/GrrW08Ksoc4kI4OVsciqZiJmOkfUAJpoYNKs/kt4+urGmImlvXhHNol6vIv5JOGbOucIwzdw5OqkxIoGhTAkQ4F1KAF1101DrFFdu8iJOJif5W2f3/AB/XX0/stulKVmfaKUpQClKUBWty/tJJ/wDdT/op0pcv7SSf/dT/AKKdKofW/RfRexL7LKJLeRIb9Yqzkpg+gQXOAh/Ia3dbe4ccXEzkXL+xlIxVu+WM5Vj5FdRAiSxxEVDJKkIoIFOYRMJBIOhhMIDoIFLqvJDLvZ+0PeJ14GtXKjh5WHNh42ROV9RpW/Bux9zzSvXkhl3s/aHvE68DTyQy72ftD3ideBpci6PQnMSa4dVueaV68kMu9n7Q94nXgaeSGXez9oe8TrwNLkXR6DMSa4dVueaV68kMu9n7Q94nXgaeSGXez9oe8TrwNLkXR6DMSa4dVueaV68kMu9n7Q94nXgaeSGXez9oe8TrwNLkXR6DMSa4dVueaV68kMu9n7Q94nXgaeSGXez9oe8TrwNLkXR6DMSa4dVueaV68kMu9n7Q94nXgaj9srZIuqauyCj7XtpNxZ0wnCPjLT64EUXPHtHwGSEGYiJOE+SKImAo7ynDTQAMZci6PQZiTXDqtzfV/ClKQNpCgUOsdADSvfkhl3s/aHvE68DTyQy72ftD3ideBpci6PQZiTXDqtzzSvXkhl3s/aHvE68DTyQy72ftD3ideBpci6PQZiTXDqtzzSvXkhl3s/aHvE68DTyQy72ftD3ideBpci6PQZiTXDqtzzSvXkhl3s/aHvE68DTyQy72ftD3ideBpci6PQZiTXDqtzzSvXkhl3s/aHvE68DTyQy72ftD3ideBpci6PQZiTXDqtzzSvXkhl3s/aHvE68DXpOzMsqnKmrFWk2II+cqWacriUPpAnNCbv8ApuClyPoxmZC/5w6oricYvHdxSh2rc6pSrJlMJA10HgJjoP79BD/zSuhbSs1ha8WLM6nP3bhUXLx2qmAGXWEAATAXrAhQKUpSlDXQpSgIiOoiraHhXZzZ8Mz9cSiaghtS+JIaUpX2nmxSlKAUpSgFKUoBSlKAUpSgFVVhv+8XO3+YDP8A2rA1atVVhv8AvFzt/mAz/wBqwNAWrSlKAUpSgFKUoBSlKAUpSgFKUoBSlKAUqv5y7ZmQknLGBfhHtGK5m53BUiKKrql6jgXeAlKUo6l6yiIiA+gADdque3f29l/VmPh6+V8VCnYk3pufPFxMKdljZatKqrnt39vZf1Zj4enPbv7ey/qzHw9Rm1S+25XNQ9H23LVpVVc9u/t7L+rMfD057d/b2X9WY+Hpm1S+24zUPR9ty1aVVXPbv7ey/qzHw9Oe3f29l/VmPh6ZtUvtuM1D0fbctWlVVz27+3sv6sx8PTnt39vZf1Zj4embVL7bjNQ9H23LVpVVc9u/t7L+rMfD057d/b2X9WY+Hpm1S+24zUPR9tycX1HXTL2XOxdj3CWBuJ1HOEomTOgRcjN4KYgiqZM5TFOUp9oiUSjqACFfntyDs/crbOXKRve2b1ZxNtQ1uS55a/CNY79KrIpx6EUjHAZU6gIAJ2HODcMCiJkXAAYAMUodo89u/t7L+rMfD1o7ftUbVlJ2btyccxz+53pZGXcIMWBTvHJUiJAoceb9Y7Eyh9Gu4fSYwizapfbcZqHo+25eNKqrnt39vZf1Zj4enPbv7ey/qzHw9M2qX23Gah6PtuWrSqq57d/b2X9WY+Hpz27+3sv6sx8PTNql9txmoej7blq0qque3f29l/VmPh6c9u/t7L+rMfD0zapfbcZqHo+25atKqrnt39vZf1Zj4enPbv7ey/qzHw9M2qX23Gah6PtuWrSqq57d/b2X9WY+Hpz27+3sv6sx8PTNql9txmoej7blq0qque3f29l/VmPh69EkLvTHcF8SSg/MCjVmJfT8+1AB/wDmmbVL7bjNQ9H23LTpUFYZUhGKIs7vfIMpBE20dhTbFy6AIKEDrEoDqIbRERASj1iGgiq64qT8Ykv7NVPlv/kiNxxjGPJCYwiPyxJh1j8wPVtKy6w4z9aS/jMp/wDdWrMr4pf+iOcKUpViRSlKAUpSgFK4/wCWFD4xneUDiJjluxJi74EIS5FPk2LjHb9bjALLYoKLX9IJQ69R0EA1DX5hCOWJNPcCxmYcm4WxPdtv44iYmNND29dKbtkm9mecH5yq0buBFZFIyaqRRNoUBOHUAgXQNFBarTRS7Vadx0rlS+OVBlGwZmAxre7rEdnXlMMXNwO305MuCQ8dHAuCbdsUxuGd07P5wG2iQheGYwAIaBX1tflbXrk+3LUhMYWtazy+rgm5aHdqqSp3EGyRjdouHxFkigo4SOVVuKRS7REVg1N5oiMXGRhxep1NSuLbeuC5+kG7VMrWPD/LxstWkxUQYyTkWaSvyckVJ6gcATOcNoFOCagCXUwlMA6VYsZmjlF3k6d37j7GtsyOP2NyKwIRq7lZOefN0HYtXMgifUG5ClOU5gRMXcYiZh3gOgCcDDgZ0bSuObGzDygrSSvKKk422riuS5snurYthEZN6Zqzc83FZYVjKlExGaSKInKRINwm3gAF11qWXHyjMwYxgciw+Q7PtuSu+y4VlcMa4hTOEo2YZuXJkAJw1RMogqU6ZiiG84DqBg6uqlxjDZ01SotjZzkp7a6bzK8Zb8dOrrqqczhHCq6DduJtUkzKKlKJ1Sl0A5gKBRMA7Q0qpeXI2ycHJ9uyZx9kVK120XDvHEqQsZx3L5ECl/RIr8QvNtQA4CcCHN5wabRDUYStdhVQ2uw6DpWDAmMeDjjnMJjGaIiIiOoiOwKzqqQKUpQClKUApSlAUlm5y5RutoVFwoQox6Y6FOIBrxFKV885ftY0/hyf9VWleO49vMx/U5U5vEZcsZ+tJfxmU/8AurVmVitExbu5ZocBBRKXfGOUQ0EOI4OqX/yRQgh+4QGsqvXwf6pHUFKUqxIpSlAKUpQEFn8YBOZgtHK/y4KHkrFSsZzDm27nPPRbjv4m4Nmzm/o2m3b/AEhp15eW8fhlPHkxYJpYYwJYiROdghxuFsVIp+puLrrs09IempfSptZNrKiyfhK4blyDEZZxve7G2bpj4tWCdmkoUJRm/jzqAqCZ0uKkYhyKgBynKcPSYogID1R9fk03e3hLPlYjNMgfIlpSL6RC45OPBy1e8+6nbY7EFCFTbCUCAmmRQOECZNphEBEb9pU3mSo2jn22uS9crN+/nryy4a4ZiUvaJvN05CEK2IBmSBUwaJkBYdqY7dCm1ESlAoDvNqcf4hyZb7jHrm2bdzm9iscvbmNcy0I2igJJFFRxzlZknIlWAStlHAiYQ4Qn2mMTfoIjXQdKX2L7Oe5vku3Q+k7hlojLoxq6t4Fvm1zlhCHPDyRkDIuCLCKujtBRMxi7BKmJQMPnCPXX0c8mK4rqjr7kcj5ORlbsvlkwihfsoUGrKMYNFhWTQQbCsY5txzKGMY6oiInDTQC6D0BSl9i+yEkQyJ0zmcA+V8hC2wUhmx0kAJ8rC6EQOQ4arGHggYDAbaQv6PbvExtn1zHjwMtYsunGgy/yV5Sxi0dz3m/H5vxC6b+HuLv0+jcGv0hUxpUW/Ei3nafCPa8xYNmO/fzdEiW7TTdtKAa6fN6K+9KVBApSlAKUpQClKUBR2cv2safw5P8Aqq0rf5Jsqcu+4ivIRg6dJtGxGqooo7wKpqY+0R16h2qFHT94UrynG8NOj4iOKGFtWnNmyo4o20joees2LnnJX5l3TJ4UoEFw0OBTHKHoKcpgMQ2nzCJRENR0ENR10/RgXtvcP3WXh6m1K9xFIlxO1r3R6OKTBE7WiE9GBe29w/dZeHp0YF7b3D91l4eptSoy0vo9XuVwJfQhPRgXtvcP3WXh6dGBe29w/dZeHqbUplpfR6vcYEvoQnowL23uH7rLw9OjAvbe4fusvD1NqUy0vo9XuMCX0IT0YF7b3D91l4enRgXtvcP3WXh6m1KZaX0er3GBL6EJ6MC9t7h+6y8PTowL23uH7rLw9TalMtL6PV7jAl9CE9GBe29w/dZeHqFWBDSd1XXkqCkLvlk29nXQhCMTIpNAOogeFjHwmVEUBAT8V8qUBKBQ2FIGmoCY111VWG/7xc7f5gM/9qwNMtL6PV7jAl9Df9GBe29w/dZeHp0YF7b3D91l4eptSmWl9Hq9xgS+hCejAvbe4fusvD06MC9t7h+6y8PU2pTLS+j1e4wJfQhPRgXtvcP3WXh6dGBe29w/dZeHqbUplpfR6vcYEvoQnowL23uH7rLw9OjAvbe4fusvD1NqUy0vo9XuMCX0IT0YF7b3D91l4enRgXtvcP3WXh6m1KZaX0er3GBL6EJ6MC9t7h+6y8PXomMUQH9LeE+qUfSUeaF+f6SoAIf+amlKZaX07vcYEvoYcVFMIVknHxqHCQT1EAEwmMYR6xMYwiImMI9YiIiI0rMpWyShViNUklYhSlKkkUpSgFKUoBSlKAUpSgFKUoBVVYb/ALxc7f5gM/8AasDU4vqzIHI1lztg3Q3MvEXFHOIt6QhtpxRWTEhhKb/CYANqBvSAgAh6K/O3kC8mHOUTylr6uDPd1XNMReLZZRpDfKb1wo2lJZRoRsnIJlUMYDASNKiUBMG4CqtwAQ4egAfpfSlKAUpSgFKUoBSlKAUpSgFKUoBSlKAUpSgFKUoBSlKAUpSgFKUoCCzOTvkjM1r4i+ROL5SwMvN/KHOdvN+YrM0+Fwtg79/PNd24NvD00Nu1CS3PdlrWVDrXFeVyxUDFNtOM+k3ibVul/wDsooIFL/Ma5x5QmXsbYU5VOKrvyndrO3YdWzbqYkduinEgrncxJik8wojqIEOPo06qrTlF8pXBeWZvFcjZ12Y5fwLSclQNed4M3juChZFFkmciBmoKIJLOlE19yRljAUmw4kHf1UB1Le+cbatdpYkxDLxk9DXvcSMEnJtpRPmrdM7ZwsLkFCgYihQ5uIaalDztdwaaDK7Wv2xr5hD3LZV6QVwRCZzkPIRcii7bFMT9cBVTMYoCX5w16vnr8uY6exQpjJ5E5EOyuK1o3lHM5OXYMbbUi0DRq8YY6TkIoROdJooZNRXZqbiJkOPXuEKl2bEbSvzpsuzkstGymPjRdnsbsfW9GqKxL1ZGTUVfCVFsBOeAmxOlzgETaimIkEwCJqA71uDNlkN8W3hlCxrgg7ya2jFv366cTLIrJnVaonUMgZVLeCZhEglHUBEPoHTSs3pfx3HR9srXfelu23IXW1QXjo+Tl0EFnB1SFMCaJVDFMqICbTzQ6/orgmNStadfZcuXHV/WJcMexwrNsJU+ObIPDwfnJALNJ0uL1Yp3ZCEU4aZSbip7wMJeoBw84u8eQtw3VJS942C0lnePLfbv7VyfboqtZ1omzEyYwj9BYrhMwmMchipkE5VyAfTqKNAfptSopiZ85k8WWfIvLccW+u6gWCykS4XUWVYGM3IItzqKeecxNdomP5wiGo9etSugFKUoBSlKAUpSgFKUoBSlKAUpSgFKUoBSlKAUpSgFKUoBXhVFJcuxZIihdddDFAQ1/nSlAegKUBEwFABN6R09NeU0kkSAmimUhA9BShoAfypSgCSKSBdiKREy666FKAB/8V/FG7dY5DqoJnMmOpBMUBEo/u+j0BSlAfSlKUApSlAKUpQClKUApSlAKUpQH//Z\n", 51 | "text/plain": [ 52 | "" 53 | ] 54 | }, 55 | "execution_count": 2, 56 | "metadata": {}, 57 | "output_type": "execute_result" 58 | } 59 | ], 60 | "source": [ 61 | "Image('./figures/gcn_prediction.jpg')" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": {}, 67 | "source": [ 68 | "Let's assume that shapes of the adjacency matrix, feature matrix are same as the shapes of the matrix used in a 'gcn-node_classification' tutorial." 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": 3, 74 | "metadata": {}, 75 | "outputs": [], 76 | "source": [ 77 | "num_nodes = 50\n", 78 | "num_features = 50\n", 79 | "X = tf.placeholder(tf.float64, [None, num_nodes, num_features])\n", 80 | "A = tf.placeholder(tf.float64, [None, num_nodes, num_nodes])\n", 81 | "Y_truth = tf.placeholder(tf.float64, [None,])" 82 | ] 83 | }, 84 | { 85 | "cell_type": "markdown", 86 | "metadata": {}, 87 | "source": [ 88 | "An implementation of graph convolution layer is same also." 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": 4, 94 | "metadata": {}, 95 | "outputs": [], 96 | "source": [ 97 | "def graph_conv(_X, _A, output_dim):\n", 98 | " output = tf.layers.dense(_X, units=output_dim, use_bias=True)\n", 99 | " output = tf.matmul(_A, output)\n", 100 | " output = tf.nn.relu(output)\n", 101 | " return output" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "We have to implement the readout function as described above.
\n", 109 | "There are two types of the implementation: node-wise summation (nw) and graph gathering (gg).
\n", 110 | "Equations are as follow, respectively.\n", 111 | "\n", 112 | "$$ R_{nw} = \\tau(\\sum_{i \\in G} MLP(H_{i}^{L})) $$\n", 113 | "$$ R_{gg} = \\tau(\\sum_{i \\in G} \\sigma(MLP_1(H_{i}^{L} | H_{i}^{0})) \\odot MLP_2(H_{i}^{L}))$$\n", 114 | "\n", 115 | "Notations :\n", 116 | "* $ \\tau $ : ReLU activation (or other non-linear activations)\n", 117 | "* $ \\sigma $ : sigmoid activation\n", 118 | "* $ \\odot $ : elementwise-multiplication - Hadamard product \n", 119 | "* $ (\\cdot|\\cdot) $ : concatenation\n", 120 | " \n", 121 | "Please refer to the following article for the more detail.
\n", 122 | "Gilmer, Justin, et al. \"Neural message passing for quantum chemistry.\" arXiv preprint arXiv:1704.01212 (2017)." 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "execution_count": 5, 128 | "metadata": {}, 129 | "outputs": [], 130 | "source": [ 131 | "def readout_nw(_X, output_dim):\n", 132 | " # _X : final node embeddings\n", 133 | " output = tf.layers.dense(_X, output_dim, use_bias=True)\n", 134 | " output = tf.reduce_sum(output, axis=1)\n", 135 | " output = tf.nn.relu(output)\n", 136 | " \n", 137 | " return output" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 6, 143 | "metadata": {}, 144 | "outputs": [], 145 | "source": [ 146 | "def readout_gg(_X, X, output_dim):\n", 147 | " # _X : final node embeddings\n", 148 | " # X : initial node features\n", 149 | " val1 = tf.layers.dense(tf.concat([_X, X], axis=2), output_dim, use_bias=True)\n", 150 | " val1 = tf.nn.sigmoid(val1)\n", 151 | " val2 = tf.layers.dense(_X, output_dim, use_bias=True)\n", 152 | " output = tf.multiply(val1, val2)\n", 153 | " output = tf.reduce_sum(output, axis=1)\n", 154 | " output = tf.nn.relu(output)\n", 155 | " \n", 156 | " return output" 157 | ] 158 | }, 159 | { 160 | "cell_type": "markdown", 161 | "metadata": {}, 162 | "source": [ 163 | "We finished preparing necessary functions in the architecture.
\n", 164 | "Therefore, the implementation of the overall architecture is as below." 165 | ] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "execution_count": 7, 170 | "metadata": {}, 171 | "outputs": [ 172 | { 173 | "data": { 174 | "text/plain": [ 175 | "" 176 | ] 177 | }, 178 | "execution_count": 7, 179 | "metadata": {}, 180 | "output_type": "execute_result" 181 | } 182 | ], 183 | "source": [ 184 | "gconv1 = graph_conv(X, A, 32)\n", 185 | "gconv2 = graph_conv(gconv1, A, 32)\n", 186 | "gconv3 = graph_conv(gconv2, A, 32)\n", 187 | "graph_feature = readout_gg(gconv3, gconv1, 128)\n", 188 | "graph_feature" 189 | ] 190 | }, 191 | { 192 | "cell_type": "code", 193 | "execution_count": 8, 194 | "metadata": {}, 195 | "outputs": [ 196 | { 197 | "data": { 198 | "text/plain": [ 199 | "" 200 | ] 201 | }, 202 | "execution_count": 8, 203 | "metadata": {}, 204 | "output_type": "execute_result" 205 | } 206 | ], 207 | "source": [ 208 | "Y_pred = tf.layers.dense(graph_feature, 128, use_bias=True, activation=tf.nn.relu)\n", 209 | "Y_pred = tf.layers.dense(Y_pred, 128, use_bias=True, activation=tf.nn.tanh)\n", 210 | "Y_pred = tf.layers.dense(Y_pred, 1, use_bias=True, activation=None)\n", 211 | "Y_pred" 212 | ] 213 | }, 214 | { 215 | "cell_type": "markdown", 216 | "metadata": {}, 217 | "source": [ 218 | "A loss function have to be minimized in this task is an l2-norm." 219 | ] 220 | }, 221 | { 222 | "cell_type": "code", 223 | "execution_count": 9, 224 | "metadata": {}, 225 | "outputs": [ 226 | { 227 | "data": { 228 | "text/plain": [ 229 | "" 230 | ] 231 | }, 232 | "execution_count": 9, 233 | "metadata": {}, 234 | "output_type": "execute_result" 235 | } 236 | ], 237 | "source": [ 238 | "Y_pred = tf.reshape(Y_pred, shape=[-1])\n", 239 | "Y_truth = tf.reshape(Y_truth, shape=[-1])\n", 240 | "loss = tf.reduce_mean(tf.pow(Y_truth - Y_pred,2))\n", 241 | "loss" 242 | ] 243 | }, 244 | { 245 | "cell_type": "markdown", 246 | "metadata": {}, 247 | "source": [ 248 | "Yes, we have completed all necessary preparations for training.\n", 249 | "\n", 250 | "I upload all codes which implement the supervised learning of prediction molecular properties at the 'gnn-molecule' folder.
\n", 251 | "Scripts for preprocessing also exist. Hope you enjoy the graph neural networks from this moment!\n" 252 | ] 253 | } 254 | ], 255 | "metadata": { 256 | "kernelspec": { 257 | "display_name": "Python 3", 258 | "language": "python", 259 | "name": "python3" 260 | }, 261 | "language_info": { 262 | "codemirror_mode": { 263 | "name": "ipython", 264 | "version": 3 265 | }, 266 | "file_extension": ".py", 267 | "mimetype": "text/x-python", 268 | "name": "python", 269 | "nbconvert_exporter": "python", 270 | "pygments_lexer": "ipython3", 271 | "version": "3.6.5" 272 | } 273 | }, 274 | "nbformat": 4, 275 | "nbformat_minor": 2 276 | } 277 | -------------------------------------------------------------------------------- /tutorials/ggnn.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Gated Graph Neural Network\n", 8 | "\n", 9 | "We have found that GCN and GAT are CNN-like versions of graph neural networks. GGNN, on the other hand, is the RNN-like version of the node updating method. \n", 10 | "\n", 11 | "First, let's look at the message passing neural network (MPNN) framework. The MPNN framework updates the route node with the following formula.
\n", 12 | "\n", 13 | "$$ H_{i}^{(l+1)} = U(H_{i}^{(l)}, m^{(l+1)}) $$\n", 14 | "\n", 15 | "The i-th node, which is a route node, is newly updated through the message state, $m^{(i+1)}$ from the neighboring nodes and previous node state, $H^{(l)}$.
\n", 16 | "\n", 17 | "Updating message state can be written as a general formulation as follow.\n", 18 | "\n", 19 | "$$ m^{(l+1)} = \\sum_{j \\in N_{i}} M(H_i^{(l)}, H_j^{(l)}, e_{ij}) $$\n", 20 | "\n", 21 | "If we know the initial edge information - $e_{ij}$, we can update the message states differently for different relations, for example a single bond, a double bond and an aromatic bond will transfer a different message to the route node.
\n", 22 | "For simpliticy, we will only consider just connectivity between the node pairs, i.e.) $A_{ij} =1$ for connected node pairs, and zero otherwise.\n", 23 | "\n", 24 | "In GGNN framework, message function is defined as simple summation of the neighbor node states.\n", 25 | "\n", 26 | "$$ m^{(l+1)} = \\sum_{j \\in N_{i}} H_j^{(l)} $$\n", 27 | "\n", 28 | "And the gated recurrent unit (GRU) is used for the node updating. Finally, the node updating is re-written as follow.\n", 29 | "\n", 30 | "$$ H_i^{(l+1)} = GRU(H_i^{(l)}, \\sum_{j \\in N_i} H_i^{(l)}) $$\n", 31 | "\n", 32 | "We will implement the updating function in the GGNN framework." 33 | ] 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": 1, 38 | "metadata": {}, 39 | "outputs": [ 40 | { 41 | "name": "stderr", 42 | "output_type": "stream", 43 | "text": [ 44 | "/Users/Lulu/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n", 45 | " from ._conv import register_converters as _register_converters\n" 46 | ] 47 | } 48 | ], 49 | "source": [ 50 | "import tensorflow as tf" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": 2, 56 | "metadata": {}, 57 | "outputs": [], 58 | "source": [ 59 | "def ggnn(_X, _A, output_dim, num_layer):\n", 60 | " num_nodes = int(_X.get_shape()[1])\n", 61 | " input_dim = int(_X.get_shape()[2])\n", 62 | " \n", 63 | " if( input_dim != output_dim ):\n", 64 | " _X = tf.layers.dense(_X, units=output_dim, use_bias=False)\n", 65 | " \n", 66 | " # Message state\n", 67 | " _m = tf.matmul(_A, _X)\n", 68 | " \n", 69 | " # Update node state using GRU cell\n", 70 | " X_total = []\n", 71 | " cell = tf.contrib.rnn.GRUCell(output_dim, name='GRUcell'+str(num_layer))\n", 72 | " \n", 73 | " for i in range(num_nodes):\n", 74 | " mi = tf.expand_dims(_m[:,i,:],1)\n", 75 | " hi = _X[:,i,:]\n", 76 | " \n", 77 | " _, _h = tf.nn.dynamic_rnn(cell, mi, initial_state=hi)\n", 78 | " X_total.append(tf.expand_dims(_h, 1))\n", 79 | " \n", 80 | " output = tf.concat(X_total, 1)\n", 81 | " \n", 82 | " return output" 83 | ] 84 | }, 85 | { 86 | "cell_type": "markdown", 87 | "metadata": {}, 88 | "source": [ 89 | "Let's check if our code is correct." 90 | ] 91 | }, 92 | { 93 | "cell_type": "code", 94 | "execution_count": 3, 95 | "metadata": {}, 96 | "outputs": [ 97 | { 98 | "data": { 99 | "text/plain": [ 100 | "" 101 | ] 102 | }, 103 | "execution_count": 3, 104 | "metadata": {}, 105 | "output_type": "execute_result" 106 | } 107 | ], 108 | "source": [ 109 | "X = tf.placeholder(tf.float64, [None, 50, 58])\n", 110 | "A = tf.placeholder(tf.float64, [None, 50, 50])\n", 111 | "\n", 112 | "ggnn1 = ggnn(X, A, 32, 1)\n", 113 | "ggnn1" 114 | ] 115 | }, 116 | { 117 | "cell_type": "markdown", 118 | "metadata": {}, 119 | "source": [ 120 | "That's right. We implemented a GGNN node updating method simply by using GRU cell.\n", 121 | "\n", 122 | "However, I have implemented it using the for statement here. I wonder if it can be implemented better by using tensorflow. In particular, when a graph neural network is applied to a molecule, a dynamic computational graph should be used when the number of atoms varies. If you know about this, please comment." 123 | ] 124 | } 125 | ], 126 | "metadata": { 127 | "kernelspec": { 128 | "display_name": "Python 3", 129 | "language": "python", 130 | "name": "python3" 131 | }, 132 | "language_info": { 133 | "codemirror_mode": { 134 | "name": "ipython", 135 | "version": 3 136 | }, 137 | "file_extension": ".py", 138 | "mimetype": "text/x-python", 139 | "name": "python", 140 | "nbconvert_exporter": "python", 141 | "pygments_lexer": "ipython3", 142 | "version": "3.6.5" 143 | } 144 | }, 145 | "nbformat": 4, 146 | "nbformat_minor": 2 147 | } 148 | --------------------------------------------------------------------------------