├── .github └── CODEOWNERS ├── .gitignore ├── GNNs ├── DGL │ ├── gcn_node_classification.ipynb │ └── rgcn_node_classification.ipynb ├── PyG │ ├── gcn_link_prediction.ipynb │ ├── gcn_node_classification.ipynb │ └── hgat_node_classification.ipynb └── Spektral │ └── gcn_node_classification.ipynb ├── README.md ├── algos ├── centrality.ipynb ├── classification.ipynb ├── community.ipynb ├── embedding.ipynb ├── pathfinding.ipynb ├── similarity.ipynb └── topologicalLinkPrediction.ipynb ├── applications ├── fraud_detection │ ├── fraud_detection.ipynb │ └── gsql │ │ ├── amounts.gsql │ │ ├── component_size.gsql │ │ ├── degrees.gsql │ │ └── downsample.gsql ├── large_language_models │ └── TigerGraph_LangChain_Demo.ipynb ├── nodepiece │ ├── nodepiece.ipynb │ └── nodepiece_gnn.ipynb └── recommendation │ └── recommendation.ipynb ├── basics ├── data_loaders.ipynb ├── datasets.ipynb ├── feature_engineering.ipynb ├── gsql_101.ipynb ├── gsql_102.ipynb ├── pyTigergraph_101.ipynb └── template_query.ipynb ├── cloud_deployment └── google_vertexai │ ├── Dockerfile │ ├── gcp_vertexai_deploy.ipynb │ ├── input.json │ └── request.json ├── config.json └── environments ├── tg-tensorflow-cpu.yml └── tg-torch-cpu.yml /.github/CODEOWNERS: -------------------------------------------------------------------------------- 1 | * @qe-tigergraph -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | **/ldbc_snb_data 2 | .ipynb_checkpoints 3 | *.tar.gz 4 | tmp.ipynb 5 | .ipynb_checkpoints 6 | **/tmp 7 | **/__pycache__ 8 | .DS_Store -------------------------------------------------------------------------------- /GNNs/DGL/gcn_node_classification.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Graph Convolutional Network\n", 8 | "This notebook demonstrates the training of [Graph Convolutional Networks (GCN)](https://arxiv.org/pdf/1609.02907.pdf) with TigerGraph. [DGL](https://www.dgl.ai/)'s implementation of GCN is used here. We train the model on the Cora dataset from [PyG datasets](https://pytorch-geometric.readthedocs.io/en/latest/modules/datasets.html#torch_geometric.datasets.Planetoid) with TigerGraph as the data store. The dataset contains 2708 machine learning papers and 10556 citation links between the papers. Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from a dictionary. The dictionary consists of 1433 unique words. Each paper is classified into one of seven classes based on the topic. The goal is to predict the class of each vertex in the graph.\n", 9 | "\n", 10 | "The following libraries are required to run this notebook. Uncomment to install them if necessary. You might need to restart the kernel after installing." 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": null, 16 | "metadata": {}, 17 | "outputs": [], 18 | "source": [ 19 | "#!pip install torch==1.12.0 --extra-index-url https://download.pytorch.org/whl/cpu\n", 20 | "#!pip install dgl -f https://data.dgl.ai/wheels/repo.html\n", 21 | "#!pip install psutil # Required for DGL\n", 22 | "#!pip install pyTigerGraph[gds]\n", 23 | "#!pip install tensorboard # If you use tensorboard for visualization later" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "## Table of Contents\n", 31 | "* [Data Processing](#data_processing) \n", 32 | "* [Train on whole graph](#train_whole) \n", 33 | "* [Train on neighborhood subgraphs](#train_subgraph) \n", 34 | "* [Inference](#inference)" 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "metadata": {}, 40 | "source": [ 41 | "## Data Processing " 42 | ] 43 | }, 44 | { 45 | "cell_type": "markdown", 46 | "metadata": {}, 47 | "source": [ 48 | "### Connect to TigerGraph\n", 49 | "\n", 50 | "The `TigerGraphConnection` class represents a connection to the TigerGraph database. Under the hood, it stores the necessary information to communicate with the database. It is able to perform quite a few database tasks. Please see its [documentation](https://docs.tigergraph.com/pytigergraph/current/intro/) for details.\n", 51 | "\n", 52 | "To connect your database, modify the `config.json` file accompanying this notebook. Set the value of `getToken` based on whether token auth is enabled for your database. Token auth is always enabled for tgcloud databases. " 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": 1, 58 | "metadata": {}, 59 | "outputs": [], 60 | "source": [ 61 | "from pyTigerGraph import TigerGraphConnection\n", 62 | "import json\n", 63 | "\n", 64 | "# Read in DB configs\n", 65 | "with open('../../config.json', \"r\") as config_file:\n", 66 | " config = json.load(config_file)\n", 67 | " \n", 68 | "conn = TigerGraphConnection(\n", 69 | " host=config[\"host\"],\n", 70 | " username=config[\"username\"],\n", 71 | " password=config[\"password\"],\n", 72 | ")" 73 | ] 74 | }, 75 | { 76 | "cell_type": "markdown", 77 | "metadata": {}, 78 | "source": [ 79 | "### Ingest Data" 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": 2, 85 | "metadata": {}, 86 | "outputs": [ 87 | { 88 | "data": { 89 | "application/vnd.jupyter.widget-view+json": { 90 | "model_id": "cb64eaa96c72469893d6931cd5535abf", 91 | "version_major": 2, 92 | "version_minor": 0 93 | }, 94 | "text/plain": [ 95 | "Downloading: 0%| | 0/166537 [00:00\n", 253 | "\n", 254 | "We first train the model on the whole graph. This will **NOT** work when the graph is large. See the section of training on subgraphs for real use. However, we still include this example for illustration purpose. Hyperparameters for the model and training environment are defined below." 255 | ] 256 | }, 257 | { 258 | "cell_type": "code", 259 | "execution_count": null, 260 | "metadata": {}, 261 | "outputs": [], 262 | "source": [ 263 | "# Hyperparameters\n", 264 | "hp = {\"hidden_dim\": 64, \n", 265 | " \"num_layers\": 2, \n", 266 | " \"dropout\": 0.6, \n", 267 | " \"lr\": 0.01, \n", 268 | " \"l2_penalty\": 5e-4}" 269 | ] 270 | }, 271 | { 272 | "cell_type": "markdown", 273 | "metadata": {}, 274 | "source": [ 275 | "### Construct graph loader" 276 | ] 277 | }, 278 | { 279 | "cell_type": "markdown", 280 | "metadata": {}, 281 | "source": [ 282 | "The `GraphLoader` can get the whole graph from database all at once (`num_batches=1`). See the tutorial on dataloaders for details." 283 | ] 284 | }, 285 | { 286 | "cell_type": "code", 287 | "execution_count": null, 288 | "metadata": {}, 289 | "outputs": [], 290 | "source": [ 291 | "graph_loader = conn.gds.graphLoader(\n", 292 | " v_in_feats=[\"x\"],\n", 293 | " v_out_labels=[\"y\"],\n", 294 | " v_extra_feats=[\"train_mask\", \"val_mask\", \"test_mask\"],\n", 295 | " num_batches=1,\n", 296 | " output_format=\"DGL\",\n", 297 | " shuffle=False\n", 298 | ")" 299 | ] 300 | }, 301 | { 302 | "cell_type": "code", 303 | "execution_count": null, 304 | "metadata": {}, 305 | "outputs": [], 306 | "source": [ 307 | "# Get the whole graph from the loader in DGL format\n", 308 | "data = graph_loader.data\n", 309 | "\n", 310 | "data" 311 | ] 312 | }, 313 | { 314 | "cell_type": "markdown", 315 | "metadata": {}, 316 | "source": [ 317 | "### Construct model and optimizer" 318 | ] 319 | }, 320 | { 321 | "cell_type": "markdown", 322 | "metadata": {}, 323 | "source": [ 324 | "We build a GCN model with 2 convolutional layers, and use the Adam optimizer with a learning rate of 0.01." 325 | ] 326 | }, 327 | { 328 | "cell_type": "code", 329 | "execution_count": null, 330 | "metadata": {}, 331 | "outputs": [], 332 | "source": [ 333 | "import dgl.function as fn\n", 334 | "import dgl.nn.pytorch as dglnn\n", 335 | "import torch\n", 336 | "import torch.nn as nn\n", 337 | "import torch.nn.functional as F" 338 | ] 339 | }, 340 | { 341 | "cell_type": "code", 342 | "execution_count": null, 343 | "metadata": {}, 344 | "outputs": [], 345 | "source": [ 346 | "class GCN(nn.Module):\n", 347 | " def __init__(self):\n", 348 | " super(GCN, self).__init__()\n", 349 | " self.layer1 = dglnn.conv.GraphConv(1433, hp['hidden_dim'])\n", 350 | " self.layer2 = dglnn.conv.GraphConv(hp['hidden_dim'], 7)\n", 351 | "\n", 352 | " def forward(self, g, features):\n", 353 | " x = F.relu(self.layer1(g, features))\n", 354 | " x = self.layer2(g, x)\n", 355 | " return x\n", 356 | "\n", 357 | "model = GCN()\n", 358 | "print(model)" 359 | ] 360 | }, 361 | { 362 | "cell_type": "code", 363 | "execution_count": null, 364 | "metadata": {}, 365 | "outputs": [], 366 | "source": [ 367 | "optimizer = torch.optim.Adam(\n", 368 | " model.parameters(), lr=hp[\"lr\"], weight_decay=hp[\"l2_penalty\"]\n", 369 | ")\n", 370 | "\n", 371 | "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")" 372 | ] 373 | }, 374 | { 375 | "cell_type": "code", 376 | "execution_count": null, 377 | "metadata": {}, 378 | "outputs": [], 379 | "source": [ 380 | "from datetime import datetime\n", 381 | "from pyTigerGraph.gds.metrics import Accumulator, Accuracy\n", 382 | "from torch.utils.tensorboard import SummaryWriter" 383 | ] 384 | }, 385 | { 386 | "cell_type": "code", 387 | "execution_count": null, 388 | "metadata": {}, 389 | "outputs": [], 390 | "source": [ 391 | "log_dir = \"logs/cora/gcn/wholegraph/\" + datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n", 392 | "tb_log = SummaryWriter(log_dir)\n", 393 | "logs = {}\n", 394 | "data = data.to(device)\n", 395 | "for epoch in range(20):\n", 396 | " # Train\n", 397 | " model.train()\n", 398 | " acc = Accuracy()\n", 399 | " # Forward pass\n", 400 | " out = model(data, data.ndata[\"x\"])\n", 401 | " # Calculate loss\n", 402 | " loss = F.cross_entropy(out[data.ndata[\"train_mask\"]], data.ndata[\"y\"][data.ndata[\"train_mask\"]])\n", 403 | " # Backward pass\n", 404 | " optimizer.zero_grad()\n", 405 | " loss.backward()\n", 406 | " optimizer.step()\n", 407 | " # Evaluate\n", 408 | " val_acc = Accuracy()\n", 409 | " with torch.no_grad():\n", 410 | " pred = out.argmax(dim=1)\n", 411 | " acc.update(pred[data.ndata[\"train_mask\"]], data.ndata[\"y\"][data.ndata[\"train_mask\"]])\n", 412 | " valid_loss = F.cross_entropy(out[data.ndata[\"val_mask\"]], data.ndata[\"y\"][data.ndata[\"val_mask\"]])\n", 413 | " val_acc.update(pred[data.ndata[\"val_mask\"]], data.ndata[\"y\"][data.ndata[\"val_mask\"]])\n", 414 | " # Logging\n", 415 | " logs[\"loss\"] = loss.item()\n", 416 | " logs[\"val_loss\"] = valid_loss.item()\n", 417 | " logs[\"acc\"] = acc.value\n", 418 | " logs[\"val_acc\"] = val_acc.value\n", 419 | " print(\n", 420 | " \"Epoch: {:02d}, Train Loss: {:.4f}, Valid Loss: {:.4f}, Train Accuracy: {:.4f}, Valid Accuracy: {:.4f}\".format(\n", 421 | " epoch, logs[\"loss\"], logs[\"val_loss\"], logs[\"acc\"], logs[\"val_acc\"]\n", 422 | " )\n", 423 | " )\n", 424 | " tb_log.add_scalars(\n", 425 | " \"Loss\", {\"Train\": logs[\"loss\"], \"Validation\": logs[\"val_loss\"]}, epoch\n", 426 | " )\n", 427 | " tb_log.add_scalars(\n", 428 | " \"Accuracy\", {\"Train\": logs[\"acc\"], \"Validation\": logs[\"val_acc\"]}, epoch\n", 429 | " )\n", 430 | " tb_log.flush()" 431 | ] 432 | }, 433 | { 434 | "cell_type": "markdown", 435 | "metadata": {}, 436 | "source": [ 437 | "### Test the model" 438 | ] 439 | }, 440 | { 441 | "cell_type": "code", 442 | "execution_count": null, 443 | "metadata": {}, 444 | "outputs": [], 445 | "source": [ 446 | "model.eval()\n", 447 | "acc = Accuracy()\n", 448 | "with torch.no_grad():\n", 449 | " pred = model(data, data.ndata[\"x\"]).argmax(dim=1)\n", 450 | " acc.update(pred[data.ndata[\"test_mask\"]], data.ndata[\"y\"][data.ndata[\"test_mask\"]])\n", 451 | "print(\"Accuracy: {:.4f}\".format(acc.value))" 452 | ] 453 | }, 454 | { 455 | "cell_type": "markdown", 456 | "metadata": {}, 457 | "source": [ 458 | "## Train on Neighborhood Subgraphs \n", 459 | "\n", 460 | "Alternatively, we train the model on the neighborhood subgraphs. Each subgraph contains the 2 hop neighborhood of certain seed vertices. This method will allow us to train the model on graphs that are way larger than the CORA dataset because we don't load the whole graph into memory all at once. \n", 461 | "\n", 462 | "We will use the same parameters as before, but we will use the NeighborLoader to load subgraphs. Once we finish iterating over all the subgraphs generated by the loader, it is guaranteed to cover all vertices in the graph (except for those filtered by a user provided mask). " 463 | ] 464 | }, 465 | { 466 | "cell_type": "code", 467 | "execution_count": null, 468 | "metadata": {}, 469 | "outputs": [], 470 | "source": [ 471 | "# Hyperparameters\n", 472 | "hp = {\"batch_size\": 64, \n", 473 | " \"num_neighbors\": 10, \n", 474 | " \"num_hops\": 2, \n", 475 | " \"hidden_dim\": 64, \n", 476 | " \"num_layers\": 2, \n", 477 | " \"dropout\": 0.6, \n", 478 | " \"lr\": 0.01, \n", 479 | " \"l2_penalty\": 5e-4}" 480 | ] 481 | }, 482 | { 483 | "cell_type": "markdown", 484 | "metadata": {}, 485 | "source": [ 486 | "### Construct neighborhood subgraph loader\n", 487 | "\n", 488 | "Here we construct 3 subgraph loaders. The `train_loader` only uses vertices in the training set as seeds, the `valid_loader` only uses vertices in the validation set, and the `test_loader` only uses vertices in the test set." 489 | ] 490 | }, 491 | { 492 | "cell_type": "code", 493 | "execution_count": null, 494 | "metadata": {}, 495 | "outputs": [], 496 | "source": [ 497 | "train_loader = conn.gds.neighborLoader(\n", 498 | " v_in_feats=[\"x\"],\n", 499 | " v_out_labels=[\"y\"],\n", 500 | " v_extra_feats=[\"train_mask\",\"val_mask\",\"test_mask\"],\n", 501 | " output_format=\"DGL\",\n", 502 | " batch_size=hp[\"batch_size\"],\n", 503 | " num_neighbors=hp[\"num_neighbors\"],\n", 504 | " num_hops=hp[\"num_hops\"],\n", 505 | " shuffle=True,\n", 506 | " filter_by=\"train_mask\",\n", 507 | " add_self_loop=True,\n", 508 | ")" 509 | ] 510 | }, 511 | { 512 | "cell_type": "code", 513 | "execution_count": null, 514 | "metadata": {}, 515 | "outputs": [], 516 | "source": [ 517 | "valid_loader = conn.gds.neighborLoader(\n", 518 | " v_in_feats=[\"x\"],\n", 519 | " v_out_labels=[\"y\"],\n", 520 | " v_extra_feats=[\"train_mask\",\"val_mask\",\"test_mask\"],\n", 521 | " output_format=\"DGL\",\n", 522 | " batch_size=hp[\"batch_size\"],\n", 523 | " num_neighbors=hp[\"num_neighbors\"],\n", 524 | " num_hops=hp[\"num_hops\"],\n", 525 | " shuffle=False,\n", 526 | " filter_by=\"val_mask\",\n", 527 | " add_self_loop=True,\n", 528 | ")" 529 | ] 530 | }, 531 | { 532 | "cell_type": "markdown", 533 | "metadata": {}, 534 | "source": [ 535 | "### Construct model and optimizer\n", 536 | "We build a GCN model with 2 convolutional layers, and use the Adam optimizer with a learning rate of 0.01." 537 | ] 538 | }, 539 | { 540 | "cell_type": "code", 541 | "execution_count": null, 542 | "metadata": {}, 543 | "outputs": [], 544 | "source": [ 545 | "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", 546 | "\n", 547 | "model = GCN().to(device)\n", 548 | "\n", 549 | "optimizer = torch.optim.Adam(\n", 550 | " model.parameters(), lr=hp[\"lr\"], weight_decay=hp[\"l2_penalty\"]\n", 551 | ")" 552 | ] 553 | }, 554 | { 555 | "cell_type": "markdown", 556 | "metadata": {}, 557 | "source": [ 558 | "### Train the model" 559 | ] 560 | }, 561 | { 562 | "cell_type": "code", 563 | "execution_count": null, 564 | "metadata": {}, 565 | "outputs": [], 566 | "source": [ 567 | "from datetime import datetime\n", 568 | "\n", 569 | "from pyTigerGraph.gds.metrics import Accumulator, Accuracy\n", 570 | "from torch.utils.tensorboard import SummaryWriter" 571 | ] 572 | }, 573 | { 574 | "cell_type": "code", 575 | "execution_count": null, 576 | "metadata": {}, 577 | "outputs": [], 578 | "source": [ 579 | "log_dir = \"logs/cora/gcn/subgraph/\" + datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n", 580 | "train_log = SummaryWriter(log_dir+\"/train\")\n", 581 | "valid_log = SummaryWriter(log_dir+\"/valid\")\n", 582 | "global_steps = 0\n", 583 | "logs = {}\n", 584 | "for epoch in range(10):\n", 585 | " # Train\n", 586 | " model.train()\n", 587 | " epoch_train_loss = Accumulator()\n", 588 | " epoch_train_acc = Accuracy()\n", 589 | " for bid, batch in enumerate(train_loader):\n", 590 | " batchsize = batch.num_nodes()\n", 591 | " batch.to(device)\n", 592 | " # Forward pass\n", 593 | " out = model(batch, batch.ndata[\"x\"])\n", 594 | " # Calculate loss\n", 595 | " loss = F.cross_entropy(out[batch.ndata[\"is_seed\"]], batch.ndata[\"y\"][batch.ndata[\"is_seed\"]])\n", 596 | " # Backward pass\n", 597 | " optimizer.zero_grad()\n", 598 | " loss.backward()\n", 599 | " optimizer.step()\n", 600 | " epoch_train_loss.update(loss.item() * batchsize, batchsize)\n", 601 | " # Predict on training data\n", 602 | " with torch.no_grad():\n", 603 | " pred = out.argmax(dim=1)\n", 604 | " epoch_train_acc.update(pred[batch.ndata[\"is_seed\"]], batch.ndata[\"y\"][batch.ndata[\"is_seed\"]])\n", 605 | " # Log training status after each batch\n", 606 | " logs[\"loss\"] = epoch_train_loss.mean\n", 607 | " logs[\"acc\"] = epoch_train_acc.value\n", 608 | " print(\n", 609 | " \"Epoch {}, Train Batch {}, Loss {:.4f}, Accuracy {:.4f}\".format(\n", 610 | " epoch, bid, logs[\"loss\"], logs[\"acc\"]\n", 611 | " )\n", 612 | " )\n", 613 | " train_log.add_scalar(\"Loss\", logs[\"loss\"], global_steps)\n", 614 | " train_log.add_scalar(\"Accuracy\", logs[\"acc\"], global_steps)\n", 615 | " train_log.flush()\n", 616 | " global_steps += 1\n", 617 | " # Evaluate\n", 618 | " model.eval()\n", 619 | " epoch_val_loss = Accumulator()\n", 620 | " epoch_val_acc = Accuracy()\n", 621 | " for batch in valid_loader:\n", 622 | " batchsize = batch.num_nodes()\n", 623 | " batch.to(device)\n", 624 | " with torch.no_grad():\n", 625 | " # Forward pass\n", 626 | " out = model(batch, batch.ndata[\"x\"])\n", 627 | " # Calculate loss\n", 628 | " valid_loss = F.cross_entropy(out[batch.ndata[\"val_mask\"]], batch.ndata[\"y\"][batch.ndata[\"val_mask\"]])\n", 629 | " epoch_val_loss.update(valid_loss.item() * batchsize, batchsize)\n", 630 | " # Prediction\n", 631 | " pred = out.argmax(dim=1)\n", 632 | " epoch_val_acc.update(pred[batch.ndata[\"val_mask\"]], batch.ndata[\"y\"][batch.ndata[\"val_mask\"]])\n", 633 | " # Log testing result after each epoch\n", 634 | " logs[\"val_loss\"] = epoch_val_loss.mean\n", 635 | " logs[\"val_acc\"] = epoch_val_acc.value\n", 636 | " print(\n", 637 | " \"Epoch {}, Valid Loss {:.4f}, Valid Accuracy {:.4f}\".format(\n", 638 | " epoch, logs[\"val_loss\"], logs[\"val_acc\"]\n", 639 | " )\n", 640 | " )\n", 641 | " valid_log.add_scalar(\"Loss\", logs[\"val_loss\"], global_steps)\n", 642 | " valid_log.add_scalar(\"Accuracy\", logs[\"val_acc\"], global_steps)\n", 643 | " valid_log.flush()" 644 | ] 645 | }, 646 | { 647 | "cell_type": "markdown", 648 | "metadata": {}, 649 | "source": [ 650 | "### Test the model" 651 | ] 652 | }, 653 | { 654 | "cell_type": "code", 655 | "execution_count": null, 656 | "metadata": {}, 657 | "outputs": [], 658 | "source": [ 659 | "test_loader = conn.gds.neighborLoader(\n", 660 | " v_in_feats=[\"x\"],\n", 661 | " v_out_labels=[\"y\"],\n", 662 | " v_extra_feats=[\"train_mask\",\"val_mask\",\"test_mask\"],\n", 663 | " output_format=\"DGL\",\n", 664 | " batch_size=hp[\"batch_size\"],\n", 665 | " num_neighbors=hp[\"num_neighbors\"],\n", 666 | " num_hops=hp[\"num_hops\"],\n", 667 | " shuffle=False,\n", 668 | " filter_by=\"test_mask\",\n", 669 | " add_self_loop=True,\n", 670 | ")" 671 | ] 672 | }, 673 | { 674 | "cell_type": "code", 675 | "execution_count": null, 676 | "metadata": {}, 677 | "outputs": [], 678 | "source": [ 679 | "model.eval()\n", 680 | "acc = Accuracy()\n", 681 | "for batch in test_loader:\n", 682 | " batch.to(device)\n", 683 | " with torch.no_grad():\n", 684 | " pred = model(batch, batch.ndata[\"x\"]).argmax(dim=1)\n", 685 | " acc.update(pred[batch.ndata[\"test_mask\"]], batch.ndata[\"y\"][batch.ndata[\"test_mask\"]])\n", 686 | "print(\"Accuracy: {:.4f}\".format(acc.value))" 687 | ] 688 | }, 689 | { 690 | "cell_type": "markdown", 691 | "metadata": {}, 692 | "source": [ 693 | "## Inference \n", 694 | "\n", 695 | "Finally, we use the trained model for node classification. At this stage, we typically do inference/prediction for specific nodes instead of random batches, so we will create a new data loader." 696 | ] 697 | }, 698 | { 699 | "cell_type": "code", 700 | "execution_count": null, 701 | "metadata": {}, 702 | "outputs": [], 703 | "source": [ 704 | "infer_loader = conn.gds.neighborLoader(\n", 705 | " v_in_feats=[\"x\"],\n", 706 | " v_out_labels=[\"y\"],\n", 707 | " v_extra_feats=[\"train_mask\",\"val_mask\",\"test_mask\"],\n", 708 | " output_format=\"DGL\",\n", 709 | " num_neighbors=hp[\"num_neighbors\"],\n", 710 | " num_hops=hp[\"num_hops\"],\n", 711 | " shuffle=False,\n", 712 | " add_self_loop=True,\n", 713 | ")" 714 | ] 715 | }, 716 | { 717 | "cell_type": "code", 718 | "execution_count": null, 719 | "metadata": {}, 720 | "outputs": [], 721 | "source": [ 722 | "# Fetch specific nodes by their IDs and do prediction. \n", 723 | "# Each node is represented by a dict with two mandatory keys: primary_id and type.\n", 724 | "input_nodes = [{\"primary_id\": 7, \"type\": \"Paper\"}, \n", 725 | " {\"primary_id\": 999, \"type\": \"Paper\"}]\n", 726 | "data = infer_loader.fetch(input_nodes)" 727 | ] 728 | }, 729 | { 730 | "cell_type": "code", 731 | "execution_count": null, 732 | "metadata": {}, 733 | "outputs": [], 734 | "source": [ 735 | "# The returned data are the neighborhood subgraphs of the input nodes.\n", 736 | "# The original IDs of the nodes in the subgraphs are stored in the \n", 737 | "# `primary_id` attribute.\n", 738 | "data" 739 | ] 740 | }, 741 | { 742 | "cell_type": "code", 743 | "execution_count": null, 744 | "metadata": {}, 745 | "outputs": [], 746 | "source": [ 747 | "# Predict. Predictions for both the input nodes and others in their \n", 748 | "# neighborhoods are generated.\n", 749 | "model.eval()\n", 750 | "pred = model(data, data.ndata[\"x\"]).argmax(dim=1)\n", 751 | "print(\"ID: Label\")\n", 752 | "for i,j in zip(data.extra_data[\"primary_id\"], pred):\n", 753 | " print(\"{}:{}\".format(i, j.item()))" 754 | ] 755 | }, 756 | { 757 | "cell_type": "code", 758 | "execution_count": null, 759 | "metadata": {}, 760 | "outputs": [], 761 | "source": [] 762 | } 763 | ], 764 | "metadata": { 765 | "kernelspec": { 766 | "display_name": "PyTorch", 767 | "language": "python", 768 | "name": "python3" 769 | }, 770 | "language_info": { 771 | "codemirror_mode": { 772 | "name": "ipython", 773 | "version": 3 774 | }, 775 | "file_extension": ".py", 776 | "mimetype": "text/x-python", 777 | "name": "python", 778 | "nbconvert_exporter": "python", 779 | "pygments_lexer": "ipython3", 780 | "version": "3.8.9" 781 | }, 782 | "vscode": { 783 | "interpreter": { 784 | "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6" 785 | } 786 | } 787 | }, 788 | "nbformat": 4, 789 | "nbformat_minor": 4 790 | } 791 | -------------------------------------------------------------------------------- /GNNs/DGL/rgcn_node_classification.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Heterogeneous Graph Convolutional Network\n", 8 | "\n", 9 | "This notebook demonstrates the training of Relational Graph Convolution Networks (RGCN) with TigerGraph ML Workbench. [DGL](https://www.dgl.ai/)'s implementation of RGCN is used here. We train the model on the IMDB dataset from [PyG datasets](https://pytorch-geometric.readthedocs.io/en/latest/modules/datasets.html#torch_geometric.datasets.IMDB) with TigerGraph as the data store. The dataset contains 3 types of vertices: 4278 movies, 5257 actors, and 2081 directors; and 4 types of edges: 12828 actor to movie edges, 12828 movie to actor edges, 4278 director to movie edges, and 4278 movie to director edges. Each vertex is described by a 0/1-valued word vector indicating the absence/presence of the corresponding keywords from the plot (for movie) or from movies they participated (for actors and directors). Each movie is classified into one of three classes, action, comedy, and drama according to their genre. The goal is to predict the class of each movie in the graph.\n", 10 | "\n", 11 | "The following libraries are required to run this notebook. Uncomment to install them if necessary. You need to restart the kernel after installing." 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": null, 17 | "metadata": {}, 18 | "outputs": [], 19 | "source": [ 20 | "#!pip install torch==1.12.0 --extra-index-url https://download.pytorch.org/whl/cpu\n", 21 | "#!pip install dgl -f https://data.dgl.ai/wheels/repo.html\n", 22 | "#!pip install psutil # Required for DGL\n", 23 | "#!pip install pyTigerGraph[gds]\n", 24 | "#!pip install tensorboard # If you use tensorboard for visualization later" 25 | ] 26 | }, 27 | { 28 | "cell_type": "markdown", 29 | "metadata": {}, 30 | "source": [ 31 | "## Table of Contents\n", 32 | "* [Data Processing](#data_processing) \n", 33 | "* [Train on whole graph](#train_whole) \n", 34 | "* [Train on neighborhood subgraphs](#train_subgraph) \n", 35 | "* [Inference](#inference)" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "## Data Processing " 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": {}, 48 | "source": [ 49 | "### Connect to TigerGraph\n", 50 | "\n", 51 | "The `TigerGraphConnection` class represents a connection to the TigerGraph database. Under the hood, it stores the necessary information to communicate with the database. It is able to perform quite a few database tasks. Please see its [documentation](https://docs.tigergraph.com/pytigergraph/current/intro/) for details.\n", 52 | "\n", 53 | "To connect your database, modify the `config.json` file accompanying this notebook. Set the value of `getToken` based on whether token auth is enabled for your database. Token auth is always enabled for tgcloud databases. " 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": null, 59 | "metadata": {}, 60 | "outputs": [], 61 | "source": [ 62 | "from pyTigerGraph import TigerGraphConnection\n", 63 | "import json\n", 64 | "\n", 65 | "# Read in DB configs\n", 66 | "with open('../../config.json', \"r\") as config_file:\n", 67 | " config = json.load(config_file)\n", 68 | " \n", 69 | "conn = TigerGraphConnection(\n", 70 | " host=config[\"host\"],\n", 71 | " username=config[\"username\"],\n", 72 | " password=config[\"password\"]\n", 73 | ")" 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "metadata": {}, 79 | "source": [ 80 | "### Ingest Data" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": null, 86 | "metadata": {}, 87 | "outputs": [], 88 | "source": [ 89 | "from pyTigerGraph.datasets import Datasets\n", 90 | "\n", 91 | "dataset = Datasets(\"imdb\")" 92 | ] 93 | }, 94 | { 95 | "cell_type": "code", 96 | "execution_count": null, 97 | "metadata": {}, 98 | "outputs": [], 99 | "source": [ 100 | "conn.ingestDataset(dataset, getToken=config[\"getToken\"])" 101 | ] 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "metadata": {}, 106 | "source": [ 107 | "### Visualize Schema" 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": null, 113 | "metadata": {}, 114 | "outputs": [], 115 | "source": [ 116 | "from pyTigerGraph.visualization import drawSchema\n", 117 | "\n", 118 | "drawSchema(conn.getSchema(force=True))" 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "metadata": {}, 124 | "source": [ 125 | "### Basic Statistics" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": null, 131 | "metadata": {}, 132 | "outputs": [], 133 | "source": [ 134 | "conn.getVertexCount('*')" 135 | ] 136 | }, 137 | { 138 | "cell_type": "code", 139 | "execution_count": null, 140 | "metadata": {}, 141 | "outputs": [], 142 | "source": [ 143 | "conn.getEdgeCount()" 144 | ] 145 | }, 146 | { 147 | "cell_type": "markdown", 148 | "metadata": {}, 149 | "source": [ 150 | "### Train/validation/test split" 151 | ] 152 | }, 153 | { 154 | "cell_type": "code", 155 | "execution_count": null, 156 | "metadata": {}, 157 | "outputs": [], 158 | "source": [ 159 | "# The code in this cell is commented out because there is no need to split the vertices into \n", 160 | "# training/validation/test sets, as the split is already done in the original dataset. \n", 161 | "# See notebook 1_data_processing for examples on the split function.\n", 162 | "\n", 163 | "#split = conn.gds.vertexSplitter(train_mask=0.8, val_mask=0.1, test_mask=0.1)\n", 164 | "#split.run()" 165 | ] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "execution_count": null, 170 | "metadata": {}, 171 | "outputs": [], 172 | "source": [ 173 | "print(\n", 174 | " \"Number of movies in training set:\",\n", 175 | " conn.getVertexCount(\"Movie\", where=\"train_mask!=0\"),\n", 176 | ")\n", 177 | "print(\n", 178 | " \"Number of movies in validation set:\",\n", 179 | " conn.getVertexCount(\"Movie\", where=\"val_mask!=0\"),\n", 180 | ")\n", 181 | "print(\n", 182 | " \"Number of movies in test set:\", \n", 183 | " conn.getVertexCount(\"Movie\", where=\"test_mask!=0\"),\n", 184 | ")" 185 | ] 186 | }, 187 | { 188 | "cell_type": "markdown", 189 | "metadata": {}, 190 | "source": [ 191 | "## Train on whole graph \n", 192 | "We first train the model on the whole graph. This will **NOT** work when the graph is large. See the section of training on subgraphs for real use. However, we still include this example for illustration purpose. Hyperparameters for the model and training environment are defined below." 193 | ] 194 | }, 195 | { 196 | "cell_type": "code", 197 | "execution_count": null, 198 | "metadata": {}, 199 | "outputs": [], 200 | "source": [ 201 | "# Hyperparameters\n", 202 | "hp = {\n", 203 | " \"hidden_dim\": 64,\n", 204 | " \"num_layers\": 2,\n", 205 | " \"dropout\": 0.1,\n", 206 | " \"lr\": 0.01,\n", 207 | " \"l2_penalty\": 0.0001,\n", 208 | "}" 209 | ] 210 | }, 211 | { 212 | "cell_type": "markdown", 213 | "metadata": {}, 214 | "source": [ 215 | "### Construct graph loader" 216 | ] 217 | }, 218 | { 219 | "cell_type": "markdown", 220 | "metadata": {}, 221 | "source": [ 222 | "The `GraphLoader` can get the whole graph from database all at once (`num_batches=1`). See the tutorial on dataloaders for details." 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "execution_count": null, 228 | "metadata": {}, 229 | "outputs": [], 230 | "source": [ 231 | "graph_loader = conn.gds.graphLoader(\n", 232 | " v_in_feats={\"Movie\": [\"x\"], \"Actor\": [\"x\"], \"Director\": [\"x\"]}, \n", 233 | " v_out_labels={\"Movie\": [\"y\"]},\n", 234 | " v_extra_feats={\"Movie\": [\"train_mask\", \"val_mask\", \"test_mask\"]},\n", 235 | " num_batches=1,\n", 236 | " output_format=\"DGL\",\n", 237 | " shuffle=False\n", 238 | ")" 239 | ] 240 | }, 241 | { 242 | "cell_type": "code", 243 | "execution_count": null, 244 | "metadata": {}, 245 | "outputs": [], 246 | "source": [ 247 | "# Get the whole graph from the loader\n", 248 | "data = graph_loader.data\n", 249 | "\n", 250 | "data" 251 | ] 252 | }, 253 | { 254 | "cell_type": "markdown", 255 | "metadata": {}, 256 | "source": [ 257 | "### Construct model and optimizer" 258 | ] 259 | }, 260 | { 261 | "cell_type": "markdown", 262 | "metadata": {}, 263 | "source": [ 264 | "We build a RGCN model with 2 convolutional layers. We use the Adam optimizer with a learning rate of 0.01 to train the model." 265 | ] 266 | }, 267 | { 268 | "cell_type": "code", 269 | "execution_count": null, 270 | "metadata": {}, 271 | "outputs": [], 272 | "source": [ 273 | "import dgl.function as fn\n", 274 | "import dgl.nn.pytorch as dglnn\n", 275 | "import torch\n", 276 | "import torch.nn as nn\n", 277 | "import torch.nn.functional as F" 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": null, 283 | "metadata": {}, 284 | "outputs": [], 285 | "source": [ 286 | "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", 287 | "\n", 288 | "class RGCN(nn.Module):\n", 289 | " def __init__(self, in_feats, hid_feats, out_feats, rel_names):\n", 290 | " super().__init__()\n", 291 | "\n", 292 | " self.conv1 = dglnn.HeteroGraphConv({\n", 293 | " rel: dglnn.GraphConv(in_feats, hid_feats)\n", 294 | " for rel in rel_names}, aggregate='sum')\n", 295 | " self.conv2 = dglnn.HeteroGraphConv({\n", 296 | " rel: dglnn.GraphConv(hid_feats, out_feats)\n", 297 | " for rel in rel_names}, aggregate='sum')\n", 298 | "\n", 299 | " def forward(self, graph, inputs):\n", 300 | " # inputs are features of nodes\n", 301 | " h = self.conv1(graph, inputs)\n", 302 | " h = {k: F.relu(v) for k, v in h.items()}\n", 303 | " h = self.conv2(graph, h)\n", 304 | " return h\n", 305 | "\n", 306 | "model = RGCN(\n", 307 | " in_feats=3066, \n", 308 | " hid_feats=hp[\"hidden_dim\"],\n", 309 | " out_feats=3, \n", 310 | " rel_names=data.etypes).to(device)\n", 311 | "\n", 312 | "optimizer = torch.optim.Adam(\n", 313 | " model.parameters(), lr=hp[\"lr\"], weight_decay=hp[\"l2_penalty\"]\n", 314 | ")" 315 | ] 316 | }, 317 | { 318 | "cell_type": "markdown", 319 | "metadata": {}, 320 | "source": [ 321 | "### Train the model" 322 | ] 323 | }, 324 | { 325 | "cell_type": "code", 326 | "execution_count": null, 327 | "metadata": {}, 328 | "outputs": [], 329 | "source": [ 330 | "from datetime import datetime\n", 331 | "from pyTigerGraph.gds.metrics import Accumulator, Accuracy\n", 332 | "from torch.utils.tensorboard import SummaryWriter" 333 | ] 334 | }, 335 | { 336 | "cell_type": "code", 337 | "execution_count": null, 338 | "metadata": {}, 339 | "outputs": [], 340 | "source": [ 341 | "log_dir = \"logs/imdb/rgcn/wholegraph/\" + datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n", 342 | "tb_log = SummaryWriter(log_dir)\n", 343 | "logs = {}\n", 344 | "data = data.to(device)\n", 345 | "for epoch in range(20):\n", 346 | " # Train\n", 347 | " model.train()\n", 348 | " acc = Accuracy()\n", 349 | " # Forward pass\n", 350 | " out = model(data, {i: data.nodes[i].data[\"x\"] for i in [\"Actor\", \"Movie\", \"Director\"]})\n", 351 | " # Calculate loss on movie vertices in the training set only\n", 352 | " movies = data.nodes['Movie'].data\n", 353 | " mask = movies[\"train_mask\"]\n", 354 | " loss = F.cross_entropy(out[\"Movie\"][mask], movies[\"y\"][mask])\n", 355 | " # Backward pass\n", 356 | " optimizer.zero_grad()\n", 357 | " loss.backward()\n", 358 | " # Update model\n", 359 | " optimizer.step()\n", 360 | " # Evaluate\n", 361 | " val_acc = Accuracy()\n", 362 | " with torch.no_grad():\n", 363 | " pred = out['Movie'].argmax(dim=1)\n", 364 | " acc.update(pred[mask], movies[\"y\"][mask])\n", 365 | " mask = movies[\"val_mask\"]\n", 366 | " valid_loss = F.cross_entropy(out['Movie'][mask], movies[\"y\"][mask])\n", 367 | " val_acc.update(pred[mask], movies[\"y\"][mask])\n", 368 | " # Logging\n", 369 | " logs[\"loss\"] = loss.item()\n", 370 | " logs[\"val_loss\"] = valid_loss.item()\n", 371 | " logs[\"acc\"] = acc.value\n", 372 | " logs[\"val_acc\"] = val_acc.value\n", 373 | " print(\n", 374 | " \"Epoch: {:02d}, Train Loss: {:.4f}, Valid Loss: {:.4f}, Train Accuracy: {:.4f}, Valid Accuracy: {:.4f}\".format(\n", 375 | " epoch, logs[\"loss\"], logs[\"val_loss\"], logs[\"acc\"], logs[\"val_acc\"]\n", 376 | " )\n", 377 | " )\n", 378 | " tb_log.add_scalars(\n", 379 | " \"Loss\", {\"Train\": logs[\"loss\"], \"Validation\": logs[\"val_loss\"]}, epoch\n", 380 | " )\n", 381 | " tb_log.add_scalars(\n", 382 | " \"Accuracy\", {\"Train\": logs[\"acc\"], \"Validation\": logs[\"val_acc\"]}, epoch\n", 383 | " )\n", 384 | " tb_log.flush()" 385 | ] 386 | }, 387 | { 388 | "cell_type": "markdown", 389 | "metadata": {}, 390 | "source": [ 391 | "### Test the model" 392 | ] 393 | }, 394 | { 395 | "cell_type": "code", 396 | "execution_count": null, 397 | "metadata": {}, 398 | "outputs": [], 399 | "source": [ 400 | "model.eval()\n", 401 | "acc = Accuracy()\n", 402 | "with torch.no_grad():\n", 403 | " pred = model(\n", 404 | " data, \n", 405 | " {i: data.nodes[i].data[\"x\"] for i in [\"Actor\", \"Movie\", \"Director\"]}\n", 406 | " )[\"Movie\"].argmax(dim=1)\n", 407 | " mask = movies[\"test_mask\"]\n", 408 | " acc.update(pred[mask], movies[\"y\"][mask])\n", 409 | "print(\"Accuracy: {:.4f}\".format(acc.value))" 410 | ] 411 | }, 412 | { 413 | "cell_type": "markdown", 414 | "metadata": {}, 415 | "source": [ 416 | "## Train on Neighborhood Subgraphs \n", 417 | "Alternatively, we train the model on the neighborhood subgraphs. Each subgraph contains the 2 hop neighborhood of certain seed vertices. This method will allow us to train the model on graphs that are way larger than the IMDB dataset because we don't load the whole graph into memory all at once. \n", 418 | "\n", 419 | "We will use the same parameters as before, but we will use the NeighborLoader to load subgraphs. Once we finish iterating over all the subgraphs generated by the loader, it is guaranteed to cover all vertices in the graph (except for those filtered by a user provided mask). " 420 | ] 421 | }, 422 | { 423 | "cell_type": "code", 424 | "execution_count": null, 425 | "metadata": {}, 426 | "outputs": [], 427 | "source": [ 428 | "# Hyperparameters\n", 429 | "hp = {\n", 430 | " \"hidden_dim\": 64,\n", 431 | " \"num_layers\": 2,\n", 432 | " \"dropout\": 0.2,\n", 433 | " \"lr\": 0.01,\n", 434 | " \"l2_penalty\": 0.0001,\n", 435 | " \"batch_size\": 128, \n", 436 | " \"num_neighbors\": 10, \n", 437 | " \"num_hops\": 2\n", 438 | "}" 439 | ] 440 | }, 441 | { 442 | "cell_type": "markdown", 443 | "metadata": {}, 444 | "source": [ 445 | "### Construct neighborhood subgraph loader" 446 | ] 447 | }, 448 | { 449 | "cell_type": "markdown", 450 | "metadata": {}, 451 | "source": [ 452 | "Here we construct 3 subgraph loaders. The `train_loader` only uses vertices in the training set as seeds, the `valid_loader` only uses vertices in the validation set, and the `test_loader` only uses vertices in the test set." 453 | ] 454 | }, 455 | { 456 | "cell_type": "code", 457 | "execution_count": null, 458 | "metadata": {}, 459 | "outputs": [], 460 | "source": [ 461 | "train_loader = conn.gds.neighborLoader(\n", 462 | " v_in_feats={\"Movie\": [\"x\"], \"Actor\": [\"x\"], \"Director\": [\"x\"]}, \n", 463 | " v_out_labels={\"Movie\": [\"y\"]},\n", 464 | " v_extra_feats={\"Movie\": [\"train_mask\", \"val_mask\", \"test_mask\"]},\n", 465 | " output_format=\"DGL\",\n", 466 | " batch_size=hp[\"batch_size\"],\n", 467 | " num_neighbors=hp[\"num_neighbors\"],\n", 468 | " num_hops=hp[\"num_hops\"],\n", 469 | " shuffle=True,\n", 470 | " filter_by={\"Movie\":\"train_mask\"},\n", 471 | ")" 472 | ] 473 | }, 474 | { 475 | "cell_type": "code", 476 | "execution_count": null, 477 | "metadata": {}, 478 | "outputs": [], 479 | "source": [ 480 | "valid_loader = conn.gds.neighborLoader(\n", 481 | " v_in_feats={\"Movie\": [\"x\"], \"Actor\": [\"x\"], \"Director\": [\"x\"]}, \n", 482 | " v_out_labels={\"Movie\": [\"y\"]},\n", 483 | " v_extra_feats={\"Movie\": [\"train_mask\", \"val_mask\", \"test_mask\"]},\n", 484 | " output_format=\"DGL\",\n", 485 | " batch_size=hp[\"batch_size\"],\n", 486 | " num_neighbors=hp[\"num_neighbors\"],\n", 487 | " num_hops=hp[\"num_hops\"],\n", 488 | " shuffle=False,\n", 489 | " filter_by={\"Movie\":\"val_mask\"},\n", 490 | ")" 491 | ] 492 | }, 493 | { 494 | "cell_type": "markdown", 495 | "metadata": {}, 496 | "source": [ 497 | "### Construct model and optimizer" 498 | ] 499 | }, 500 | { 501 | "cell_type": "markdown", 502 | "metadata": {}, 503 | "source": [ 504 | "We build a RGCN model with 2 convolutional layers. We use the Adam optimizer with a learning rate of 0.01 to train the model." 505 | ] 506 | }, 507 | { 508 | "cell_type": "code", 509 | "execution_count": null, 510 | "metadata": {}, 511 | "outputs": [], 512 | "source": [ 513 | "model = RGCN(\n", 514 | " in_feats=3066, \n", 515 | " hid_feats=hp[\"hidden_dim\"],\n", 516 | " out_feats=3, \n", 517 | " rel_names=data.etypes).to(device)\n", 518 | "\n", 519 | "optimizer = torch.optim.Adam(\n", 520 | " model.parameters(), lr=hp[\"lr\"], weight_decay=hp[\"l2_penalty\"]\n", 521 | ")" 522 | ] 523 | }, 524 | { 525 | "cell_type": "markdown", 526 | "metadata": {}, 527 | "source": [ 528 | "### Train the model" 529 | ] 530 | }, 531 | { 532 | "cell_type": "code", 533 | "execution_count": null, 534 | "metadata": {}, 535 | "outputs": [], 536 | "source": [ 537 | "from datetime import datetime\n", 538 | "\n", 539 | "from pyTigerGraph.gds.metrics import Accumulator, Accuracy\n", 540 | "from torch.utils.tensorboard import SummaryWriter" 541 | ] 542 | }, 543 | { 544 | "cell_type": "code", 545 | "execution_count": null, 546 | "metadata": {}, 547 | "outputs": [], 548 | "source": [ 549 | "log_dir = \"logs/imdb/rgcn/subgraph/\" + datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n", 550 | "train_log = SummaryWriter(log_dir+\"/train\")\n", 551 | "valid_log = SummaryWriter(log_dir+\"/valid\")\n", 552 | "global_steps = 0\n", 553 | "logs = {}\n", 554 | "for epoch in range(10):\n", 555 | " # Train\n", 556 | " model.train()\n", 557 | " epoch_train_loss = Accumulator()\n", 558 | " epoch_train_acc = Accuracy()\n", 559 | " # Iterate through the loader to get a stream of subgraphs instead of the whole graph\n", 560 | " for bid, batch in enumerate(train_loader):\n", 561 | " batch.to(device)\n", 562 | " # Forward pass\n", 563 | " out = model(batch, {i: batch.nodes[i].data[\"x\"] for i in [\"Actor\", \"Movie\", \"Director\"]})\n", 564 | " # Calculate loss\n", 565 | " movies = batch.nodes['Movie'].data\n", 566 | " mask = movies[\"is_seed\"]\n", 567 | " loss = F.cross_entropy(out[\"Movie\"][mask], movies[\"y\"][mask])\n", 568 | " # Backward pass\n", 569 | " optimizer.zero_grad()\n", 570 | " loss.backward()\n", 571 | " optimizer.step()\n", 572 | " batchsize = mask.sum().item()\n", 573 | " epoch_train_loss.update(loss.item() * batchsize, batchsize)\n", 574 | " # Predict on training data\n", 575 | " with torch.no_grad():\n", 576 | " pred = out[\"Movie\"].argmax(dim=1)\n", 577 | " epoch_train_acc.update(pred[mask], movies[\"y\"][mask])\n", 578 | " # Log training status after each batch\n", 579 | " logs[\"loss\"] = epoch_train_loss.mean\n", 580 | " logs[\"acc\"] = epoch_train_acc.value\n", 581 | " print(\n", 582 | " \"Epoch {}, Train Batch {}, Loss {:.4f}, Accuracy {:.4f}\".format(\n", 583 | " epoch, bid, logs[\"loss\"], logs[\"acc\"]\n", 584 | " )\n", 585 | " )\n", 586 | " train_log.add_scalar(\"Loss\", logs[\"loss\"], global_steps)\n", 587 | " train_log.add_scalar(\"Accuracy\", logs[\"acc\"], global_steps)\n", 588 | " train_log.flush()\n", 589 | " global_steps += 1\n", 590 | " # Evaluate\n", 591 | " model.eval()\n", 592 | " epoch_val_loss = Accumulator()\n", 593 | " epoch_val_acc = Accuracy()\n", 594 | " for batch in valid_loader:\n", 595 | " batch.to(device)\n", 596 | " with torch.no_grad():\n", 597 | " # Forward pass\n", 598 | " out = model(batch, {i: batch.nodes[i].data[\"x\"] for i in [\"Actor\", \"Movie\", \"Director\"]})\n", 599 | " # Calculate loss\n", 600 | " movies = batch.nodes['Movie'].data\n", 601 | " mask = movies[\"is_seed\"]\n", 602 | " valid_loss = F.cross_entropy(out[\"Movie\"][mask], movies[\"y\"][mask])\n", 603 | " batchsize = mask.sum().item()\n", 604 | " epoch_val_loss.update(valid_loss.item() * batchsize, batchsize)\n", 605 | " # Prediction\n", 606 | " pred = out[\"Movie\"].argmax(dim=1)\n", 607 | " epoch_val_acc.update(pred[mask], movies[\"y\"][mask])\n", 608 | " # Log testing result after each epoch\n", 609 | " logs[\"val_loss\"] = epoch_val_loss.mean\n", 610 | " logs[\"val_acc\"] = epoch_val_acc.value\n", 611 | " print(\n", 612 | " \"Epoch {}, Valid Loss {:.4f}, Valid Accuracy {:.4f}\".format(\n", 613 | " epoch, logs[\"val_loss\"], logs[\"val_acc\"]\n", 614 | " )\n", 615 | " )\n", 616 | " valid_log.add_scalar(\"Loss\", logs[\"val_loss\"], global_steps)\n", 617 | " valid_log.add_scalar(\"Accuracy\", logs[\"val_acc\"], global_steps)\n", 618 | " valid_log.flush()" 619 | ] 620 | }, 621 | { 622 | "cell_type": "markdown", 623 | "metadata": {}, 624 | "source": [ 625 | "### Test the model" 626 | ] 627 | }, 628 | { 629 | "cell_type": "code", 630 | "execution_count": null, 631 | "metadata": {}, 632 | "outputs": [], 633 | "source": [ 634 | "test_loader = conn.gds.neighborLoader(\n", 635 | " v_in_feats={\"Movie\": [\"x\"], \"Actor\": [\"x\"], \"Director\": [\"x\"]}, \n", 636 | " v_out_labels={\"Movie\": [\"y\"]},\n", 637 | " v_extra_feats={\"Movie\": [\"train_mask\", \"val_mask\", \"test_mask\"]},\n", 638 | " output_format=\"DGL\",\n", 639 | " batch_size=hp[\"batch_size\"],\n", 640 | " num_neighbors=hp[\"num_neighbors\"],\n", 641 | " num_hops=hp[\"num_hops\"],\n", 642 | " shuffle=False,\n", 643 | " filter_by={\"Movie\":\"test_mask\"},\n", 644 | ")" 645 | ] 646 | }, 647 | { 648 | "cell_type": "code", 649 | "execution_count": null, 650 | "metadata": {}, 651 | "outputs": [], 652 | "source": [ 653 | "model.eval()\n", 654 | "acc = Accuracy()\n", 655 | "for batch in test_loader:\n", 656 | " batch.to(device)\n", 657 | " with torch.no_grad():\n", 658 | " pred = model(\n", 659 | " batch, \n", 660 | " {i: batch.nodes[i].data[\"x\"] for i in [\"Actor\", \"Movie\", \"Director\"]}\n", 661 | " )[\"Movie\"].argmax(dim=1)\n", 662 | " movies = batch.nodes['Movie'].data\n", 663 | " mask = movies[\"is_seed\"]\n", 664 | " acc.update(pred[mask], movies[\"y\"][mask])\n", 665 | "print(\"Accuracy: {:.4f}\".format(acc.value))" 666 | ] 667 | }, 668 | { 669 | "cell_type": "markdown", 670 | "metadata": {}, 671 | "source": [ 672 | "## Inference \n", 673 | "\n", 674 | "Finally, we use the trained model for node classification. At this stage, we typically do inference/prediction for specific nodes instead of random batches, so we will create a new data loader. " 675 | ] 676 | }, 677 | { 678 | "cell_type": "code", 679 | "execution_count": null, 680 | "metadata": {}, 681 | "outputs": [], 682 | "source": [ 683 | "infer_loader = conn.gds.neighborLoader(\n", 684 | " v_in_feats={\"Movie\": [\"x\"], \"Actor\": [\"x\"], \"Director\": [\"x\"]}, \n", 685 | " v_out_labels={\"Movie\": [\"y\"]},\n", 686 | " v_extra_feats={\"Movie\": [\"train_mask\", \"val_mask\", \"test_mask\"]},\n", 687 | " output_format=\"DGL\",\n", 688 | " num_neighbors=hp[\"num_neighbors\"],\n", 689 | " num_hops=hp[\"num_hops\"],\n", 690 | " shuffle=False\n", 691 | ")" 692 | ] 693 | }, 694 | { 695 | "cell_type": "code", 696 | "execution_count": null, 697 | "metadata": {}, 698 | "outputs": [], 699 | "source": [ 700 | "# Fetch specific nodes by their IDs and do prediction. \n", 701 | "# Each node is represented by a dict with two mandatory keys: primary_id and type.\n", 702 | "input_nodes = [{\"primary_id\": 7, \"type\": \"Movie\"}, \n", 703 | " {\"primary_id\": 55, \"type\": \"Movie\"}]\n", 704 | "data = infer_loader.fetch(input_nodes)" 705 | ] 706 | }, 707 | { 708 | "cell_type": "code", 709 | "execution_count": null, 710 | "metadata": {}, 711 | "outputs": [], 712 | "source": [ 713 | "# The returned data are the neighborhood subgraphs of the input nodes.\n", 714 | "# The original IDs of the nodes in the subgraphs are stored in the \n", 715 | "# `primary_id` attribute.\n", 716 | "data" 717 | ] 718 | }, 719 | { 720 | "cell_type": "code", 721 | "execution_count": null, 722 | "metadata": {}, 723 | "outputs": [], 724 | "source": [ 725 | "# Predict. Predictions for both the input nodes and others in their \n", 726 | "# neighborhoods are generated.\n", 727 | "model.eval()\n", 728 | "pred = model(\n", 729 | " data, \n", 730 | " {i: data.nodes[i].data[\"x\"] for i in [\"Actor\", \"Movie\", \"Director\"]}\n", 731 | ")[\"Movie\"].argmax(dim=1)\n", 732 | "print(\"ID: Label\")\n", 733 | "for i,j in zip(data.extra_data[\"Movie\"][\"primary_id\"], pred):\n", 734 | " print(\"{}:{}\".format(i, j.item()))" 735 | ] 736 | }, 737 | { 738 | "cell_type": "code", 739 | "execution_count": null, 740 | "metadata": {}, 741 | "outputs": [], 742 | "source": [] 743 | } 744 | ], 745 | "metadata": { 746 | "environment": { 747 | "name": "pytorch-gpu.1-9.m81", 748 | "type": "gcloud", 749 | "uri": "gcr.io/deeplearning-platform-release/pytorch-gpu.1-9:m81" 750 | }, 751 | "kernelspec": { 752 | "display_name": "Python 3.8.9 64-bit", 753 | "language": "python", 754 | "name": "python3" 755 | }, 756 | "language_info": { 757 | "codemirror_mode": { 758 | "name": "ipython", 759 | "version": 3 760 | }, 761 | "file_extension": ".py", 762 | "mimetype": "text/x-python", 763 | "name": "python", 764 | "nbconvert_exporter": "python", 765 | "pygments_lexer": "ipython3", 766 | "version": "3.8.9" 767 | }, 768 | "vscode": { 769 | "interpreter": { 770 | "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6" 771 | } 772 | } 773 | }, 774 | "nbformat": 4, 775 | "nbformat_minor": 4 776 | } 777 | -------------------------------------------------------------------------------- /GNNs/PyG/gcn_link_prediction.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "e3482340-811d-429a-ba1f-80baca631c7e", 6 | "metadata": {}, 7 | "source": [ 8 | "# Graph Convolutional Network for Link Prediction\n", 9 | "This notebook demonstrates the training of [Graph Convolutional Networks (GCN)](https://arxiv.org/pdf/1609.02907.pdf) for Link Prediction with TigerGraph. Pytorch Geometric's implementation of GCN is used here. We train the model on the Cora dataset from [PyG datasets](https://pytorch-geometric.readthedocs.io/en/latest/modules/datasets.html#torch_geometric.datasets.Planetoid) with TigerGraph as the data store. The dataset contains 2708 machine learning papers and 10556 citation links between the papers. Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from a dictionary. The dictionary consists of 1433 unique words. Each paper is classified into one of seven classes based on the topic. The goal is to predict whether two papers are linked or not.\n", 10 | "\n", 11 | "The following libraries are required to run this notebook. Uncomment to install them if necessary. You might need to restart the kernel after installing." 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": null, 17 | "id": "f6af90ac", 18 | "metadata": {}, 19 | "outputs": [], 20 | "source": [ 21 | "#!pip install torch==1.12.0 --extra-index-url https://download.pytorch.org/whl/cpu\n", 22 | "#!pip install torch-scatter==2.0.9 torch-sparse==0.6.14 torch-cluster==1.6.0 torch-spline-conv==1.2.1 torch-geometric==2.0.4 -f https://data.pyg.org/whl/torch-1.12.0+cpu.html\n", 23 | "#!pip install pyTigerGraph[gds]\n", 24 | "#!pip install tensorboard # If you use tensorboard for visualization later" 25 | ] 26 | }, 27 | { 28 | "cell_type": "markdown", 29 | "id": "de5da30d", 30 | "metadata": {}, 31 | "source": [ 32 | "## Table of Contents\n", 33 | "* [Data Processing](#data_processing) \n", 34 | "* [Whole Graph Training](#train_whole) \n", 35 | "* [Stochastic Batch Training](#train_subgraph) " 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "id": "d786d1fe-3758-4810-8c34-e66e59687a58", 41 | "metadata": {}, 42 | "source": [ 43 | "## Data Processing \n", 44 | "\n", 45 | "For each edge, the original dataset include `is_train` and `is_val` attributes. You may add `is_test` if you want the train/validation/test splits. Otherwise, you can just use the edgeSplitter to get train/validation sets." 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "id": "81383884-2fb9-46f6-9ce6-ad227abf4e52", 51 | "metadata": {}, 52 | "source": [ 53 | "### Connect to TigerGraph\n", 54 | "\n", 55 | "The `TigerGraphConnection` class represents a connection to the TigerGraph database. Under the hood, it stores the necessary information to communicate with the database. It is able to perform quite a few database tasks. Please see its [documentation](https://docs.tigergraph.com/pytigergraph/current/intro/) for details.\n", 56 | "\n", 57 | "To connect your database, modify the `config.json` file accompanying this notebook. Set the value of `getToken` based on whether token auth is enabled for your database. Token auth is always enabled for tgcloud databases. " 58 | ] 59 | }, 60 | { 61 | "cell_type": "code", 62 | "execution_count": 1, 63 | "id": "ea45afd1-4fc3-4bc6-b739-2e89450df296", 64 | "metadata": {}, 65 | "outputs": [], 66 | "source": [ 67 | "from pyTigerGraph import TigerGraphConnection\n", 68 | "import json\n", 69 | "\n", 70 | "# Read in DB configs\n", 71 | "with open('../../config.json', \"r\") as config_file:\n", 72 | " config = json.load(config_file)\n", 73 | " \n", 74 | "conn = TigerGraphConnection(\n", 75 | " host=config[\"host\"],\n", 76 | " username=config[\"username\"],\n", 77 | " password=config[\"password\"],\n", 78 | ")" 79 | ] 80 | }, 81 | { 82 | "cell_type": "markdown", 83 | "id": "77f59a78-4ea1-484d-98d3-caf908341395", 84 | "metadata": {}, 85 | "source": [ 86 | "### Ingest Data" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": 2, 92 | "id": "a7d25f25-0c8a-47ad-a9bb-efaf82699ffb", 93 | "metadata": {}, 94 | "outputs": [ 95 | { 96 | "data": { 97 | "application/vnd.jupyter.widget-view+json": { 98 | "model_id": "d9d7f736003a49f0937019846f40d70d", 99 | "version_major": 2, 100 | "version_minor": 0 101 | }, 102 | "text/plain": [ 103 | "Downloading: 0%| | 0/166537 [00:00\n", 281 | "\n", 282 | "Here, we use the full graph for link prediction. This will **NOT** work when the graph is very large. See the section of Stochastic Mini-Batch Training for real use. However, we still include this example for illustration purposes.\n", 283 | "\n", 284 | "We load the whole graph from TigerGraph which includes the feature and split results." 285 | ] 286 | }, 287 | { 288 | "cell_type": "markdown", 289 | "id": "182afd6a-a68d-4b28-8ac6-86e6a6b51e8f", 290 | "metadata": {}, 291 | "source": [ 292 | "### Construct graph loader and negative edges" 293 | ] 294 | }, 295 | { 296 | "cell_type": "code", 297 | "execution_count": 7, 298 | "id": "2544116b-4169-48ac-bd0f-049dfbcfc00d", 299 | "metadata": {}, 300 | "outputs": [ 301 | { 302 | "name": "stdout", 303 | "output_type": "stream", 304 | "text": [ 305 | "Installing and optimizing queries. It might take a minute if this is the first time you use this loader.\n", 306 | "Query installation finished.\n" 307 | ] 308 | } 309 | ], 310 | "source": [ 311 | "graph_loader = conn.gds.graphLoader(\n", 312 | " num_batches=1,\n", 313 | " v_in_feats = [\"x\"],\n", 314 | " e_extra_feats=[\"is_train\",\"is_val\"],\n", 315 | " output_format = \"PyG\")" 316 | ] 317 | }, 318 | { 319 | "cell_type": "code", 320 | "execution_count": 8, 321 | "id": "874be99d-9c9b-485c-ad75-7581132780a3", 322 | "metadata": {}, 323 | "outputs": [], 324 | "source": [ 325 | "data = graph_loader.data" 326 | ] 327 | }, 328 | { 329 | "cell_type": "code", 330 | "execution_count": 9, 331 | "id": "29a1443b-e53d-4c0d-994d-b3e2a50e8a04", 332 | "metadata": {}, 333 | "outputs": [ 334 | { 335 | "data": { 336 | "text/plain": [ 337 | "Data(edge_index=[2, 10556], is_train=[10556], is_val=[10556], x=[2708, 1433])" 338 | ] 339 | }, 340 | "execution_count": 9, 341 | "metadata": {}, 342 | "output_type": "execute_result" 343 | } 344 | ], 345 | "source": [ 346 | "data" 347 | ] 348 | }, 349 | { 350 | "cell_type": "code", 351 | "execution_count": 10, 352 | "id": "bca4fdb7-94d1-4d46-a041-ad280efeb6f3", 353 | "metadata": {}, 354 | "outputs": [], 355 | "source": [ 356 | "train_edge_index = data.edge_index[:, data.is_train]\n", 357 | "val_edge_index = data.edge_index[:, data.is_val]" 358 | ] 359 | }, 360 | { 361 | "cell_type": "code", 362 | "execution_count": 11, 363 | "id": "10410e24-e984-4c4d-9b54-61f66078f3c0", 364 | "metadata": {}, 365 | "outputs": [], 366 | "source": [ 367 | "import torch\n", 368 | "\n", 369 | "neg_val_edge = torch.randint(0, data.x.shape[0], val_edge_index.size(), dtype=torch.long)" 370 | ] 371 | }, 372 | { 373 | "cell_type": "code", 374 | "execution_count": 12, 375 | "id": "abdd6e53-13e0-4299-8fee-b5ee9a90b2d3", 376 | "metadata": {}, 377 | "outputs": [ 378 | { 379 | "data": { 380 | "text/plain": [ 381 | "(torch.Size([2, 8454]), torch.Size([2, 2102]), torch.Size([2, 2102]))" 382 | ] 383 | }, 384 | "execution_count": 12, 385 | "metadata": {}, 386 | "output_type": "execute_result" 387 | } 388 | ], 389 | "source": [ 390 | "train_edge_index.shape, val_edge_index.shape, neg_val_edge.shape" 391 | ] 392 | }, 393 | { 394 | "cell_type": "markdown", 395 | "id": "ef04bf96-ffc1-45ad-91f8-b490258668a3", 396 | "metadata": {}, 397 | "source": [ 398 | "### Construct GCN Model\n", 399 | "\n", 400 | "We use dot product to measure the similarity of two nodes in a decode function." 401 | ] 402 | }, 403 | { 404 | "cell_type": "code", 405 | "execution_count": 13, 406 | "id": "d4bc349b-4830-4459-bf95-206cbe89d555", 407 | "metadata": {}, 408 | "outputs": [], 409 | "source": [ 410 | "import torch\n", 411 | "import torch.nn.functional as F\n", 412 | "from torch_geometric.nn import GCNConv\n", 413 | "\n", 414 | "\n", 415 | "class GCN(torch.nn.Module):\n", 416 | " def __init__(self, in_channels, hidden_channels, out_channels, num_layers, dropout, **kwargs):\n", 417 | " super(GCN, self).__init__()\n", 418 | " self.convs = torch.nn.ModuleList()\n", 419 | " self.convs.append(GCNConv(in_channels, hidden_channels))\n", 420 | " for _ in range(num_layers - 2):\n", 421 | " self.convs.append(GCNConv(hidden_channels, hidden_channels))\n", 422 | " self.convs.append(GCNConv(hidden_channels, out_channels))\n", 423 | " self.dropout = dropout\n", 424 | "\n", 425 | " def reset_parameters(self):\n", 426 | " for conv in self.convs:\n", 427 | " conv.reset_parameters()\n", 428 | "\n", 429 | " def forward(self, x, adj_t):\n", 430 | " for i, conv in enumerate(self.convs[:-1]):\n", 431 | " x = conv(x, adj_t)\n", 432 | " x = F.relu(x)\n", 433 | " x = F.dropout(x, p=self.dropout, training=self.training)\n", 434 | " x = self.convs[-1](x, adj_t)\n", 435 | " return x\n", 436 | "\n", 437 | " def decode(self, z, pos_edge_index, neg_edge_index):\n", 438 | " edge_index = torch.cat([pos_edge_index, neg_edge_index], dim=-1) # concatenate pos and neg edges\n", 439 | " logits = (z[edge_index[0]] * z[edge_index[1]]).sum(dim=-1) # dot product \n", 440 | " return logits\n" 441 | ] 442 | }, 443 | { 444 | "cell_type": "markdown", 445 | "id": "65a6bfba-3157-4330-ae6c-49e25a18491e", 446 | "metadata": {}, 447 | "source": [ 448 | "### Get binary labels for positive and negative edges" 449 | ] 450 | }, 451 | { 452 | "cell_type": "code", 453 | "execution_count": 14, 454 | "id": "8d5123bd-2ebc-43a6-a9b3-ac0d503c0f74", 455 | "metadata": {}, 456 | "outputs": [], 457 | "source": [ 458 | "def get_link_labels(pos_edge_index, neg_edge_index):\n", 459 | " E = pos_edge_index.size(1) + neg_edge_index.size(1)\n", 460 | " link_labels = torch.zeros(E, dtype=torch.float)\n", 461 | " link_labels[:pos_edge_index.size(1)] = 1.\n", 462 | " return link_labels" 463 | ] 464 | }, 465 | { 466 | "cell_type": "markdown", 467 | "id": "939e07f7-e6ae-4bea-ad82-3f549ddeba42", 468 | "metadata": {}, 469 | "source": [ 470 | "### Define Hyperparameters" 471 | ] 472 | }, 473 | { 474 | "cell_type": "code", 475 | "execution_count": 15, 476 | "id": "a631ec03-1a9a-41a0-9734-ab7ff3dea269", 477 | "metadata": {}, 478 | "outputs": [], 479 | "source": [ 480 | "# Hyperparameters\n", 481 | "hp = {\"hidden_dim\": 128, \"out_dim\": 64, \"num_layers\": 2,\n", 482 | " \"dropout\": 0.6, \"lr\": 0.01, \"l2_penalty\": 5e-4}" 483 | ] 484 | }, 485 | { 486 | "cell_type": "markdown", 487 | "id": "6db9abec-dc1c-401c-a3f0-7693c768ae3f", 488 | "metadata": {}, 489 | "source": [ 490 | "### Instantiate Model and optimizer" 491 | ] 492 | }, 493 | { 494 | "cell_type": "code", 495 | "execution_count": 16, 496 | "id": "bcbc8fca-6eff-4238-928f-f51eb936496b", 497 | "metadata": {}, 498 | "outputs": [], 499 | "source": [ 500 | "model = GCN(1433, hp[\"hidden_dim\"], hp[\"out_dim\"], hp[\"num_layers\"], hp[\"dropout\"])\n", 501 | "optimizer = torch.optim.Adam(\n", 502 | " model.parameters(), lr=hp[\"lr\"], weight_decay=hp[\"l2_penalty\"]\n", 503 | ")" 504 | ] 505 | }, 506 | { 507 | "cell_type": "code", 508 | "execution_count": 17, 509 | "id": "f0e9a355-e46b-4626-8a35-c5360dec07bf", 510 | "metadata": {}, 511 | "outputs": [ 512 | { 513 | "data": { 514 | "text/plain": [ 515 | "tensor([1., 1., 1., ..., 0., 0., 0.])" 516 | ] 517 | }, 518 | "execution_count": 17, 519 | "metadata": {}, 520 | "output_type": "execute_result" 521 | } 522 | ], 523 | "source": [ 524 | "val_labels = get_link_labels(val_edge_index, neg_val_edge)\n", 525 | "val_labels" 526 | ] 527 | }, 528 | { 529 | "cell_type": "markdown", 530 | "id": "6a700bfa-253a-4868-877b-194598b30c07", 531 | "metadata": {}, 532 | "source": [ 533 | "### Train the model" 534 | ] 535 | }, 536 | { 537 | "cell_type": "code", 538 | "execution_count": 18, 539 | "id": "5cde228b-fd55-4116-88ad-dbcb787be5c0", 540 | "metadata": {}, 541 | "outputs": [], 542 | "source": [ 543 | "from sklearn.metrics import roc_auc_score" 544 | ] 545 | }, 546 | { 547 | "cell_type": "code", 548 | "execution_count": 19, 549 | "id": "4a894353-20f1-42d0-9d7e-6297a3cb4e4d", 550 | "metadata": {}, 551 | "outputs": [ 552 | { 553 | "name": "stdout", 554 | "output_type": "stream", 555 | "text": [ 556 | "Epoch: 0, training loss: 0.6500421166419983, valid roc_auc_score: 0.8383314427562532\n", 557 | "Epoch: 1, training loss: 1.081446886062622, valid roc_auc_score: 0.812061323500522\n", 558 | "Epoch: 2, training loss: 1.1687164306640625, valid roc_auc_score: 0.7562404886470317\n", 559 | "Epoch: 3, training loss: 0.6771160364151001, valid roc_auc_score: 0.8145703742799436\n", 560 | "Epoch: 4, training loss: 0.6512936949729919, valid roc_auc_score: 0.8144599271592186\n", 561 | "Epoch: 5, training loss: 0.6514720320701599, valid roc_auc_score: 0.8029452490084655\n", 562 | "Epoch: 6, training loss: 0.6470925211906433, valid roc_auc_score: 0.7915663664979481\n", 563 | "Epoch: 7, training loss: 0.6444706916809082, valid roc_auc_score: 0.7924901163406516\n", 564 | "Epoch: 8, training loss: 0.6450539827346802, valid roc_auc_score: 0.8071957657108768\n", 565 | "Epoch: 9, training loss: 0.6447546482086182, valid roc_auc_score: 0.7999462022938599\n", 566 | "Epoch: 10, training loss: 0.6415925025939941, valid roc_auc_score: 0.7999011634065152\n", 567 | "Epoch: 11, training loss: 0.6420110464096069, valid roc_auc_score: 0.7940671563759222\n", 568 | "Epoch: 12, training loss: 0.6354835033416748, valid roc_auc_score: 0.8104411683494763\n", 569 | "Epoch: 13, training loss: 0.6298680305480957, valid roc_auc_score: 0.8130789986610549\n", 570 | "Epoch: 14, training loss: 0.617591917514801, valid roc_auc_score: 0.8237532149617826\n", 571 | "Epoch: 15, training loss: 0.6072716116905212, valid roc_auc_score: 0.845905105101299\n", 572 | "Epoch: 16, training loss: 0.5972340703010559, valid roc_auc_score: 0.8635679308637236\n", 573 | "Epoch: 17, training loss: 0.5859787464141846, valid roc_auc_score: 0.8642446458042314\n", 574 | "Epoch: 18, training loss: 0.569355845451355, valid roc_auc_score: 0.8725765004739267\n", 575 | "Epoch: 19, training loss: 0.559634268283844, valid roc_auc_score: 0.880606549333198\n", 576 | "Epoch: 20, training loss: 0.5440987348556519, valid roc_auc_score: 0.8762095996653996\n", 577 | "Epoch: 21, training loss: 0.5339546799659729, valid roc_auc_score: 0.884704295940344\n", 578 | "Epoch: 22, training loss: 0.5243106484413147, valid roc_auc_score: 0.883151133305148\n", 579 | "Epoch: 23, training loss: 0.513563871383667, valid roc_auc_score: 0.890787374807736\n", 580 | "Epoch: 24, training loss: 0.5073722004890442, valid roc_auc_score: 0.8890371953311649\n", 581 | "Epoch: 25, training loss: 0.5005218982696533, valid roc_auc_score: 0.895701366375732\n", 582 | "Epoch: 26, training loss: 0.4956182837486267, valid roc_auc_score: 0.9078041528117392\n", 583 | "Epoch: 27, training loss: 0.4941596984863281, valid roc_auc_score: 0.9012601609087806\n", 584 | "Epoch: 28, training loss: 0.4928068220615387, valid roc_auc_score: 0.8992277528265863\n", 585 | "Epoch: 29, training loss: 0.48640885949134827, valid roc_auc_score: 0.9051891814329338\n" 586 | ] 587 | } 588 | ], 589 | "source": [ 590 | "for epoch in range(30):\n", 591 | " model.train()\n", 592 | " neg_train_edge = torch.randint(0, data.x.shape[0], train_edge_index.size(), dtype=torch.long)\n", 593 | " h = model(data.x.float(), train_edge_index)\n", 594 | " logits = model.decode(h, train_edge_index, neg_train_edge)\n", 595 | " labels = get_link_labels(train_edge_index, neg_train_edge)\n", 596 | " loss = F.binary_cross_entropy_with_logits(logits, labels)\n", 597 | " optimizer.zero_grad()\n", 598 | " loss.backward()\n", 599 | " optimizer.step()\n", 600 | " model.eval()\n", 601 | " with torch.no_grad():\n", 602 | " val_logits = model.decode(h, val_edge_index, neg_val_edge)\n", 603 | " val_logits = val_logits.sigmoid()\n", 604 | " print('Epoch: {}, training loss: {}, valid roc_auc_score: {}'.format(epoch, loss.item(), roc_auc_score(val_labels, val_logits)))" 605 | ] 606 | }, 607 | { 608 | "cell_type": "markdown", 609 | "id": "1ad2acb3-f850-42aa-8333-ab82733d72bb", 610 | "metadata": {}, 611 | "source": [ 612 | "## Stochastic Batch Training \n", 613 | "\n", 614 | "For stochastic batch training, we split the training edges into batches. At each specific batch, to do the link prediction, we need to know the neighbor graphs for each pair of nodes that has an edge.\n", 615 | "\n", 616 | "We use the edgeNeighborLoader, which can load the neighbors of the pair nodes of an edge and has the same parameters as neighborLoader(). The result of a batch is, for example,\n", 617 | "\n", 618 | "`Data(edge_index=[2, 6917], is_train=[6917], is_val=[6917], is_test=[6917], is_seed=[6917], x=[2188, 1433], y=[2188])`\n", 619 | "\n", 620 | "where `is_seed` indicates whether each edge is a seed edge or not\n" 621 | ] 622 | }, 623 | { 624 | "cell_type": "code", 625 | "execution_count": 20, 626 | "id": "3a41d8aa-55f7-4e31-a2b0-3edcfc1cd73d", 627 | "metadata": {}, 628 | "outputs": [], 629 | "source": [ 630 | "# Hyperparameters\n", 631 | "hp = {\"hidden_dim\": 128, \"out_dim\": 64, \"num_layers\": 2,\n", 632 | " \"dropout\": 0.6, \"lr\": 0.01, \"l2_penalty\": 5e-4}" 633 | ] 634 | }, 635 | { 636 | "cell_type": "code", 637 | "execution_count": 21, 638 | "id": "31f89253-93ca-479b-8d1c-5a06bef9aaa3", 639 | "metadata": {}, 640 | "outputs": [], 641 | "source": [ 642 | "model = GCN(1433, hp[\"hidden_dim\"], hp[\"out_dim\"], hp[\"num_layers\"], hp[\"dropout\"])\n", 643 | "optimizer = torch.optim.Adam(\n", 644 | " model.parameters(), lr=hp[\"lr\"], weight_decay=hp[\"l2_penalty\"]\n", 645 | ")" 646 | ] 647 | }, 648 | { 649 | "cell_type": "markdown", 650 | "id": "74bb697c-7355-46e0-b6ab-f54750eedb85", 651 | "metadata": {}, 652 | "source": [ 653 | "### Construct the edge_neighbor_loader for train/val edges" 654 | ] 655 | }, 656 | { 657 | "cell_type": "code", 658 | "execution_count": 22, 659 | "id": "c2711014-81ce-4ff4-8959-3e251b7bf394", 660 | "metadata": {}, 661 | "outputs": [ 662 | { 663 | "name": "stdout", 664 | "output_type": "stream", 665 | "text": [ 666 | "Installing and optimizing queries. It might take a minute if this is the first time you use this loader.\n", 667 | "Query installation finished.\n" 668 | ] 669 | } 670 | ], 671 | "source": [ 672 | "train_edge_neighbor_loader = conn.gds.edgeNeighborLoader(\n", 673 | " v_in_feats=[\"x\"],\n", 674 | " v_out_labels=[\"y\"],\n", 675 | " num_batches=5,\n", 676 | " e_extra_feats=[\"is_train\",\"is_val\"],\n", 677 | " output_format=\"PyG\",\n", 678 | " num_neighbors=10,\n", 679 | " num_hops=2,\n", 680 | " filter_by=\"is_train\",\n", 681 | " shuffle=False,\n", 682 | ")" 683 | ] 684 | }, 685 | { 686 | "cell_type": "code", 687 | "execution_count": 23, 688 | "id": "7e02ab37-dcf2-4681-971d-04a2ae8ed0e6", 689 | "metadata": {}, 690 | "outputs": [], 691 | "source": [ 692 | "val_edge_neighbor_loader = conn.gds.edgeNeighborLoader(\n", 693 | " v_in_feats=[\"x\"],\n", 694 | " v_out_labels=[\"y\"],\n", 695 | " num_batches=5,\n", 696 | " e_extra_feats=[\"is_train\",\"is_val\"],\n", 697 | " output_format=\"PyG\",\n", 698 | " num_neighbors=10,\n", 699 | " num_hops=2,\n", 700 | " filter_by=\"is_val\",\n", 701 | " shuffle=False,\n", 702 | ")" 703 | ] 704 | }, 705 | { 706 | "cell_type": "code", 707 | "execution_count": null, 708 | "id": "2bd6ab8a-a266-48f5-97ae-38aab5a2bab5", 709 | "metadata": {}, 710 | "outputs": [ 711 | { 712 | "name": "stdout", 713 | "output_type": "stream", 714 | "text": [ 715 | "Epoch: 0, training loss: 3.904411494731903, valid roc_auc_score: 0.8237392959086584\n", 716 | "Epoch: 1, training loss: 3.1914963126182556, valid roc_auc_score: 0.8996884395360859\n", 717 | "Epoch: 2, training loss: 2.9413991570472717, valid roc_auc_score: 0.9132218330419763\n", 718 | "Epoch: 3, training loss: 2.5968366861343384, valid roc_auc_score: 0.9252033087060395\n", 719 | "Epoch: 4, training loss: 2.427817314863205, valid roc_auc_score: 0.9299228861824314\n", 720 | "Epoch: 5, training loss: 2.3323494493961334, valid roc_auc_score: 0.9444609410999989\n", 721 | "Epoch: 6, training loss: 2.3284645080566406, valid roc_auc_score: 0.9524406324093495\n", 722 | "Epoch: 7, training loss: 2.2777881622314453, valid roc_auc_score: 0.9523057420733823\n", 723 | "Epoch: 8, training loss: 2.2383748292922974, valid roc_auc_score: 0.9618454989629739\n", 724 | "Epoch: 9, training loss: 2.2285644710063934, valid roc_auc_score: 0.9601150098542369\n", 725 | "Epoch: 10, training loss: 2.2250851690769196, valid roc_auc_score: 0.9643573788182338\n" 726 | ] 727 | } 728 | ], 729 | "source": [ 730 | "for epoch in range(10):\n", 731 | " model.train()\n", 732 | " total_loss = 0\n", 733 | " for bid, batch in enumerate(train_edge_neighbor_loader):\n", 734 | " # get the training edges and negative edges sampled in the same batch\n", 735 | " train_edges = batch.edge_index[:, batch.is_seed]\n", 736 | " neg_train_edges = torch.randint(0, batch.x.shape[0], train_edges.size(), dtype=torch.long)\n", 737 | " # The graph only include the edges whose is_train is True\n", 738 | " train_graph_edges = batch.edge_index[:, batch.is_train]\n", 739 | " h = model(batch.x.float(), train_graph_edges)\n", 740 | " logits = model.decode(h, train_edges, neg_train_edges)\n", 741 | " labels = get_link_labels(train_edges, neg_train_edges)\n", 742 | " loss = F.binary_cross_entropy_with_logits(logits, labels)\n", 743 | " optimizer.zero_grad()\n", 744 | " loss.backward()\n", 745 | " optimizer.step()\n", 746 | " total_loss += loss.item()\n", 747 | " model.eval()\n", 748 | " all_labels = []\n", 749 | " all_logits = []\n", 750 | " for batch in val_edge_neighbor_loader:\n", 751 | " val_edges = batch.edge_index[:, batch.is_seed]\n", 752 | " neg_val_edges = torch.randint(0, batch.x.shape[0], val_edges.size(), dtype=torch.long)\n", 753 | " # Need to use the train edge for GCN\n", 754 | " val_graph_edges = batch.edge_index[:, batch.is_train]\n", 755 | " with torch.no_grad():\n", 756 | " h = model(batch.x.float(), val_graph_edges)\n", 757 | " logits = model.decode(h, val_edges, neg_val_edges)\n", 758 | " labels = get_link_labels(val_edges, neg_val_edges)\n", 759 | " logits = logits.sigmoid()\n", 760 | " all_labels.extend(labels)\n", 761 | " all_logits.extend(logits)\n", 762 | " print('Epoch: {}, training loss: {}, valid roc_auc_score: {}'.format(epoch, total_loss, roc_auc_score(all_labels, all_logits)))\n", 763 | " " 764 | ] 765 | }, 766 | { 767 | "cell_type": "code", 768 | "execution_count": null, 769 | "id": "3ca8f8df-e966-4c91-845e-03c2161ad8fa", 770 | "metadata": {}, 771 | "outputs": [], 772 | "source": [] 773 | } 774 | ], 775 | "metadata": { 776 | "kernelspec": { 777 | "display_name": "PyTorch", 778 | "language": "python", 779 | "name": "python3" 780 | }, 781 | "language_info": { 782 | "codemirror_mode": { 783 | "name": "ipython", 784 | "version": 3 785 | }, 786 | "file_extension": ".py", 787 | "mimetype": "text/x-python", 788 | "name": "python", 789 | "nbconvert_exporter": "python", 790 | "pygments_lexer": "ipython3", 791 | "version": "3.9.13" 792 | }, 793 | "vscode": { 794 | "interpreter": { 795 | "hash": "fc5eadac82f5951e7eb836bb06f3c9df8e6d1eda5537a95773af6c6ed24cb2d0" 796 | } 797 | } 798 | }, 799 | "nbformat": 4, 800 | "nbformat_minor": 5 801 | } 802 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # TigerGraph ML Workbench: Graph ML as a Service 2 | 3 | TigerGraph’s Machine Learning Workbench is a Python-based toolkit that accelerates the development of graph-enhanced machine learning, which leverages the added insight from connected data and graph features for better predictions. The GitHub repository of the notebooks are found [here](https://github.com/tigergraph/graph-ml-notebooks). 4 | 5 | 6 | ## Set Up Your Workbench 7 | 8 | 13 | 14 | ### Step 1. Create DB credentials and set access permission 15 | 16 | To access the graph database through pyTigerGraph, we need to create a database username and password, then put these credentials in config.json. 17 | 18 | 1. Go back to your browser tab/window for TigerGraph Cloud. 19 | 2. Click on `Cluster` on the left side menu. For the cluster containing this workbench, click `Access Management`. 20 | 3. Create a database user and grant appropriate permissions (e.g., `globaldesigner`). More details about managing database users can be found here: https://docs.tigergraph.com/cloud/security/manage-db-users. More details about access control settings can be found here: https://docs.tigergraph.com/tigergraph-server/current/user-access/access-control-model#_built_in_roles. 21 | 22 | 23 | 24 | 25 | 26 | 27 | ### Step 2. Update the database credentials in config.json 28 | 29 | 30 | 31 | 1. After creating the login credentials in Step 1, go back to the ML Workbench and edit `config.json` in the root jupyter notebook folder to replace the host, username and password with your new credentials. Example: [config.json](./config.json) 32 | ```json 33 | { 34 | "host": "https://subdomain.i.tgcloud.io", 35 | "username": "user_1", 36 | "password": "MyPassword1!", 37 | "getToken": true 38 | } 39 | ``` 40 | Note: For the `host` parameter, it is the domain name of the Cluster. You can find it in Cluster’s Details page which can be found by clicking on Clusters on tgCloud's left panel, then by clicking on the cluster’s name in the list (`Details -> Network Information -> Domain`). Replace the substring `subdomain` with your actual subdomain. Make sure to keep the “https://” at the beginning of the domain in the json config. 41 | 42 | 43 | 44 | 45 | 2. Once the credentials are updated, all the example notebooks and demos will refer to this config for database connections via pyTigerGraph. For example, here is how the [algos/centrality.ipynb](algos/centrality.ipynb) notebook connects to the database: 46 | 47 | ```python 48 | from pyTigerGraph import TigerGraphConnection 49 | conn = TigerGraphConnection( 50 | host=config["host"], 51 | username=config["username"], 52 | password=config["password"] 53 | ) 54 | ``` 55 | 56 | ## Learn Graph ML from Example Notebooks 57 | 58 | The ML Workbench comes with a collection of canonical Python notebooks that will introduce you to a number of features of the TigerGraph ML ecosystem. 59 | 60 | 64 | 65 | - The `basics` directory contains notebooks on how to get started with pyTigerGraph. 66 | - The `algos` directory contains notebooks for each category of algorithms within TigerGraph's [Graph Data Science Library](https://docs.tigergraph.com/graph-ml/current/intro/). You can run these algorithms via the pyTigerGraph Featurizer functionality. 67 | - The `GNNs` directory contains tutorial notebooks on how to train GNNs using data stored in a TigerGraph database. 68 | - The `applications` directory contains end to end demos of common applications such as fraud detection and recommendation. 69 | 70 | We recommend starting with the tutorials in the `basics` folders if you are new to pyTigerGraph. Once you are familiar with our pyTigerGraph client, familiarize yourself with a few graph algorithms with the examples in the `algos` folder before going through the `GNNs` and end-to-end `applications` tutorials. 71 | 72 | 73 | ### 1. Getting Started with pyTigerGraph and GSQL 74 | 75 | | folder | notebook | intro | 76 | | :--- | :--- | :--- | 77 | | basics | [datasets.ipynb](./basics/datasets.ipynb) | Load Data into TigerGraph | 78 | | basics | [feature_engineering.ipynb](./basics/feature_engineering.ipynb) | Util functions about building graph features from TigerGraph | 79 | | basics | [pyTigergraph_101.ipynb](./basics/pyTigergraph_101.ipynb) | Basic pyTigerGraph examples| 80 | | basics | [gsql_101.ipynb](./basics/gsql_101.ipynb) | Basic GSQL 101 using pyTigerGraph | 81 | | basics | [gsql_102.ipynb](./basics/gsql_102.ipynb) | Advanced GSQL 102 (pattern match) using pyTigerGraph | 82 | | basics | [template_query.ipynb](./basics/template_query.ipynb) | How to call template query with pyTigerGraph | 83 | 84 | ### 2. Graph Algorithms 85 | 86 | | folder | notebook | intro | 87 | | :--- | :--- | :--- | 88 | | algos | [centrality.ipynb](./algos/centrality.ipynb) | Centrality algorithms | 89 | | algos | [community.ipynb](./algos/community.ipynb) | Community detection algorithms | 90 | | algos | [similarity.ipynb](./algos/similarity.ipynb) | Similarity algorithms | 91 | | algos | [pathfinding.ipynb](./algos/pathfinding.ipynb) | Pathfinding between vertices | 92 | | algos | [embedding.ipynb](./algos/embedding.ipynb) | Graph embedding algorithms | 93 | | algos | [classification.ipynb](./algos/classification.ipynb) | Node classification algorithms | 94 | | algos | [topologicalLinkPrediction.ipynb](./algos/topologicalLinkPrediction.ipynb) | Topological link predictions | 95 | 96 | 97 | ### 3. Graph Neural Networks with TigerGraph 98 | 99 | | folder | notebook | intro | 100 | | :--- | :--- | :--- | 101 | | GNNs/PyG | [gcn_node_classification.ipynb](./GNNs/PyG/gcn_node_classification.ipynb) | Node classification using PyG | 102 | | GNNs/PyG | [gcn_link_prediction.ipynb](./GNNs/PyG/gcn_link_prediction.ipynb) | Link prediction using PyG | 103 | | GNNs/PyG | [hgat_node_classification.ipynb](./GNNs/PyG/hgat_node_classification.ipynb) | Heterogeneous Graph Attention Network using PyG | 104 | | GNNs/DGL | [gcn_node_classification.ipynb](./GNNs/DGL/gcn_node_classification.ipynb) | Node classification using DGL | 105 | | GNNs/DGL | [rgcn_node_classification.ipynb](./GNNs/DGL/rgcn_node_classification.ipynb) | Heterogeneous Graph Convolutional Network using DGL | 106 | | GNNs/Spektral | [gcn_node_classification.ipynb](./GNNs/Spektral/gcn_node_classification.ipynb) | Node classification using Spektral for Tensorflow | 107 | 108 | 109 | ### 4. End-to-end Applications using Graph ML 110 | 111 | | folder | notebook | intro | 112 | | :--- | :--- | :--- | 113 | | applications/fraud_detection | [fraud_detection.ipynb](./applications/fraud_detection/fraud_detection.ipynb) | End-to-end fraud detection using Graph ML | 114 | | applications/recommendation | [recommendation.ipynb](./applications/recommendation/recommendation.ipynb) | End-to-end recommendation using Graph ML | 115 | -------------------------------------------------------------------------------- /algos/pathfinding.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "0e0831b7-cce4-41de-8520-a48260ff4825", 6 | "metadata": {}, 7 | "source": [ 8 | "# TigerGraph Graph Data Science Library 101 - Path Finding Algorithm\n", 9 | "\n", 10 | "This notebook shows the examples of using the most common path finding algorithms in TigerGraph Graph Science Library. More detailed explanations of these algorithms can be four in the official documentation (https://docs.tigergraph.com/graph-ml/current/pathfinding-algorithms/). " 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "id": "ec177fb7-4d6b-4a5d-82c8-0ee3c0486e63", 16 | "metadata": {}, 17 | "source": [ 18 | "## Step 1: Setting things up\n", 19 | "- Connect and Load data" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 1, 25 | "id": "eb94a475-0005-4aa0-ae68-23a5f462dad0", 26 | "metadata": {}, 27 | "outputs": [ 28 | { 29 | "data": { 30 | "application/vnd.jupyter.widget-view+json": { 31 | "model_id": "8029222c00254f4586c3fadd11c22bdf", 32 | "version_major": 2, 33 | "version_minor": 0 34 | }, 35 | "text/plain": [ 36 | "Downloading: 0%| | 0/286678171 [00:00 e_type_set: Edge type to traverse\n", 269 | "* SET reverse_e_type_set: Reverse edge type to traverse\n", 270 | "* STRING weight_attribute: The edge attribute to use as the weight of the edge.\n", 271 | "* INT top_k: The number of vertices to return\n", 272 | "* INT print_limit: The maximum number of vertices to return\n", 273 | "* BOOL print_results: Whether to output the final results to the console in JSON format\n", 274 | "* STRING filepath: If provided, the algorithm will save the output in CSV format to this file\n", 275 | "* STRING similarity_edge: If provided, the similarity score will be saved to this edge" 276 | ] 277 | }, 278 | { 279 | "cell_type": "code", 280 | "execution_count": 9, 281 | "id": "9ea085b8-f2c5-4d12-ae3e-01c409e772de", 282 | "metadata": {}, 283 | "outputs": [], 284 | "source": [ 285 | "params = {\n", 286 | " \"source\": {\"id\": \"Alex\", \"type\": \"Person\"},\n", 287 | " \"e_type_set\": [\"Likes\"],\n", 288 | " \"reverse_e_type_set\": [\"reverse_Likes\"],\n", 289 | " \"weight_attribute\": \"weight\",\n", 290 | " \"top_k\": 5,\n", 291 | " \"print_limit\": 5,\n", 292 | " \"print_results\": True,\n", 293 | " \"file_path\": \"\",\n", 294 | " \"similarity_edge\": \"Similarity\"\n", 295 | "}\n", 296 | "\n", 297 | "results = feat.runAlgorithm(\"tg_cosine_nbor_ss\", params=params)" 298 | ] 299 | }, 300 | { 301 | "cell_type": "markdown", 302 | "id": "d06b0363-2940-4ce4-9678-f1599196bb20", 303 | "metadata": {}, 304 | "source": [ 305 | "## Results\n", 306 | "\n", 307 | "The output size is almost always 𝑘, except in cases where the number of total vertices is lower than 𝑘. The algorithm may arbitrarily choose to output one vertex over another if there are tied similarity scores.\n", 308 | "\n", 309 | "using Movie graph, one way to calculate similarity between two people would be to see which movies they both rated similarly. Starting from one person’s name, this algorithm calculates the cosine similarity between the given person and every other person in the graph, as long as there is at least one movie they have both rated.\n", 310 | "\n", 311 | "Given the source vertex \"Alex\", and top_k is set to 5, then we calculate the cosine similarity between him and two other persons, Jing and Kevin (since the example graph does not have enough data to return 5 Person vertices). The output shows the most similar vertices and their similarity scores in descending order." 312 | ] 313 | }, 314 | { 315 | "cell_type": "code", 316 | "execution_count": 10, 317 | "id": "6bdeea5f-9df7-468a-aa42-4bd32c8ea412", 318 | "metadata": {}, 319 | "outputs": [ 320 | { 321 | "data": { 322 | "text/html": [ 323 | "
\n", 324 | "\n", 337 | "\n", 338 | " \n", 339 | " \n", 340 | " \n", 341 | " \n", 342 | " \n", 343 | " \n", 344 | " \n", 345 | " \n", 346 | " \n", 347 | " \n", 348 | " \n", 349 | " \n", 350 | " \n", 351 | " \n", 352 | " \n", 353 | " \n", 354 | " \n", 355 | " \n", 356 | " \n", 357 | " \n", 358 | " \n", 359 | " \n", 360 | "
v_idv_typeattributes.neighbours.@sum_similarity
0JingPerson0.42173
1KevinPerson0.14248
\n", 361 | "
" 362 | ], 363 | "text/plain": [ 364 | " v_id v_type attributes.neighbours.@sum_similarity\n", 365 | "0 Jing Person 0.42173\n", 366 | "1 Kevin Person 0.14248" 367 | ] 368 | }, 369 | "metadata": {}, 370 | "output_type": "display_data" 371 | } 372 | ], 373 | "source": [ 374 | "df_cosine_nbor_ss = pd.json_normalize(results, record_path =['neighbours'])\n", 375 | "\n", 376 | "# display(df_cosine_nbor_ss.columns)\n", 377 | "\n", 378 | "display(df_cosine_nbor_ss.sort_values(by='attributes.neighbours.@sum_similarity', ascending=False))" 379 | ] 380 | }, 381 | { 382 | "cell_type": "markdown", 383 | "id": "d318d305-ef63-4326-a997-f884bf86aa9e", 384 | "metadata": {}, 385 | "source": [ 386 | "## tg_jaccard_nbor_ss\n", 387 | "The Jaccard index measures the relative overlap between two sets. To compare two vertices by Jaccard similarity, first select a set of attribute values for each vertex (https://docs.tigergraph.com/graph-ml/current/similarity-algorithms/jaccard-similarity-of-neighborhoods-single-source)." 388 | ] 389 | }, 390 | { 391 | "cell_type": "markdown", 392 | "id": "261230a9-505b-4076-a244-7708abd02187", 393 | "metadata": {}, 394 | "source": [ 395 | "## Input Parameters\n", 396 | "\n", 397 | "* VERTEX source: Source vertex {\"id\": \"vertex_id\", \"type\": \"vertex_type\"}\n", 398 | "* STRING e_type: Edge type to traverse\n", 399 | "* STRING reverse_e_type: Reverse edge type to traverse\n", 400 | "* INT top_k: The number of vertices to return\n", 401 | "* BOOL print_results: Whether to output the final results to the console in JSON format\n", 402 | "* STRING similarity_edge_type: If provided, the similarity score will be saved to this edge\n", 403 | "* STRING filepath: If provided, the algorithm will save the output in CSV format to this file" 404 | ] 405 | }, 406 | { 407 | "cell_type": "code", 408 | "execution_count": 11, 409 | "id": "a1a29abb-945f-4ccf-8534-b609d3116250", 410 | "metadata": {}, 411 | "outputs": [], 412 | "source": [ 413 | "params = {\n", 414 | " \"source\": {\"id\": \"Neil\", \"type\": \"Person\"},\n", 415 | " \"e_type\": \"Likes\",\n", 416 | " \"reverse_e_type\": \"reverse_Likes\",\n", 417 | " \"top_k\": 5,\n", 418 | " \"print_results\": True,\n", 419 | " \"similarity_edge_type\": \"Similarity\",\n", 420 | " \"file_path\": \"\",\n", 421 | "}\n", 422 | "\n", 423 | "results = feat.runAlgorithm(\"tg_jaccard_nbor_ss\", params=params)" 424 | ] 425 | }, 426 | { 427 | "cell_type": "markdown", 428 | "id": "1c3104e4-bf78-4e29-b2c6-1d576637fe2b", 429 | "metadata": {}, 430 | "source": [ 431 | "## Results\n", 432 | "\n", 433 | "This example uses Movie graph consisting of Person and Movie vertices. There are Likes edges that are weighted according to how much the person liked the movie. Each person in the dataset liked at least one movie, but not all movies were liked by all people.\n", 434 | "\n", 435 | "When comparing similarity to Neil, Kat is ranked higher than Kevin. This makes intuitive sense, because Kat likes two movies, both of which were also liked by Neil. Kevin also likes two movies that Neil likes. However, Kevin also likes a third movie that Neil doesn’t like, and is therefore less similar than Kat was.\n", 436 | "\n", 437 | "Although we set top_k to 5, only three vertices were returned because neither Alex nor Elena likes any movies that Kevin likes.\n", 438 | "\n", 439 | "If the source vertex (Person) doesn’t have any common neighbors (Movie) with any other vertex (Person), such as Elena in our example, the result is an empty list." 440 | ] 441 | }, 442 | { 443 | "cell_type": "code", 444 | "execution_count": 12, 445 | "id": "db9a50a8-2b10-4839-b891-18d63a154ec5", 446 | "metadata": {}, 447 | "outputs": [ 448 | { 449 | "data": { 450 | "text/html": [ 451 | "
\n", 452 | "\n", 465 | "\n", 466 | " \n", 467 | " \n", 468 | " \n", 469 | " \n", 470 | " \n", 471 | " \n", 472 | " \n", 473 | " \n", 474 | " \n", 475 | " \n", 476 | " \n", 477 | " \n", 478 | " \n", 479 | " \n", 480 | " \n", 481 | " \n", 482 | " \n", 483 | " \n", 484 | " \n", 485 | " \n", 486 | " \n", 487 | " \n", 488 | " \n", 489 | " \n", 490 | " \n", 491 | " \n", 492 | " \n", 493 | " \n", 494 | "
v_idv_typeattributes.Others.@sum_similarity
0KatPerson0.5
2KevinPerson0.4
1JingPerson0.2
\n", 495 | "
" 496 | ], 497 | "text/plain": [ 498 | " v_id v_type attributes.Others.@sum_similarity\n", 499 | "0 Kat Person 0.5\n", 500 | "2 Kevin Person 0.4\n", 501 | "1 Jing Person 0.2" 502 | ] 503 | }, 504 | "metadata": {}, 505 | "output_type": "display_data" 506 | } 507 | ], 508 | "source": [ 509 | "df_jaccard_nbor_ss = pd.json_normalize(results, record_path =['Others'])\n", 510 | "\n", 511 | "display(df_jaccard_nbor_ss.sort_values(by='attributes.Others.@sum_similarity', ascending=False))" 512 | ] 513 | }, 514 | { 515 | "cell_type": "markdown", 516 | "id": "37cf00ad-865d-44e4-9a08-cfd7fe697141", 517 | "metadata": {}, 518 | "source": [ 519 | "## tg_jaccard_nbor_ap_batch\n", 520 | "This algorithm computes the same similarity scores as the Jaccard similarity of neighborhoods, single source. Instead of selecting a single source vertex, however, it calculates similarity scores for all vertex pairs in the graph in parallel. Since this is a memory-intensive operation, it is split into batches to reduce peak memory usage. The user can specify how many batches it is to be split into. (https://docs.tigergraph.com/graph-ml/current/similarity-algorithms/jaccard-similarity-of-neighborhoods-batch)" 521 | ] 522 | }, 523 | { 524 | "cell_type": "markdown", 525 | "id": "70f5e6d6-f828-4a67-a66b-0f8a1ce8459d", 526 | "metadata": {}, 527 | "source": [ 528 | "## Input Parameters\n", 529 | "\n", 530 | "* INT top_k: The number of vertices to return\n", 531 | "* SET v_type_set: Vertex type used to calculate similarity score\n", 532 | "* SET feat_v_type: Feature vertex type\n", 533 | "* SET e_type_set: Edge type to traverse\n", 534 | "* SET reverse_e_type_set: Reverse edge type to traverse\n", 535 | "* STRING similarity_edge: If provided, the similarity score will be saved to this edge\n", 536 | "* INT src_batch_num: The number of batches to split the source vertices into\n", 537 | "* INT nbor_batch_num: The number of batches to split the 2-hop neighbor vertices into\n", 538 | "* BOOL print_accum: Whether to output the final results to the console in JSON format\n", 539 | "* INT print_limit: The number of source vertices to print, -1 to print all\n", 540 | "* STRING filepath: If provided, the algorithm will save the output in CSV format to this file" 541 | ] 542 | }, 543 | { 544 | "cell_type": "code", 545 | "execution_count": 13, 546 | "id": "aea0ca3d-61b0-4b78-9fcb-8663ea3c4a47", 547 | "metadata": {}, 548 | "outputs": [], 549 | "source": [ 550 | "params = {\n", 551 | " \"top_k\": 10,\n", 552 | " \"v_type_set\": [\"Person\"],\n", 553 | " \"feat_v_type\": [\"Movie\"],\n", 554 | " \"e_type_set\": [\"Likes\"],\n", 555 | " \"reverse_e_type_set\": [\"reverse_Likes\"],\n", 556 | " \"similarity_edge\": \"Similarity\",\n", 557 | " \"src_batch_num\": 50,\n", 558 | " \"nbor_batch_num\": 10,\n", 559 | " \"print_results\": True,\n", 560 | " \"print_limit\": 50,\n", 561 | " \"file_path\": \"\"\n", 562 | "}\n", 563 | "\n", 564 | "results = feat.runAlgorithm(\"tg_jaccard_nbor_ap_batch\", params=params)" 565 | ] 566 | }, 567 | { 568 | "cell_type": "markdown", 569 | "id": "2adde08d-c110-4949-856f-bad747f8aa2c", 570 | "metadata": {}, 571 | "source": [ 572 | "## Results\n", 573 | "\n", 574 | "The result contains the top k Jaccard similarity scores for each vertex and its corresponding pair. A pair is only included if its similarity is greater than 0, meaning there is at least one common neighbor between the pair." 575 | ] 576 | }, 577 | { 578 | "cell_type": "code", 579 | "execution_count": 14, 580 | "id": "941aa887-d4e1-41f3-be52-3cde41feff74", 581 | "metadata": {}, 582 | "outputs": [ 583 | { 584 | "name": "stdout", 585 | "output_type": "stream", 586 | "text": [ 587 | "Kevin Person\n", 588 | "{'ver': 'Neil', 'val': 0.4}\n", 589 | "{'ver': 'Kat', 'val': 0.25}\n", 590 | "{'ver': 'Alex', 'val': 0.2}\n", 591 | "Neil Person\n", 592 | "{'ver': 'Kat', 'val': 0.5}\n", 593 | "{'ver': 'Kevin', 'val': 0.4}\n", 594 | "{'ver': 'Jing', 'val': 0.2}\n", 595 | "Jing Person\n", 596 | "{'ver': 'Alex', 'val': 0.25}\n", 597 | "{'ver': 'Neil', 'val': 0.2}\n", 598 | "Kat Person\n", 599 | "{'ver': 'Neil', 'val': 0.5}\n", 600 | "{'ver': 'Kevin', 'val': 0.25}\n", 601 | "Alex Person\n", 602 | "{'ver': 'Jing', 'val': 0.25}\n", 603 | "{'ver': 'Kevin', 'val': 0.2}\n", 604 | "Elena Person\n" 605 | ] 606 | } 607 | ], 608 | "source": [ 609 | "df_jaccard_nbor_ap_batch = pd.json_normalize(results, record_path =['print_batch'])\n", 610 | "\n", 611 | "df_jaccard_nbor_ap_batch.columns = ['v_id', 'v_type', 'sim_heap']\n", 612 | "\n", 613 | "df_jaccard_nbor_ap_batch = df_jaccard_nbor_ap_batch.reset_index()\n", 614 | "\n", 615 | "for index, row in df_jaccard_nbor_ap_batch.iterrows():\n", 616 | " print(row['v_id'], row['v_type'])\n", 617 | " for p in row['sim_heap']:\n", 618 | " print(p)" 619 | ] 620 | } 621 | ], 622 | "metadata": { 623 | "kernelspec": { 624 | "display_name": "Python 3 (ipykernel)", 625 | "language": "python", 626 | "name": "python3" 627 | }, 628 | "language_info": { 629 | "codemirror_mode": { 630 | "name": "ipython", 631 | "version": 3 632 | }, 633 | "file_extension": ".py", 634 | "mimetype": "text/x-python", 635 | "name": "python", 636 | "nbconvert_exporter": "python", 637 | "pygments_lexer": "ipython3", 638 | "version": "3.9.6" 639 | } 640 | }, 641 | "nbformat": 4, 642 | "nbformat_minor": 5 643 | } 644 | -------------------------------------------------------------------------------- /algos/topologicalLinkPrediction.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "9cfa203e-9881-473d-95a3-e9ab912ccd8d", 6 | "metadata": {}, 7 | "source": [ 8 | "# TigerGraph Data Science Library 101 - Topological Link Predication Algorithm\n", 9 | "This notebook shows the examples of using the most common topological link predication algorithms in TigerGraph Graph Science Library. More detailed explanations of these algorithms can be found in the official documentation (https://docs.tigergraph.com/graph-ml/current/link-prediction/).\n", 10 | "\n", 11 | "\n", 12 | "## Step1: Setting things up\n", 13 | "- Connect and Load data\n", 14 | "- Visualize the graph schema \n", 15 | "- Get basic stats, e.g., counts of nodes & edges" 16 | ] 17 | }, 18 | { 19 | "cell_type": "markdown", 20 | "id": "b02309b4-cd57-4074-b849-6a09c6232e70", 21 | "metadata": {}, 22 | "source": [ 23 | "### Create connection" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 1, 29 | "id": "2ead3778-60b5-4cc0-9d54-cc6c5209e4b7", 30 | "metadata": {}, 31 | "outputs": [], 32 | "source": [ 33 | "import json\n", 34 | "import pandas as pd\n", 35 | "from pyTigerGraph import TigerGraphConnection\n", 36 | "\n", 37 | "# Read in DB configs\n", 38 | "with open('../config.json', \"r\") as config_file:\n", 39 | " config = json.load(config_file)\n", 40 | "\n", 41 | "conn = TigerGraphConnection(\n", 42 | " host=config[\"host\"],\n", 43 | " username=config[\"username\"],\n", 44 | " password=config[\"password\"],\n", 45 | ")" 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "id": "93364e5a-ce51-4bfc-9c0c-492886eb8301", 51 | "metadata": {}, 52 | "source": [ 53 | "### Download movie dataset" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": 2, 59 | "id": "63e14e42-7360-438d-bd51-6cf3ef13295f", 60 | "metadata": {}, 61 | "outputs": [ 62 | { 63 | "data": { 64 | "application/vnd.jupyter.widget-view+json": { 65 | "model_id": "9be1d8f44ace40f49f118e30fbbfec66", 66 | "version_major": 2, 67 | "version_minor": 0 68 | }, 69 | "text/plain": [ 70 | "Downloading: 0%| | 0/1970 [00:00 e_type_set: Edge types to traverse \n", 272 | "* BOOL print_results: if True, print result (True by default)" 273 | ] 274 | }, 275 | { 276 | "cell_type": "code", 277 | "execution_count": 9, 278 | "id": "aceadb2c-f554-4d97-a383-fbfc58541b60", 279 | "metadata": {}, 280 | "outputs": [], 281 | "source": [ 282 | "params = {\n", 283 | " \"v_source\": {\"id\": \"Alex\", \"type\": \"Person\"},\n", 284 | " \"v_target\": {\"id\": \"Bob\", \"type\": \"Person\"},\n", 285 | " \"e_type_set\": [\"Coworker\"],\n", 286 | " \"print_results\": True\n", 287 | "}\n", 288 | "\n", 289 | "results = feat.runAlgorithm(\"tg_adamic_adar\", params=params)" 290 | ] 291 | }, 292 | { 293 | "cell_type": "markdown", 294 | "id": "cb5deed4-bc9f-49f6-8e2b-328f4e9dace7", 295 | "metadata": {}, 296 | "source": [ 297 | "## Results\n", 298 | "\n", 299 | "Returns Adamic Adar index between the two given vertices. If the two vertices do not have common neighbors, the algorithm will return a division by 0 error" 300 | ] 301 | }, 302 | { 303 | "cell_type": "code", 304 | "execution_count": 10, 305 | "id": "7f56edcc-2579-43f5-b396-fe5141a5c583", 306 | "metadata": {}, 307 | "outputs": [ 308 | { 309 | "data": { 310 | "text/html": [ 311 | "
\n", 312 | "\n", 325 | "\n", 326 | " \n", 327 | " \n", 328 | " \n", 329 | " \n", 330 | " \n", 331 | " \n", 332 | " \n", 333 | " \n", 334 | " \n", 335 | " \n", 336 | " \n", 337 | " \n", 338 | "
@@sum_closeness
03.32193
\n", 339 | "
" 340 | ], 341 | "text/plain": [ 342 | " @@sum_closeness\n", 343 | "0 3.32193" 344 | ] 345 | }, 346 | "metadata": {}, 347 | "output_type": "display_data" 348 | } 349 | ], 350 | "source": [ 351 | "df_adamic_adar = pd.json_normalize(results)\n", 352 | "\n", 353 | "display(df_adamic_adar)" 354 | ] 355 | }, 356 | { 357 | "cell_type": "markdown", 358 | "id": "1083c852-223c-4c78-967f-9ce928a1f9d7", 359 | "metadata": {}, 360 | "source": [ 361 | "## tg_common_neighbors\n", 362 | "\n", 363 | "A vertex 𝐴 that is connected to vertices 𝐵 and 𝐶 is considered to be a \"common neighbor\" of 𝐵 and 𝐶. The common neighbors algorithm counts the number of common neighbors between two vertices. This algorithm ignores edge weights. (https://docs.tigergraph.com/graph-ml/current/link-prediction/common-neighbors)\n" 364 | ] 365 | }, 366 | { 367 | "cell_type": "markdown", 368 | "id": "8fc11216-4137-474a-ba8a-e68821e1f2a6", 369 | "metadata": {}, 370 | "source": [ 371 | "## Input Parameters\n", 372 | "\n", 373 | "* VERTEX v_source: The first vertex to compare {\"id\": \"vertex_id\", \"type\": \"vertex_type\"}\n", 374 | "* VERTEX v_target: The second vertex to compare with the first {\"id\": \"vertex_id\", \"type\": \"vertex_type\"}\n", 375 | "* SET e_type_set: Edge types to traverse \n", 376 | "* BOOL print_results: if True, print result (True by default)" 377 | ] 378 | }, 379 | { 380 | "cell_type": "code", 381 | "execution_count": 11, 382 | "id": "6e6bc7b9-41b2-4992-b073-204c4be9413a", 383 | "metadata": {}, 384 | "outputs": [], 385 | "source": [ 386 | "params = {\n", 387 | " \"v_source\": {\"id\": \"Alex\", \"type\": \"Person\"},\n", 388 | " \"v_target\": {\"id\": \"Bob\", \"type\": \"Person\"},\n", 389 | " \"e_type_set\": [\"Coworker\"],\n", 390 | " \"print_results\": True\n", 391 | "}\n", 392 | "\n", 393 | "results = feat.runAlgorithm(\"tg_common_neighbors\", params=params)" 394 | ] 395 | }, 396 | { 397 | "cell_type": "markdown", 398 | "id": "3e695052-c939-4794-a86e-3e9176383ae4", 399 | "metadata": {}, 400 | "source": [ 401 | "## Results\n", 402 | "\n", 403 | "Returns the number of common neighbors between two vertices expressed as a closeness value." 404 | ] 405 | }, 406 | { 407 | "cell_type": "code", 408 | "execution_count": 12, 409 | "id": "8980aeaf-f344-4e0f-bc33-c8ab694091ca", 410 | "metadata": {}, 411 | "outputs": [ 412 | { 413 | "data": { 414 | "text/html": [ 415 | "
\n", 416 | "\n", 429 | "\n", 430 | " \n", 431 | " \n", 432 | " \n", 433 | " \n", 434 | " \n", 435 | " \n", 436 | " \n", 437 | " \n", 438 | " \n", 439 | " \n", 440 | " \n", 441 | " \n", 442 | "
closeness
01
\n", 443 | "
" 444 | ], 445 | "text/plain": [ 446 | " closeness\n", 447 | "0 1" 448 | ] 449 | }, 450 | "metadata": {}, 451 | "output_type": "display_data" 452 | } 453 | ], 454 | "source": [ 455 | "df_common_neighbors = pd.json_normalize(results)\n", 456 | "\n", 457 | "display(df_common_neighbors)" 458 | ] 459 | }, 460 | { 461 | "cell_type": "markdown", 462 | "id": "e5d6b309-f77f-45e3-95d5-4b02fbece770", 463 | "metadata": {}, 464 | "source": [ 465 | "## tg_preferential_attachment\n", 466 | "\n", 467 | "Preferential Attachment is a measure to compute the closeness of vertices based on the number of their neighbors. The algorithm returns the product of the number of neighbors of the first vertex and the number of neighbors of the second vertex. (https://docs.tigergraph.com/graph-ml/current/link-prediction/preferential-attachment)" 468 | ] 469 | }, 470 | { 471 | "cell_type": "markdown", 472 | "id": "b9c7e418-944c-45d2-af34-205cc5170610", 473 | "metadata": {}, 474 | "source": [ 475 | "## Input Parameters\n", 476 | "\n", 477 | "* VERTEX v_source: The first vertex to compare {\"id\": \"vertex_id\", \"type\": \"vertex_type\"}\n", 478 | "* VERTEX v_target: The second vertex to compare with the first {\"id\": \"vertex_id\", \"type\": \"vertex_type\"}\n", 479 | "* SET e_type_set: Edge types to traverse\n", 480 | "* BOOL print_results: if True, print result (True by default)" 481 | ] 482 | }, 483 | { 484 | "cell_type": "code", 485 | "execution_count": 13, 486 | "id": "10fd47b7-c349-428f-b05c-8388603ffc10", 487 | "metadata": {}, 488 | "outputs": [], 489 | "source": [ 490 | "params = {\n", 491 | " \"v_source\": {\"id\": \"Alex\", \"type\": \"Person\"},\n", 492 | " \"v_target\": {\"id\": \"Bob\", \"type\": \"Person\"},\n", 493 | " \"e_type_set\": [\"Coworker\"],\n", 494 | " \"print_results\": True\n", 495 | "}\n", 496 | "\n", 497 | "results = feat.runAlgorithm(\"tg_preferential_attachment\", params=params)" 498 | ] 499 | }, 500 | { 501 | "cell_type": "markdown", 502 | "id": "b602a173-348a-4cd9-908e-7c6c9582e315", 503 | "metadata": {}, 504 | "source": [ 505 | "## Results\n", 506 | "\n", 507 | "The product of the number of neighbors of the two vertices. " 508 | ] 509 | }, 510 | { 511 | "cell_type": "code", 512 | "execution_count": 14, 513 | "id": "4ee45c39-23d6-4000-80c4-d1c3886b733b", 514 | "metadata": {}, 515 | "outputs": [ 516 | { 517 | "data": { 518 | "text/html": [ 519 | "
\n", 520 | "\n", 533 | "\n", 534 | " \n", 535 | " \n", 536 | " \n", 537 | " \n", 538 | " \n", 539 | " \n", 540 | " \n", 541 | " \n", 542 | " \n", 543 | " \n", 544 | " \n", 545 | " \n", 546 | "
closeness
04
\n", 547 | "
" 548 | ], 549 | "text/plain": [ 550 | " closeness\n", 551 | "0 4" 552 | ] 553 | }, 554 | "metadata": {}, 555 | "output_type": "display_data" 556 | } 557 | ], 558 | "source": [ 559 | "df_preferential_attachment = pd.json_normalize(results)\n", 560 | "\n", 561 | "display(df_preferential_attachment)" 562 | ] 563 | }, 564 | { 565 | "cell_type": "markdown", 566 | "id": "141081ef-af67-40f5-8aff-ea95d5f47372", 567 | "metadata": {}, 568 | "source": [ 569 | "## tg_resource_allocation\n", 570 | "\n", 571 | "Resource Allocation is used to compute the closeness of nodes based on their shared neighbors. (https://docs.tigergraph.com/graph-ml/current/link-prediction/resource-allocation)" 572 | ] 573 | }, 574 | { 575 | "cell_type": "markdown", 576 | "id": "50ea581a-297b-4ea7-859d-0dd4408fcd61", 577 | "metadata": {}, 578 | "source": [ 579 | "## Input Parameters\n", 580 | "\n", 581 | "* VERTEX v_source: The first vertex to compare {\"id\": \"vertex_id\", \"type\": \"vertex_type\"}\n", 582 | "* VERTEX v_target: The second vertex to compare with the first {\"id\": \"vertex_id\", \"type\": \"vertex_type\"}\n", 583 | "* SET e_type_set: Edge types to traverse \n", 584 | "* BOOL print_results: if True, print result (True by default)" 585 | ] 586 | }, 587 | { 588 | "cell_type": "code", 589 | "execution_count": 15, 590 | "id": "76a2d7ab-bdb1-4850-94c0-ec0342806e5c", 591 | "metadata": {}, 592 | "outputs": [], 593 | "source": [ 594 | "params = {\n", 595 | " \"v_source\": {\"id\": \"Alex\", \"type\": \"Person\"},\n", 596 | " \"v_target\": {\"id\": \"Bob\", \"type\": \"Person\"},\n", 597 | " \"e_type_set\": [\"Coworker\"],\n", 598 | " \"print_results\": True\n", 599 | "}\n", 600 | "\n", 601 | "results = feat.runAlgorithm(\"tg_resource_allocation\", params=params)" 602 | ] 603 | }, 604 | { 605 | "cell_type": "markdown", 606 | "id": "67d9357d-dded-44c8-b459-bb5cbbd1292a", 607 | "metadata": {}, 608 | "source": [ 609 | "## Results\n", 610 | "\n", 611 | "Returns a closeness value for two input vertices." 612 | ] 613 | }, 614 | { 615 | "cell_type": "code", 616 | "execution_count": 16, 617 | "id": "18ff8bc7-3e90-44bc-8427-06455b7bc6b5", 618 | "metadata": {}, 619 | "outputs": [ 620 | { 621 | "data": { 622 | "text/html": [ 623 | "
\n", 624 | "\n", 637 | "\n", 638 | " \n", 639 | " \n", 640 | " \n", 641 | " \n", 642 | " \n", 643 | " \n", 644 | " \n", 645 | " \n", 646 | " \n", 647 | " \n", 648 | " \n", 649 | " \n", 650 | "
@@sum_closeness
00.5
\n", 651 | "
" 652 | ], 653 | "text/plain": [ 654 | " @@sum_closeness\n", 655 | "0 0.5" 656 | ] 657 | }, 658 | "metadata": {}, 659 | "output_type": "display_data" 660 | } 661 | ], 662 | "source": [ 663 | "df_resource_allocation = pd.json_normalize(results)\n", 664 | "\n", 665 | "display(df_resource_allocation)" 666 | ] 667 | }, 668 | { 669 | "cell_type": "markdown", 670 | "id": "dfafde54-447a-4afe-a2a6-1faae19b62cb", 671 | "metadata": { 672 | "tags": [] 673 | }, 674 | "source": [ 675 | "## tg_total_neighbors\n", 676 | "\n", 677 | "The algorithm counts the total number of neighbors, or vertices connected by one hop, of two vertices. (https://docs.tigergraph.com/graph-ml/current/link-prediction/total-neighbors)" 678 | ] 679 | }, 680 | { 681 | "cell_type": "markdown", 682 | "id": "96003131-be56-443a-bbc1-6ce971a3ec51", 683 | "metadata": {}, 684 | "source": [ 685 | "## Input Parameters\n", 686 | "\n", 687 | "* VERTEX v_source: The first vertex to compare {\"id\": \"vertex_id\", \"type\": \"vertex_type\"}\n", 688 | "* VERTEX v_target: The second vertex to compare with the first {\"id\": \"vertex_id\", \"type\": \"vertex_type\"}\n", 689 | "* SET e_type_set: Edge types to traverse \n", 690 | "* BOOL print_results: if True, print result (True by default)" 691 | ] 692 | }, 693 | { 694 | "cell_type": "code", 695 | "execution_count": 17, 696 | "id": "32ed10a2-3fcf-48bb-9f1d-6f28befbba96", 697 | "metadata": {}, 698 | "outputs": [], 699 | "source": [ 700 | "params = {\n", 701 | " \"v_source\": {\"id\": \"Alex\", \"type\": \"Person\"},\n", 702 | " \"v_target\": {\"id\": \"Bob\", \"type\": \"Person\"},\n", 703 | " \"e_type_set\": [\"Coworker\"],\n", 704 | " \"print_results\": True\n", 705 | "}\n", 706 | "\n", 707 | "results = feat.runAlgorithm(\"tg_total_neighbors\", params=params)" 708 | ] 709 | }, 710 | { 711 | "cell_type": "code", 712 | "execution_count": null, 713 | "id": "13662066", 714 | "metadata": {}, 715 | "outputs": [], 716 | "source": [ 717 | "print(results)" 718 | ] 719 | }, 720 | { 721 | "cell_type": "markdown", 722 | "id": "9281c0ae-ef03-4e5a-98d6-02802f6afbd2", 723 | "metadata": {}, 724 | "source": [ 725 | "## Results\n", 726 | "\n", 727 | "The total number of neighbors of two vertices as a closeness value." 728 | ] 729 | }, 730 | { 731 | "cell_type": "code", 732 | "execution_count": 18, 733 | "id": "dc6e6c05-cc84-4bb3-b2ee-1cc65382d379", 734 | "metadata": {}, 735 | "outputs": [ 736 | { 737 | "data": { 738 | "text/html": [ 739 | "
\n", 740 | "\n", 753 | "\n", 754 | " \n", 755 | " \n", 756 | " \n", 757 | " \n", 758 | " \n", 759 | " \n", 760 | " \n", 761 | " \n", 762 | " \n", 763 | " \n", 764 | " \n", 765 | " \n", 766 | "
closeness
03
\n", 767 | "
" 768 | ], 769 | "text/plain": [ 770 | " closeness\n", 771 | "0 3" 772 | ] 773 | }, 774 | "metadata": {}, 775 | "output_type": "display_data" 776 | } 777 | ], 778 | "source": [ 779 | "df_total_neighbors = pd.json_normalize(results)\n", 780 | "\n", 781 | "display(df_total_neighbors)" 782 | ] 783 | } 784 | ], 785 | "metadata": { 786 | "kernelspec": { 787 | "display_name": "Python 3 (ipykernel)", 788 | "language": "python", 789 | "name": "python3" 790 | }, 791 | "language_info": { 792 | "codemirror_mode": { 793 | "name": "ipython", 794 | "version": 3 795 | }, 796 | "file_extension": ".py", 797 | "mimetype": "text/x-python", 798 | "name": "python", 799 | "nbconvert_exporter": "python", 800 | "pygments_lexer": "ipython3", 801 | "version": "3.9.6" 802 | } 803 | }, 804 | "nbformat": 4, 805 | "nbformat_minor": 5 806 | } 807 | -------------------------------------------------------------------------------- /applications/fraud_detection/gsql/amounts.gsql: -------------------------------------------------------------------------------- 1 | CREATE QUERY amounts(STRING send_min="send_min", STRING send_amount="send_amount", STRING recv_min = "recv_min", STRING recv_amount = "recv_amount") SYNTAX V1 { 2 | MinAccum @send_min; 3 | SumAccum @send_amount; 4 | MinAccum @recv_min; 5 | SumAccum @recv_amount; 6 | 7 | res = 8 | SELECT p 9 | FROM ANY:p -(Transaction:e)- :q 10 | ACCUM p.@send_min += e.amount, q.@recv_min += e.amount, 11 | p.@send_amount += e.amount, q.@recv_amount += e.amount; 12 | 13 | res = 14 | SELECT p 15 | FROM ANY:p 16 | POST-ACCUM 17 | p.setAttr(recv_min, p.@recv_min), 18 | p.setAttr(recv_amount, p.@recv_amount), 19 | p.setAttr(send_min, p.@send_min), 20 | p.setAttr(send_amount, p.@send_amount), 21 | IF (p.in_degree == 0) THEN p.setAttr(recv_min, 0) END, 22 | IF (p.out_degree == 0) THEN p.setAttr(send_min, 0) END; 23 | 24 | PRINT "Amounts computed successfully" as Status; 25 | } -------------------------------------------------------------------------------- /applications/fraud_detection/gsql/component_size.gsql: -------------------------------------------------------------------------------- 1 | CREATE QUERY component_size(STRING result_attr) FOR GRAPH Ethereum { 2 | MapAccum> @@component_count; 3 | 4 | res = SELECT s FROM Account:s POST-ACCUM @@component_count += (s.wcc_id -> 1); 5 | 6 | res = SELECT s FROM Account:s POST-ACCUM 7 | INT tmp = @@component_count.get(s.wcc_id), 8 | s.setAttr(result_attr, tmp); 9 | } -------------------------------------------------------------------------------- /applications/fraud_detection/gsql/degrees.gsql: -------------------------------------------------------------------------------- 1 | CREATE QUERY degrees(STRING in_degree="in_degree", STRING out_degree="out_degree") SYNTAX V1 { 2 | SumAccum @in_degree; 3 | SumAccum @out_degree; 4 | 5 | res1 = 6 | SELECT p 7 | FROM ANY:p - (Transaction:e) - :q 8 | ACCUM p.@out_degree += 1, q.@in_degree += 1; 9 | 10 | res2 = 11 | SELECT p 12 | FROM ANY:p 13 | POST-ACCUM 14 | p.setAttr(in_degree, p.@in_degree), 15 | p.setAttr(out_degree, p.@out_degree); 16 | 17 | PRINT "Degrees computed Successfully" as Status; 18 | } -------------------------------------------------------------------------------- /applications/fraud_detection/gsql/downsample.gsql: -------------------------------------------------------------------------------- 1 | CREATE QUERY downsample() FOR GRAPH Ethereum SYNTAX V2 { 2 | 3 | S = {Account.*}; 4 | 5 | DELETE s FROM S:s 6 | WHERE s.is_fraud == FALSE AND s.recv_amount == 0 AND s.send_amount == 0; 7 | PRINT "Success!"; 8 | } -------------------------------------------------------------------------------- /basics/datasets.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "300b2d48-024f-4d54-8688-8ac2cf7c10e0", 6 | "metadata": {}, 7 | "source": [ 8 | "# Datasets" 9 | ] 10 | }, 11 | { 12 | "cell_type": "markdown", 13 | "id": "122d8f47-a77e-4c70-8bfd-9e1692620ebf", 14 | "metadata": {}, 15 | "source": [ 16 | "This notebook demostrates how to load two example datasets into your TigerGraph database. It uses [pyTigerGraph](https://github.com/tigergraph/pyTigerGraph) to download the datasets and ingest them into your database. Those datasets will be used throughout the remaining notebooks in the basics directory.\n", 17 | "\n", 18 | "The **Cora** dataset contains 2708 machine learning papers and 10556 citation links between the papers. Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from a dictionary. The dictionary consists of 1433 unique words. Each paper is classified into one of seven classes based on the topic.\n", 19 | "\n", 20 | "The **IMDB** dataset contains 3 types of vertices: 4278 movies, 5257 actors, and 2081 directors; and 4 types of edges: 12828 actor to movie edges, 12828 movie to actor edges, 4278 director to movie edges, and 4278 movie to director edges. Each vertex is described by a 0/1-valued word vector indicating the absence/presence of the corresponding keywords. For movies, the keywords are extracted from their plots; and for actors and directors, the keywords are extracted from the plots of movies they participated. Each movie is classified into one of three classes, action, comedy, and drama according to their genre. The goal is to predict the class of each movie in the graph.\n", 21 | "\n", 22 | "To connect your database, modify the `config.json` file accompanying this notebook. Set the value of `getToken` based on whether token auth is enabled for your database. Token auth is always enabled for tgcloud databases. " 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "id": "ac7c3281", 28 | "metadata": {}, 29 | "source": [ 30 | "## Cora Dataset" 31 | ] 32 | }, 33 | { 34 | "cell_type": "markdown", 35 | "id": "706cf86a", 36 | "metadata": {}, 37 | "source": [ 38 | "### Download dataset" 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": 1, 44 | "id": "ad11bd6c", 45 | "metadata": {}, 46 | "outputs": [ 47 | { 48 | "data": { 49 | "application/vnd.jupyter.widget-view+json": { 50 | "model_id": "9e72b47531ad4e8899626ef2c611e0e0", 51 | "version_major": 2, 52 | "version_minor": 0 53 | }, 54 | "text/plain": [ 55 | "Downloading: 0%| | 0/166537 [00:00