├── P&ID_Symbol_detection.ipynb └── README.md /P&ID_Symbol_detection.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "accelerator": "GPU", 6 | "colab": { 7 | "name": "3P_ID_object_detection_tutorial_12_07_2021.ipynb", 8 | "private_outputs": true, 9 | "provenance": [], 10 | "collapsed_sections": [], 11 | "include_colab_link": true 12 | }, 13 | "kernelspec": { 14 | "display_name": "Python 3", 15 | "name": "python3" 16 | } 17 | }, 18 | "cells": [ 19 | { 20 | "cell_type": "markdown", 21 | "metadata": { 22 | "id": "view-in-github", 23 | "colab_type": "text" 24 | }, 25 | "source": [ 26 | "\"Open" 27 | ] 28 | }, 29 | { 30 | "cell_type": "markdown", 31 | "metadata": { 32 | "id": "NUtwE4KGxKT-" 33 | }, 34 | "source": [ 35 | "# Symbol detection from P&ID " 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": { 41 | "id": "3cIrseUv6WKz" 42 | }, 43 | "source": [ 44 | " Tensorflow Object Detection to detect dynamic symbol from P&ID" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": { 50 | "id": "p3UGXxUii5Ym" 51 | }, 52 | "source": [ 53 | "### Install Trensorflow" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "metadata": { 59 | "id": "hGL97-GXjSUw" 60 | }, 61 | "source": [ 62 | "!pip install -U --pre tensorflow==\"2.*\"\n", 63 | "!pip install tf_slim" 64 | ], 65 | "execution_count": null, 66 | "outputs": [] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": { 71 | "id": "n_ap_s9ajTHH" 72 | }, 73 | "source": [ 74 | "Make sure you have `pycocotools` installed" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "metadata": { 80 | "id": "Bg8ZyA47i3pY" 81 | }, 82 | "source": [ 83 | "!pip install pycocotools" 84 | ], 85 | "execution_count": null, 86 | "outputs": [] 87 | }, 88 | { 89 | "cell_type": "markdown", 90 | "metadata": { 91 | "id": "-vsOL3QR6kqs" 92 | }, 93 | "source": [ 94 | "Get `tensorflow/models` or `cd` to parent directory of the repository." 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "metadata": { 100 | "id": "ykA0c-om51s1" 101 | }, 102 | "source": [ 103 | "import os\n", 104 | "import pathlib\n", 105 | "\n", 106 | "\n", 107 | "if \"models\" in pathlib.Path.cwd().parts:\n", 108 | " while \"models\" in pathlib.Path.cwd().parts:\n", 109 | " os.chdir('..')\n", 110 | "elif not pathlib.Path('models').exists():\n", 111 | " !git clone --depth 1 https://github.com/tensorflow/models" 112 | ], 113 | "execution_count": null, 114 | "outputs": [] 115 | }, 116 | { 117 | "cell_type": "markdown", 118 | "metadata": { 119 | "id": "O219m6yWAj9l" 120 | }, 121 | "source": [ 122 | "Compile protobufs and install the object_detection package" 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "metadata": { 128 | "id": "cTDU30R7HDgl" 129 | }, 130 | "source": [ 131 | "%%bash\n", 132 | "cd models/research/\n", 133 | "protoc object_detection/protos/*.proto --python_out=.\n", 134 | "cp object_detection/packages/tf2/setup.py .\n", 135 | "python -m pip install ." 136 | ], 137 | "execution_count": null, 138 | "outputs": [] 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "metadata": { 143 | "id": "8xo80Swh2mJf" 144 | }, 145 | "source": [ 146 | "# Configure Google drive to access Annotation image file and other data " 147 | ] 148 | }, 149 | { 150 | "cell_type": "code", 151 | "metadata": { 152 | "id": "CS07oQ07kVdq" 153 | }, 154 | "source": [ 155 | "from google.colab import drive\n", 156 | "drive.mount('/content/drive')" 157 | ], 158 | "execution_count": null, 159 | "outputs": [] 160 | }, 161 | { 162 | "cell_type": "code", 163 | "metadata": { 164 | "id": "1AkUZimAkd2B" 165 | }, 166 | "source": [ 167 | "os.chdir('/content/drive/MyDrive/object_detect')" 168 | ], 169 | "execution_count": null, 170 | "outputs": [] 171 | }, 172 | { 173 | "cell_type": "code", 174 | "metadata": { 175 | "id": "YNXZ6FwJk85H" 176 | }, 177 | "source": [ 178 | "pwd" 179 | ], 180 | "execution_count": null, 181 | "outputs": [] 182 | }, 183 | { 184 | "cell_type": "markdown", 185 | "metadata": { 186 | "id": "LBdjK2G5ywuc" 187 | }, 188 | "source": [ 189 | "### Imports Library dependencies" 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "metadata": { 195 | "id": "hV4P5gyTWKMI" 196 | }, 197 | "source": [ 198 | "import numpy as np\n", 199 | "import os\n", 200 | "import six.moves.urllib as urllib\n", 201 | "import sys\n", 202 | "import tarfile\n", 203 | "import tensorflow as tf\n", 204 | "import zipfile\n", 205 | "\n", 206 | "from collections import defaultdict\n", 207 | "from io import StringIO\n", 208 | "from matplotlib import pyplot as plt\n", 209 | "from PIL import Image\n", 210 | "from IPython.display import display" 211 | ], 212 | "execution_count": null, 213 | "outputs": [] 214 | }, 215 | { 216 | "cell_type": "markdown", 217 | "metadata": { 218 | "id": "r5FNuiRPWKMN" 219 | }, 220 | "source": [ 221 | "Import the object detection module." 222 | ] 223 | }, 224 | { 225 | "cell_type": "code", 226 | "metadata": { 227 | "id": "4-IMl4b6BdGO" 228 | }, 229 | "source": [ 230 | "from object_detection.utils import ops as utils_ops\n", 231 | "from object_detection.utils import label_map_util\n", 232 | "from object_detection.utils import visualization_utils as vis_util" 233 | ], 234 | "execution_count": null, 235 | "outputs": [] 236 | }, 237 | { 238 | "cell_type": "markdown", 239 | "metadata": { 240 | "id": "RYPCiag2iz_q" 241 | }, 242 | "source": [ 243 | "Patches:" 244 | ] 245 | }, 246 | { 247 | "cell_type": "code", 248 | "metadata": { 249 | "id": "mF-YlMl8c_bM" 250 | }, 251 | "source": [ 252 | "# patch tf1 into `utils.ops`\n", 253 | "utils_ops.tf = tf.compat.v1\n", 254 | "\n", 255 | "# Patch the location of gfile\n", 256 | "tf.gfile = tf.io.gfile" 257 | ], 258 | "execution_count": null, 259 | "outputs": [] 260 | }, 261 | { 262 | "cell_type": "code", 263 | "metadata": { 264 | "id": "STlwq2_zuJsX" 265 | }, 266 | "source": [ 267 | "os.chdir('/content/drive/MyDrive/object_detect/shailaj_pro_12july/data/annotation')\n", 268 | "!ls" 269 | ], 270 | "execution_count": null, 271 | "outputs": [] 272 | }, 273 | { 274 | "cell_type": "markdown", 275 | "metadata": { 276 | "id": "0G_H9hAElfFY" 277 | }, 278 | "source": [ 279 | "#Integration and Conversion of Images and Annotation Files into tf_record" 280 | ] 281 | }, 282 | { 283 | "cell_type": "code", 284 | "metadata": { 285 | "id": "dlzvonSHuaJi" 286 | }, 287 | "source": [ 288 | "!python generate_tfrecord.py --csv_input=train_labels.csv --output_path=train.record" 289 | ], 290 | "execution_count": null, 291 | "outputs": [] 292 | }, 293 | { 294 | "cell_type": "code", 295 | "metadata": { 296 | "id": "6E_ulcDpwEIc" 297 | }, 298 | "source": [ 299 | "!python generate_tfrecord.py --csv_input=test_labels.csv --output_path=test.record" 300 | ], 301 | "execution_count": null, 302 | "outputs": [] 303 | }, 304 | { 305 | "cell_type": "code", 306 | "metadata": { 307 | "id": "IsJeDQMUB97i" 308 | }, 309 | "source": [ 310 | "pip install tensorflow-addons" 311 | ], 312 | "execution_count": null, 313 | "outputs": [] 314 | }, 315 | { 316 | "cell_type": "code", 317 | "metadata": { 318 | "id": "Ka5dV_DfCFbq" 319 | }, 320 | "source": [ 321 | "pip install tensorflow-addons[tensorflow]" 322 | ], 323 | "execution_count": null, 324 | "outputs": [] 325 | }, 326 | { 327 | "cell_type": "code", 328 | "metadata": { 329 | "id": "xe7EEipqCLPi" 330 | }, 331 | "source": [ 332 | "import tensorflow_addons as tfa" 333 | ], 334 | "execution_count": null, 335 | "outputs": [] 336 | }, 337 | { 338 | "cell_type": "markdown", 339 | "metadata": { 340 | "id": "UhQss9kEnEt_" 341 | }, 342 | "source": [ 343 | "# Train the model\n", 344 | "Copy TensorFlow/models/research/object_detection/model_main_tf2.py script and paste it straight into our data folder. " 345 | ] 346 | }, 347 | { 348 | "cell_type": "code", 349 | "metadata": { 350 | "id": "lCdeWwXr0Aj7" 351 | }, 352 | "source": [ 353 | "!python '/content/models/research/object_detection/model_main_tf2.py' --model_dir=/content/drive/MyDrive/object_detect/shailaj_pro_1 2july/Model/RCNN1 --pipeline_config_path=/content/drive/MyDrive/object_detect/shailaj_pro_12july/Model/ssd_mobilenet_v2_320x320/SSD_mobilenet_v2_pipeline.config" 354 | ], 355 | "execution_count": null, 356 | "outputs": [] 357 | }, 358 | { 359 | "cell_type": "code", 360 | "metadata": { 361 | "id": "OCt_nGas6ZLk" 362 | }, 363 | "source": [ 364 | "\n", 365 | "%%capture\n", 366 | "!pip install -q tf-nightly-gpu-2.0-preview\n", 367 | "# Load the TensorBoard notebook extension\n", 368 | "# %load_ext tensorboard.notebook # For older versions\n", 369 | "%load_ext tensorboard" 370 | ], 371 | "execution_count": null, 372 | "outputs": [] 373 | }, 374 | { 375 | "cell_type": "code", 376 | "metadata": { 377 | "id": "swfXvxqN6cf7" 378 | }, 379 | "source": [ 380 | "from tensorflow.keras.callbacks import TensorBoard" 381 | ], 382 | "execution_count": null, 383 | "outputs": [] 384 | }, 385 | { 386 | "cell_type": "code", 387 | "metadata": { 388 | "id": "xZEGLK29xJUJ" 389 | }, 390 | "source": [ 391 | "%load_ext tensorboard" 392 | ], 393 | "execution_count": null, 394 | "outputs": [] 395 | }, 396 | { 397 | "cell_type": "code", 398 | "metadata": { 399 | "id": "UaYuHTH3r4_X" 400 | }, 401 | "source": [ 402 | "pwd" 403 | ], 404 | "execution_count": null, 405 | "outputs": [] 406 | }, 407 | { 408 | "cell_type": "code", 409 | "metadata": { 410 | "id": "htSyWkg52-sH" 411 | }, 412 | "source": [ 413 | "# Configure path to 'train folder contains events. file'\n", 414 | "%tensorboard --logdir '/content/drive/MyDrive/object_detect/shailaj_pro_12july/Model/RCNN1/train'" 415 | ], 416 | "execution_count": null, 417 | "outputs": [] 418 | }, 419 | { 420 | "cell_type": "code", 421 | "metadata": { 422 | "id": "td-fBIS5DE6b" 423 | }, 424 | "source": [ 425 | "# Configure path to 'train folder contains events. file'\n", 426 | "%tensorboard --logdir '/content/drive/MyDrive/object_detect/shailaj_pro_12july/Model/RCNN1/eval'" 427 | ], 428 | "execution_count": null, 429 | "outputs": [] 430 | }, 431 | { 432 | "cell_type": "markdown", 433 | "metadata": { 434 | "id": "4EYkRhpl7KfX" 435 | }, 436 | "source": [ 437 | "# Export a trained Model" 438 | ] 439 | }, 440 | { 441 | "cell_type": "code", 442 | "metadata": { 443 | "id": "7-atMEFu7J32" 444 | }, 445 | "source": [ 446 | "!python '/content/models/research/object_detection/exporter_main_v2.py' --input_type image_tensor --pipeline_config_path=/content/drive/MyDrive/object_detect/shailaj_pro_12july/Model/ssd_mobilenet_v2_320x320/SSD_mobilenet_v2_pipeline.config --trained_checkpoint_dir=/content/drive/MyDrive/object_detect/shailaj_pro_12july/Model/RCNN1 --output_directory=/content/drive/MyDrive/object_detect/shailaj_pro_12july/Model/Output/output_inference_graph1.pb\n" 447 | ], 448 | "execution_count": null, 449 | "outputs": [] 450 | }, 451 | { 452 | "cell_type": "markdown", 453 | "metadata": { 454 | "id": "OlowgVEW8OLZ" 455 | }, 456 | "source": [ 457 | "# Evaluating the Model" 458 | ] 459 | }, 460 | { 461 | "cell_type": "code", 462 | "metadata": { 463 | "id": "67Zbq_SV8Nk5" 464 | }, 465 | "source": [ 466 | "!python '/content/models/research/object_detection/model_main_tf2.py' --model_dir=/content/drive/MyDrive/object_detect/shailaj_pro_12july/Model/RCNN1 --pipeline_config_path=/content/drive/MyDrive/object_detect/shailaj_pro_12july/Model/ssd_mobilenet_v2_320x320/SSD_mobilenet_v2_pipeline.config --checkpoint_dir=/content/drive/MyDrive/object_detect/shailaj_pro_12july/Model/RCNN1\n" 467 | ], 468 | "execution_count": null, 469 | "outputs": [] 470 | }, 471 | { 472 | "cell_type": "code", 473 | "metadata": { 474 | "id": "DCYp43zk8Nhg" 475 | }, 476 | "source": [ 477 | "" 478 | ], 479 | "execution_count": null, 480 | "outputs": [] 481 | }, 482 | { 483 | "cell_type": "markdown", 484 | "metadata": { 485 | "id": "cfn_tRFOWKMO" 486 | }, 487 | "source": [ 488 | "# Model preparation " 489 | ] 490 | }, 491 | { 492 | "cell_type": "markdown", 493 | "metadata": { 494 | "id": "X_sEBLpVWKMQ" 495 | }, 496 | "source": [ 497 | "## Variables\n", 498 | "\n", 499 | "Any model exported using the `export_inference_graph.py` tool can be loaded here simply by changing the path.\n", 500 | "I have used an \"SSD with Mobilenet\" model from Tensorflow Model Zoo" 501 | ] 502 | }, 503 | { 504 | "cell_type": "markdown", 505 | "metadata": { 506 | "id": "7ai8pLZZWKMS" 507 | }, 508 | "source": [ 509 | "## Loader" 510 | ] 511 | }, 512 | { 513 | "cell_type": "code", 514 | "metadata": { 515 | "id": "zm8xp-0eoItE" 516 | }, 517 | "source": [ 518 | "def load_model(model_name):\n", 519 | " base_url = 'http://download.tensorflow.org/models/object_detection/'\n", 520 | " model_file = model_name + '.tar.gz'\n", 521 | " model_dir = tf.keras.utils.get_file(\n", 522 | " fname=model_name, \n", 523 | " origin=base_url + model_file,\n", 524 | " untar=True)\n", 525 | "\n", 526 | " model_dir = pathlib.Path(model_dir)/\"saved_model\"\n", 527 | "\n", 528 | " model = tf.saved_model.load(str(model_dir))\n", 529 | "\n", 530 | " return model" 531 | ], 532 | "execution_count": null, 533 | "outputs": [] 534 | }, 535 | { 536 | "cell_type": "markdown", 537 | "metadata": { 538 | "id": "_1MVVTcLWKMW" 539 | }, 540 | "source": [ 541 | "## Loading label map\n", 542 | "Label maps map indices to category names" 543 | ] 544 | }, 545 | { 546 | "cell_type": "code", 547 | "metadata": { 548 | "id": "hDbpHkiWWKMX" 549 | }, 550 | "source": [ 551 | "# List of the strings that is used to add correct label for each box.\n", 552 | "#PATH_TO_LABELS = 'models/research/object_detection/data/mscoco_label_map.pbtxt'\n", 553 | "PATH_TO_LABELS= '/content/drive/MyDrive/object_detect/shailaj_pro_12july/data/annotation/label_map.pbtxt'\n", 554 | "\n", 555 | "category_index = label_map_util.create_category_index_from_labelmap(PATH_TO_LABELS, use_display_name=True)" 556 | ], 557 | "execution_count": null, 558 | "outputs": [] 559 | }, 560 | { 561 | "cell_type": "code", 562 | "metadata": { 563 | "id": "MZvYbApnHw7f" 564 | }, 565 | "source": [ 566 | "category_index" 567 | ], 568 | "execution_count": null, 569 | "outputs": [] 570 | }, 571 | { 572 | "cell_type": "markdown", 573 | "metadata": { 574 | "id": "oVU3U_J6IJVb" 575 | }, 576 | "source": [ 577 | "For the sake of simplicity we will test on 2 images:" 578 | ] 579 | }, 580 | { 581 | "cell_type": "code", 582 | "metadata": { 583 | "id": "jG-zn5ykWKMd" 584 | }, 585 | "source": [ 586 | "# If you want to test the code with your images, just add path to the images to the TEST_IMAGE_PATHS.\n", 587 | "#PATH_TO_TEST_IMAGES_DIR = pathlib.Path('models/research/object_detection/test_images')\n", 588 | "PATH_TO_TEST_IMAGES_DIR = pathlib.Path('/content/')\n", 589 | "TEST_IMAGE_PATHS = sorted(list(PATH_TO_TEST_IMAGES_DIR.glob(\"*.JPG\")))\n", 590 | "#TEST_IMAGE_PATHS = sorted(list('/content/T12.jpg'))\n", 591 | "TEST_IMAGE_PATHS" 592 | ], 593 | "execution_count": null, 594 | "outputs": [] 595 | }, 596 | { 597 | "cell_type": "markdown", 598 | "metadata": { 599 | "id": "H0_1AGhrWKMc" 600 | }, 601 | "source": [ 602 | "# Detection" 603 | ] 604 | }, 605 | { 606 | "cell_type": "markdown", 607 | "metadata": { 608 | "id": "f7aOtOlebK7h" 609 | }, 610 | "source": [ 611 | "Load an object detection model:" 612 | ] 613 | }, 614 | { 615 | "cell_type": "code", 616 | "metadata": { 617 | "id": "1XNT0wxybKR6" 618 | }, 619 | "source": [ 620 | "model_name = '/content/drive/MyDrive/object_detect/shailaj_pro_12july/Model/Output/save_model1/saved_model'\n" 621 | ], 622 | "execution_count": null, 623 | "outputs": [] 624 | }, 625 | { 626 | "cell_type": "code", 627 | "metadata": { 628 | "id": "XmXzT3cqsd4j" 629 | }, 630 | "source": [ 631 | "detection_model = tf.saved_model.load(model_name)" 632 | ], 633 | "execution_count": null, 634 | "outputs": [] 635 | }, 636 | { 637 | "cell_type": "markdown", 638 | "metadata": { 639 | "id": "yN1AYfAEJIGp" 640 | }, 641 | "source": [ 642 | "Check the model's input signature, it expects a batch of 3-color images of type uint8:" 643 | ] 644 | }, 645 | { 646 | "cell_type": "code", 647 | "metadata": { 648 | "id": "CK4cnry6wsHY" 649 | }, 650 | "source": [ 651 | "print(detection_model.signatures['serving_default'].inputs)" 652 | ], 653 | "execution_count": null, 654 | "outputs": [] 655 | }, 656 | { 657 | "cell_type": "markdown", 658 | "metadata": { 659 | "id": "Q8u3BjpMJXZF" 660 | }, 661 | "source": [ 662 | "And returns several outputs:" 663 | ] 664 | }, 665 | { 666 | "cell_type": "code", 667 | "metadata": { 668 | "id": "oLSZpfaYwuSk" 669 | }, 670 | "source": [ 671 | "detection_model.signatures['serving_default'].output_dtypes" 672 | ], 673 | "execution_count": null, 674 | "outputs": [] 675 | }, 676 | { 677 | "cell_type": "code", 678 | "metadata": { 679 | "id": "FZyKUJeuxvpT" 680 | }, 681 | "source": [ 682 | "detection_model.signatures['serving_default'].output_shapes" 683 | ], 684 | "execution_count": null, 685 | "outputs": [] 686 | }, 687 | { 688 | "cell_type": "markdown", 689 | "metadata": { 690 | "id": "JP5qZ7sXJpwG" 691 | }, 692 | "source": [ 693 | "Add a wrapper function to call the model, and cleanup the outputs:" 694 | ] 695 | }, 696 | { 697 | "cell_type": "code", 698 | "metadata": { 699 | "id": "ajmR_exWyN76" 700 | }, 701 | "source": [ 702 | "def run_inference_for_single_image(model, image):\n", 703 | " image = np.asarray(image)\n", 704 | " # The input needs to be a tensor, convert it using `tf.convert_to_tensor`.\n", 705 | " input_tensor = tf.convert_to_tensor(image)\n", 706 | " # The model expects a batch of images, so add an axis with `tf.newaxis`.\n", 707 | " input_tensor = input_tensor[tf.newaxis,...]\n", 708 | "\n", 709 | " # Run inference\n", 710 | " model_fn = model.signatures['serving_default']\n", 711 | " output_dict = model_fn(input_tensor)\n", 712 | "\n", 713 | " # All outputs are batches tensors.\n", 714 | " # Convert to numpy arrays, and take index [0] to remove the batch dimension.\n", 715 | " # We're only interested in the first num_detections.\n", 716 | " num_detections = int(output_dict.pop('num_detections'))\n", 717 | " output_dict = {key:value[0, :num_detections].numpy() \n", 718 | " for key,value in output_dict.items()}\n", 719 | " output_dict['num_detections'] = num_detections\n", 720 | "\n", 721 | " # detection_classes should be ints.\n", 722 | " output_dict['detection_classes'] = output_dict['detection_classes'].astype(np.int64)\n", 723 | " \n", 724 | " # Handle models with masks:\n", 725 | " if 'detection_masks' in output_dict:\n", 726 | " # Reframe the the bbox mask to the image size.\n", 727 | " detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(\n", 728 | " output_dict['detection_masks'], output_dict['detection_boxes'],\n", 729 | " image.shape[0], image.shape[1]) \n", 730 | " detection_masks_reframed = tf.cast(detection_masks_reframed > 0.5,\n", 731 | " tf.uint8)\n", 732 | " output_dict['detection_masks_reframed'] = detection_masks_reframed.numpy()\n", 733 | " \n", 734 | " return output_dict" 735 | ], 736 | "execution_count": null, 737 | "outputs": [] 738 | }, 739 | { 740 | "cell_type": "markdown", 741 | "metadata": { 742 | "id": "z1wq0LVyMRR_" 743 | }, 744 | "source": [ 745 | "Run it on each test image and show the results:" 746 | ] 747 | }, 748 | { 749 | "cell_type": "code", 750 | "metadata": { 751 | "id": "DWh_1zz6aqxs" 752 | }, 753 | "source": [ 754 | "def show_inference(model, image_path):\n", 755 | " # the array based representation of the image will be used later in order to prepare the\n", 756 | " # result image with boxes and labels on it.\n", 757 | " image_np = np.array(Image.open(image_path))\n", 758 | " # Actual detection.\n", 759 | " output_dict = run_inference_for_single_image(model, image_np)\n", 760 | " # Visualization of the results of a detection.\n", 761 | " vis_util.visualize_boxes_and_labels_on_image_array(\n", 762 | " image_np,\n", 763 | " output_dict['detection_boxes'],\n", 764 | " output_dict['detection_classes'],\n", 765 | " output_dict['detection_scores'],\n", 766 | " category_index,\n", 767 | " instance_masks=output_dict.get('detection_masks_reframed', None),\n", 768 | " use_normalized_coordinates=True,\n", 769 | " line_thickness=8)\n", 770 | "\n", 771 | " display(Image.fromarray(image_np))" 772 | ], 773 | "execution_count": null, 774 | "outputs": [] 775 | }, 776 | { 777 | "cell_type": "code", 778 | "metadata": { 779 | "id": "3a5wMHN8WKMh" 780 | }, 781 | "source": [ 782 | "\n", 783 | "for image_path in TEST_IMAGE_PATHS:\n", 784 | " show_inference(detection_model, image_path)\n" 785 | ], 786 | "execution_count": null, 787 | "outputs": [] 788 | }, 789 | { 790 | "cell_type": "markdown", 791 | "metadata": { 792 | "id": "vdXJz88o6tsP" 793 | }, 794 | "source": [ 795 | "" 796 | ] 797 | } 798 | ] 799 | } 800 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Process-Symbol-detection 2 | 3 | A piping and instrument diagrams (P&IDs) are a key component of the process industry; they contain information about the plant, including the instruments, lines, valves, and control logic. However, the complexity of these diagrams makes it difficult to extract the information automatically. Although used universally across industries such as manufacturing and oil & gas, P&IDs are usually trapped in image files with limited metadata, making their contents unsearchable and siloed from operational or enterprise systems. In this project I have used computer vision object detection technique to detect symbols in this diagram. 4 | 5 | The demand for this project enable an automatic engineering diagram digitalization and improve productivity and gain a competitive edge for the company in the global market. 6 | 7 | I have trained Object detection model by retraining final layer of SSD MobileNet v2 320x320 pretrained model. 8 | 9 | **P&ID diagram** 10 | 11 | ![image](https://user-images.githubusercontent.com/49098763/125462171-8cec37ff-33ae-4041-8775-815b51075b66.png) 12 | 13 |

Dynamic symbols

14 | To detect dynamic symbol like Pump, motorised valve, control valve using Object detection technique 15 | 16 | **Pump** 17 | 18 | ![image](https://user-images.githubusercontent.com/49098763/125462394-583ab407-c6f5-4f6c-a4fa-e677bf970ce1.png) 19 | 20 | **Motorised Valve** 21 | 22 | ![image](https://user-images.githubusercontent.com/49098763/125462458-5d03307b-416b-4bff-af4c-abcb3e2e5cdb.png) 23 | 24 | **Control Valve** 25 | 26 | ![image](https://user-images.githubusercontent.com/49098763/125462480-d3b3a2e7-f121-403b-9235-d8700c34e349.png) 27 | 28 | Optional Additional Software: 29 | 34 | 35 | 36 |

Training Preparation

37 | 38 |

Conversion from PDF

39 | 40 | For my use case, the images to be processed were 11 x 17 size pages in a PDF. I extracted and converted each page from the PDF into an invidual jpg file. Segmentation of the file was done using code similar to the segmentation block in detection application script. 41 | 42 |

Image Segmentation into Smaller Sizes

43 | 44 | The Google MobileNet V2 model is trained on images of size 299 by 299 so I had to segment the images into smaller sections that were 200 by 200. This size gave me a good ratio of segment image dimensions to target object bounding box dimension. In other words, the bounding box for the object to be detected was not too small or too large compared to the overall segment image dimensions. To handle situations where the target object to be detected might be split between two different segments, I overlapped the segments by 20 pixels on each side. 45 | 46 |

Training Data Annotation

47 | 48 | After saving the image segments, I used labelImg to draw bounding boxes. 49 | 50 |

Conversion of Annotation from XML to Text

51 | 52 | lableImg saves the bounding box annotation data in COCO format as XML. I then used a modified version of the conversion script from Guanghan to convert the individual annotation file for each trainiing image into a single CSV master file that had one line per training image containing the image file name and data on the class(es) and bounding box(es). The data folder in my repo contains the image files, the xml annotation from labelIMG and the combined csv text file describing annotations for the all the images used for training and testing. The script for combined and convert xml annotation is xml_to_csv.py. It should be run in the same directory as where all the xml files from labelIMG are stored. Line 31 should be modified to change the file name for the csv file for your own project. 53 | 54 |

Manual Split of Annotation Text Files into Train and Test

55 | 56 | After the python script generated the csv file, I then used a spreadsheet to split the records into two files - "train_labels.csv" and "test_labels.csv". It was pure cut and paste operation. No data was edited in generating the two files. I aimed for about 10% of the records for testing. There is probably a way to automate this in Tensorflow to have random splits for each training step but I did not implement this for my project. Modifying the code in this example for random split of test and train would deliver better results and reduce the over-fitting. In this case, over-fitting and a high number of false positives was not a concern in the production and actual use environment. 57 | 58 |

Creation of label map file

59 | 60 | A simple json file is also needed to tell the class names for the bounding boxes in the training data. So I created the label_map.pbtxt inside my input folder directory file containing the following text.
61 | ``` 62 | { 63 | id: 1 64 | name: 'psv' 65 | } 66 | ``` 67 | The id for the first object class in the map must start at 1 and not 0 68 | 69 |

Integration and Conversion of Images and Annotation Files into tf_record

70 | 71 | The generate_tfrecord.py script was used to convert the images and the csv files into tf record expected for tensorflow. On Linux, the following commands were run from the data directory:
72 | ```shell 73 | python3.6 generate_tfrecord.py --csv_input=train_labels.csv --output_path=train.record 74 | 75 | python3.6 generate_tfrecord.py --csv_input=test_labels.csv --output_path=test.record 76 | ``` 77 | 78 | The files train.record and test.record are created from the two commands above. 79 | 80 |

Downloading the Pre-Trained Model

81 | 82 | The next step is to download the pre-trained model from the Tensorflow detection model zoo. In this case, I used the model ssd_mobilenet_v2_coco_17_tpu8 model. The tarball should be moved into the models directory and exracted with the following command.
83 | ```shell 84 | tar -xvf ssd_mobilenet_v2_coco_17_tpu8.tar.gz 85 | ``` 86 | 87 | Finally, the training pipeline configuration file is customized. There are four customizations needed in this file: 88 |
    89 |
  1. Update the number of clases on line 9
    2. 90 |
  2. Update the file location path to the train and test record files in lines 171 and 185 respectively
    3. 91 |
  3. Update the file location path to the class label map created earlier in line 173 and 187.
    4. 92 |
  4. Update the file location path to the pre-trained model in line 152.
    5. 93 |
94 | 95 |

Using Other Models

96 | You can also download and trainother pre-trained models for final layer training as long as you update and use the matching config file from the tensorflow models repository and specifically the scripts in the object detection folder. The repo should be cloned to your local directory and the script setup.py in the research folder executed using pip before starting the training.
101 | 102 | ```shell 103 | python3.6 pip -m setup.py install 104 | ``` can 105 | 106 | Installing the scripts as well as tensorflow on the Ubuntu machine that I used gave some unique installation errors and headaches that I had to research and resolve. You will likely face some errors and may have to do the same. If you get an error about "no module named 'depolyment'". Execute the following command from the models subirectory in the repo you downloaded above. 107 | 108 | ```shell 109 | export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim 110 | ``` 111 | 112 | The script to train the model is trainer.py and it should be executed using python from the models directory. 113 | 114 | ```shell 115 | python3.6 object_detection/train.py --logtostderr --pipeline_config_path= --train_dir= 116 | ``` 117 | 118 | Once the training script starts, you will see the training time per step as well as the loss. The loss should decrease overall but could go up a little between adjacent training iterations. The configuration file is set to train for 500 steps but you can stop the training at anytime using the interrupt key combinations. 119 | 120 | 121 |

Saving the Retrained Network Model

122 | When the training stops after reaching the training iteration limits or interrupted manually, the network model up to the last saved checkpoint can be obtained using the export_inference_graph.py script as follows: 123 | 124 | ```shell 125 | python3.6 object_detection/export_inference_graph.py --input_type image_tensor --pipeline_config_path --trained_checkpoint_prefix --output_directory output_inference_graph.pb 126 | ``` 127 | 128 | The command above will generat an output_inference_grap.pb file with the neural network model weights using the last available checkpoint saved data from training. The output_inference_grap.pb file is used in the actual object detection. 129 | 130 | 131 |

Tensorboard Visualization

132 | 133 | ![image](https://user-images.githubusercontent.com/49098763/125624978-6f0a0623-a599-4e54-b9d3-686e8c028fee.png) 134 | 135 | 136 |

Model Evaluation

137 | 138 | ![image](https://user-images.githubusercontent.com/49098763/125624844-269ec57d-ca29-4f58-b05d-e1779a36c033.png) 139 | 140 | ![image](https://user-images.githubusercontent.com/49098763/125625041-c9760c2d-8604-4c9b-8389-5b56f54582a1.png) 141 | 142 |

Model Output

143 | 144 | ![image](https://user-images.githubusercontent.com/49098763/125625133-dd542178-2d86-4b0f-abc1-d67b4495fa4a.png) 145 | 146 | 147 | 148 | 149 |

References

150 | https://towardsdatascience.com/how-to-train-your-own-object-detector-with-tensorflows-object-detector-api-bec72ecfe1d9 151 | https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html 152 | https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md 153 | 154 | 155 | --------------------------------------------------------------------------------