├── LICENSE ├── Notebooks ├── Train_BigTransfer_on_Flower_Dataset.ipynb └── Train_KNN.ipynb ├── README.md ├── Scripts ├── train_bigtransfer_on_flower_dataset.py └── train_knn.py ├── data ├── Indentical Image Retrieval.drawio ├── proposed_architecture.png ├── sample_dataset_one.png └── sample_dataset_two.png ├── graphs ├── scatter_plot_1.png ├── scatter_plot_2.png └── train_graph.png ├── logs └── train_bit.csv └── result ├── ouput_four.png ├── ouput_three.png ├── outout_one.png ├── output_five.png ├── output_six.png ├── output_two.png └── result.png /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright 2021 Sayan Nath 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Identical-Image-Retrieval 2 | 3 | ## Abstract 4 | In recent years, we know that the interaction with images has increased. Image similarity involves fetching similar-looking images abiding by a given reference image. The target is to find out whether the image searched as a query can result in similar pictures. We are using the BigTransfer Model, which is a state-of-art model itself. BigTransfer(BiT) is essentially a ResNet but pre-trained on a larger dataset like ImageNet and ImageNet-21k with additional modifications. Using the fine-tuned pre-trained Convolution Neural Network Model, we extract the key features and train on the K- Nearest Neighbor model to obtain the nearest neighbor. The application of our model is to find similar images, which are hard to achieve through text queries within a low inference time. We analyse the benchmark of our model based on this application. 5 | 6 | ## Description 7 | This project presents a simple framework to retrieve images similar to a query image using Deep Learning. The framework is as follows: 8 | 9 | * Train a CNN model (A) on a set of labeled images with Triplet Loss (I used this one). 10 | * Use the trained CNN model (A) to extract features from the validation set. 11 | * Train a kNN model (B) on these extracted features with k set to the number of neighbors wanted. 12 | * Grab an image (I) from the validation set and extract its features using the same CNN model (A). 13 | * Use the same kNN model (B) to calculate the nearest neighbors of I. 14 | 15 | 16 | 17 | I experimented with the Flower Dataset. 18 | 19 | 20 | 21 | ## Model Used 22 | 23 | I fine-tuned pre-trained models for minimizing the Triplet Loss. I experimented with the following pre-trained models: 24 | 25 | * BigTransfer Model (also referred to as BiT) which is essentially a ResNet but pre-trained on a larger dataset with additional modifications. 26 | 27 | ### Train Graph 28 | 29 | 30 | ### Visualization of the embedding space 31 | 32 | 33 | 34 | 35 | ## Results 36 | 37 | 38 | -------------------------------------------------------------------------------- /Scripts/train_bigtransfer_on_flower_dataset.py: -------------------------------------------------------------------------------- 1 | """ 2 | # Train BigTransfer on Flower Dataset 3 | 4 | ## Initial-Setup 5 | """ 6 | 7 | # !nvidia-smi 8 | 9 | # !pip install -q tensorflow-addons 10 | 11 | """## Import the necessary modules""" 12 | 13 | import os 14 | import time 15 | import numpy as np 16 | import pandas as pd 17 | import matplotlib.pyplot as plt 18 | 19 | import tensorflow as tf 20 | from tensorflow.keras.layers import * 21 | from tensorflow.keras.models import * 22 | from tensorflow.keras.applications import * 23 | import tensorflow_hub as hub 24 | import tensorflow_addons as tfa 25 | 26 | import tensorflow_datasets as tfds 27 | 28 | tfds.disable_progress_bar() 29 | 30 | # Fix the random seeds 31 | SEEDS = 666 32 | 33 | np.random.seed(SEEDS) 34 | tf.random.set_seed(SEEDS) 35 | 36 | """## Data Gathering 37 | 38 | Importing the Flower Dataset 39 | """ 40 | 41 | # Gather Flowers dataset 42 | train_ds, validation_ds = tfds.load( 43 | "tf_flowers", 44 | split=["train[:85%]", "train[85%:]"], 45 | as_supervised=True, 46 | ) 47 | 48 | """#### Define the class""" 49 | 50 | CLASSES = ["daisy", "dandelion", "roses", "sunflowers", "tulips"] 51 | 52 | """#### Count of Training and Validation Samples""" 53 | 54 | print("Number of Training Samples: ", len(train_ds)) 55 | print("Number of Validation Samples: ", len(validation_ds)) 56 | 57 | """### Visualise the Dataset""" 58 | 59 | plt.figure(figsize=(10, 10)) 60 | for i, (image, label) in enumerate(train_ds.take(9)): 61 | ax = plt.subplot(3, 3, i + 1) 62 | plt.imshow(image) 63 | plt.title(CLASSES[int(label)]) 64 | plt.axis("off") 65 | 66 | """## Training setup""" 67 | 68 | # @title Set dataset-dependent hyperparameters 69 | 70 | IMAGE_SIZE = "=\u003C96x96 px" # @param ["=<96x96 px","> 96 x 96 px"] 71 | DATASET_SIZE = "\u003C20k examples" # @param ["<20k examples", "20k-500k examples", ">500k examples"] 72 | 73 | if IMAGE_SIZE == "=<96x96 px": 74 | RESIZE_TO = 160 75 | CROP_TO = 128 76 | else: 77 | RESIZE_TO = 512 78 | CROP_TO = 480 79 | 80 | if DATASET_SIZE == "<20k examples": 81 | SCHEDULE_LENGTH = 500 82 | SCHEDULE_BOUNDARIES = [200, 300, 400] 83 | elif DATASET_SIZE == "20k-500k examples": 84 | SCHEDULE_LENGTH = 10000 85 | SCHEDULE_BOUNDARIES = [3000, 6000, 9000] 86 | else: 87 | SCHEDULE_LENGTH = 20000 88 | SCHEDULE_BOUNDARIES = [6000, 12000, 18000] 89 | 90 | """## Define the Hyperparameters""" 91 | 92 | BATCH_SIZE = 64 93 | NUM_CLASSES = 5 94 | SCHEDULE_LENGTH = SCHEDULE_LENGTH * 512 / BATCH_SIZE 95 | STEPS_PER_EPOCH = 10 96 | DATASET_NUM_TRAIN_EXAMPLES = len([image for image in train_ds]) 97 | AUTO = tf.data.AUTOTUNE 98 | CSV_PATH = "train_bit.csv" 99 | 100 | """## Dataloader Function""" 101 | 102 | 103 | def preprocess_train(image, label): 104 | image = tf.image.random_flip_left_right(image) 105 | image = tf.image.resize(image, [RESIZE_TO, RESIZE_TO]) 106 | image = tf.image.random_crop(image, [CROP_TO, CROP_TO, 3]) 107 | image = tf.cast(image, tf.float32) / 255.0 108 | return image, label 109 | 110 | 111 | def preprocess_test(image, label): 112 | image = tf.image.resize(image, [RESIZE_TO, RESIZE_TO]) 113 | image = tf.cast(image, tf.float32) / 255.0 114 | return image, label 115 | 116 | 117 | """## Create the Data Pipeline""" 118 | 119 | pipeline_train = ( 120 | train_ds.shuffle(10000) 121 | .repeat( 122 | int(SCHEDULE_LENGTH * BATCH_SIZE / DATASET_NUM_TRAIN_EXAMPLES * STEPS_PER_EPOCH) 123 | + 1 124 | + 50 125 | ) 126 | .map(preprocess_train, num_parallel_calls=AUTO) 127 | .batch(BATCH_SIZE) 128 | .prefetch(AUTO) 129 | ) 130 | 131 | pipeline_test = ( 132 | validation_ds.map(preprocess_test, num_parallel_calls=AUTO) 133 | .batch(BATCH_SIZE) 134 | .prefetch(AUTO) 135 | ) 136 | 137 | """## Visualise the dataset""" 138 | 139 | image_batch, label_batch = next(iter(pipeline_train)) 140 | 141 | plt.figure(figsize=(10, 10)) 142 | for n in range(25): 143 | ax = plt.subplot(5, 5, n + 1) 144 | plt.imshow(image_batch[n]) 145 | plt.title(CLASSES[label_batch[n].numpy()]) 146 | plt.axis("off") 147 | 148 | """## Load model into KerasLayer""" 149 | 150 | model_url = "https://tfhub.dev/google/bit/m-r50x1/1" 151 | module = hub.KerasLayer(model_url, trainable=True) 152 | 153 | """## BiT Model""" 154 | 155 | 156 | class MyBiTModel(tf.keras.Model): 157 | def __init__(self, module): 158 | super().__init__() 159 | self.dense1 = tf.keras.layers.Dense(128) 160 | self.normalize = Lambda(lambda a: tf.math.l2_normalize(a, axis=1)) 161 | self.bit_model = module 162 | 163 | def call(self, images): 164 | bit_embedding = self.bit_model(images) 165 | dense1_representations = self.dense1(bit_embedding) 166 | return self.normalize(dense1_representations) 167 | 168 | 169 | model = MyBiTModel(module=module) 170 | 171 | """## Define the optimiser and loss""" 172 | 173 | lr = 0.003 * BATCH_SIZE / 512 174 | 175 | # Decay learning rate by a factor of 10 at SCHEDULE_BOUNDARIES. 176 | lr_schedule = tf.keras.optimizers.schedules.PiecewiseConstantDecay( 177 | boundaries=SCHEDULE_BOUNDARIES, values=[lr, lr * 0.1, lr * 0.001, lr * 0.0001] 178 | ) 179 | 180 | optimizer = tf.keras.optimizers.SGD(learning_rate=lr_schedule, momentum=0.9) 181 | 182 | loss_fn = tfa.losses.TripletSemiHardLoss() 183 | 184 | """### Compile the Model""" 185 | 186 | model.compile(optimizer=optimizer, loss=loss_fn) 187 | 188 | """## Setting up Callback""" 189 | 190 | train_callbacks = [ 191 | tf.keras.callbacks.EarlyStopping( 192 | monitor="val_loss", 193 | patience=5, 194 | verbose=2, 195 | mode="auto", 196 | restore_best_weights=True, 197 | ), 198 | tf.keras.callbacks.CSVLogger(CSV_PATH), 199 | ] 200 | 201 | """## Plot the results""" 202 | 203 | 204 | def plot_training(H, embedding_dim): 205 | with plt.xkcd(): 206 | plt.plot(H.history["loss"], label="train_loss") 207 | plt.plot(H.history["val_loss"], label="val_loss") 208 | plt.title("Embedding dim: {}".format(embedding_dim)) 209 | plt.legend(loc="lower left") 210 | plt.show() 211 | 212 | 213 | """## Train the `BiT` Model""" 214 | 215 | print("Training started!") 216 | 217 | start = time.time() 218 | history = model.fit( 219 | pipeline_train, 220 | batch_size=BATCH_SIZE, 221 | steps_per_epoch=STEPS_PER_EPOCH, 222 | epochs=int(SCHEDULE_LENGTH / STEPS_PER_EPOCH), 223 | validation_data=pipeline_test, 224 | callbacks=train_callbacks, 225 | ) 226 | 227 | end = time.time() - start 228 | print("Model takes {} seconds to train".format(end)) 229 | 230 | plot_training(history, 128) 231 | 232 | """## Save the `BiT` model""" 233 | 234 | KERAS_FILE = "model_bit.h5" 235 | model.save_weights(KERAS_FILE) 236 | 237 | # !cp -r '/content/model_bit.h5' '/content/drive/MyDrive/Similar-Image-Search/' 238 | 239 | # !cp -r '/content/train_bit.csv' '/content/drive/MyDrive/Similar-Image-Search/' 240 | 241 | """Saved the Model""" -------------------------------------------------------------------------------- /Scripts/train_knn.py: -------------------------------------------------------------------------------- 1 | """ 2 | # Train KNN 3 | 4 | ## Initial-Setup 5 | """ 6 | 7 | # !nvidia-smi 8 | 9 | # !cp -r '/content/drive/MyDrive/Similar-Image-Search/model_bit.h5' '/content/' 10 | 11 | """## Import the necessary modules""" 12 | 13 | import os 14 | import time 15 | import numpy as np 16 | import pandas as pd 17 | import matplotlib.pyplot as plt 18 | 19 | from sklearn.neighbors import NearestNeighbors 20 | from sklearn.manifold import TSNE 21 | 22 | import tensorflow as tf 23 | from tensorflow.keras.layers import * 24 | from tensorflow.keras.models import * 25 | from tensorflow.keras.applications import * 26 | import tensorflow_hub as hub 27 | 28 | import tensorflow_datasets as tfds 29 | 30 | tfds.disable_progress_bar() 31 | 32 | # Fix the random seeds 33 | SEEDS = 666 34 | 35 | np.random.seed(SEEDS) 36 | tf.random.set_seed(SEEDS) 37 | 38 | """## Data Gathering 39 | 40 | Importing the Flower Dataset 41 | """ 42 | 43 | # Gather Flowers dataset 44 | train_ds, validation_ds = tfds.load( 45 | "tf_flowers", 46 | split=["train[:85%]", "train[85%:]"], 47 | as_supervised=True, 48 | ) 49 | 50 | """#### Define the class""" 51 | 52 | CLASSES = ["daisy", "dandelion", "roses", "sunflowers", "tulips"] 53 | 54 | """#### Count of Training and Validation Samples""" 55 | 56 | print("Number of Training Samples: ", len(train_ds)) 57 | print("Number of Validation Samples: ", len(validation_ds)) 58 | 59 | """### Visualise the Dataset""" 60 | 61 | plt.figure(figsize=(10, 10)) 62 | for i, (image, label) in enumerate(train_ds.take(9)): 63 | ax = plt.subplot(3, 3, i + 1) 64 | plt.imshow(image) 65 | plt.title(CLASSES[int(label)]) 66 | plt.axis("off") 67 | 68 | """## Define the Hyperparameters""" 69 | 70 | IMAGE_SIZE = 160 71 | BATCH_SIZE = 64 72 | AUTO = tf.data.AUTOTUNE 73 | 74 | """## Dataloader""" 75 | 76 | # Image preprocessing utils 77 | def preprocess_test(image, label): 78 | image = tf.image.resize(image, [IMAGE_SIZE, IMAGE_SIZE]) 79 | image = tf.cast(image, tf.float32) / 255.0 80 | return image, label 81 | 82 | 83 | """## Creating the pipeline for validation sample""" 84 | 85 | validation_ds = ( 86 | validation_ds.map(preprocess_test, num_parallel_calls=AUTO) 87 | .batch(BATCH_SIZE) 88 | .prefetch(buffer_size=AUTO) 89 | ) 90 | 91 | """## Load model into KerasLayer""" 92 | 93 | model_url = "https://tfhub.dev/google/bit/m-r50x1/1" 94 | module = hub.KerasLayer(model_url, trainable=False) 95 | 96 | """## BiT Model""" 97 | 98 | 99 | class MyBiTModel(tf.keras.Model): 100 | def __init__(self, module): 101 | super().__init__() 102 | self.dense1 = tf.keras.layers.Dense(128) 103 | self.normalize = Lambda(lambda a: tf.math.l2_normalize(a, axis=1)) 104 | self.bit_model = module 105 | 106 | def call(self, images): 107 | bit_embedding = self.bit_model(images) 108 | dense1_representations = self.dense1(bit_embedding) 109 | return self.normalize(dense1_representations) 110 | 111 | 112 | model = MyBiTModel(module=module) 113 | 114 | """## Load the weights of the trained BiT Model""" 115 | 116 | model.build(input_shape=(None, IMAGE_SIZE, IMAGE_SIZE, 3)) 117 | model.load_weights("model_bit.h5") 118 | 119 | """### Checking the Validation Pipeline""" 120 | 121 | images, labels = next(iter(validation_ds.take(1))) 122 | print(images.shape, labels.shape) 123 | 124 | random_index = int(np.random.choice(images.shape[0], 1)) 125 | plt.imshow(images[random_index]) 126 | plt.show() 127 | 128 | """## Train a Nearest Neighbors' Model 129 | 130 | Determining out nearest neighbors for the features of our query image 131 | """ 132 | 133 | validation_features = model.predict(images) 134 | start = time.time() 135 | neighbors = NearestNeighbors(n_neighbors=5, algorithm="brute", metric="euclidean").fit( 136 | validation_features 137 | ) 138 | print("Time taken: {:.5f} secs".format(time.time() - start)) 139 | 140 | """### Determine the neighbors nearest to our query image""" 141 | 142 | distances, indices = neighbors.kneighbors([validation_features[random_index]]) 143 | for i in range(5): 144 | print(distances[0][i]) 145 | 146 | """### Visualize a neighbor""" 147 | 148 | plt.imshow(images[indices[0][1]], interpolation="lanczos") 149 | plt.show() 150 | 151 | """## Visualizing the nearest neighbors on images""" 152 | 153 | 154 | def plot_images(images, labels, distances): 155 | plt.figure(figsize=(20, 10)) 156 | columns = 4 157 | for (i, image) in enumerate(images): 158 | ax = plt.subplot(len(images) / columns + 1, columns, i + 1) 159 | if i == 0: 160 | ax.set_title("Query Image\n" + "Label: {}".format(CLASSES[labels[i]])) 161 | else: 162 | ax.set_title( 163 | "Similar Image # " 164 | + str(i) 165 | + "\nDistance: " 166 | + str(float("{0:.2f}".format(distances[i]))) 167 | + "\nLabel: {}".format(CLASSES[labels[i]]) 168 | ) 169 | plt.imshow(image) 170 | 171 | 172 | for i in range(6): 173 | random_index = int(np.random.choice(images.shape[0], 1)) 174 | distances, indices = neighbors.kneighbors([validation_features[random_index]]) 175 | 176 | # Don't take the first closest image as it will be the same image 177 | similar_images = [images[random_index]] + [ 178 | images[indices[0][i]] for i in range(1, 4) 179 | ] 180 | similar_labels = [labels[random_index]] + [ 181 | labels[indices[0][i]] for i in range(1, 4) 182 | ] 183 | plot_images(similar_images, similar_labels, distances[0]) 184 | 185 | """## Visualizing the embedding space for the current validation batch""" 186 | 187 | tsne_results = TSNE(n_components=2).fit_transform(validation_features) 188 | 189 | color_map = plt.cm.get_cmap("coolwarm") 190 | scatter_plot = plt.scatter( 191 | tsne_results[:, 0], tsne_results[:, 1], c=labels, cmap=color_map 192 | ) 193 | plt.colorbar(scatter_plot) 194 | plt.show() 195 | 196 | """## Visualizing the embedding space for the entire validation pipeline""" 197 | 198 | validation_labels = [label for _, labels in validation_ds for label in labels] 199 | print(len(validation_labels)) 200 | 201 | validation_features = model.predict(validation_ds) 202 | 203 | tsne_results = TSNE(n_components=2).fit_transform(validation_features) 204 | 205 | color_map = plt.cm.get_cmap("coolwarm") 206 | scatter_plot = plt.scatter( 207 | tsne_results[:, 0], tsne_results[:, 1], c=validation_labels, cmap=color_map 208 | ) 209 | plt.colorbar(scatter_plot) 210 | plt.show() 211 | -------------------------------------------------------------------------------- /data/proposed_architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sayannath/Identical-Image-Retrieval/cd7d280ac0b874f62a09646065db31f9d6677420/data/proposed_architecture.png -------------------------------------------------------------------------------- /data/sample_dataset_one.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sayannath/Identical-Image-Retrieval/cd7d280ac0b874f62a09646065db31f9d6677420/data/sample_dataset_one.png -------------------------------------------------------------------------------- /data/sample_dataset_two.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sayannath/Identical-Image-Retrieval/cd7d280ac0b874f62a09646065db31f9d6677420/data/sample_dataset_two.png -------------------------------------------------------------------------------- /graphs/scatter_plot_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sayannath/Identical-Image-Retrieval/cd7d280ac0b874f62a09646065db31f9d6677420/graphs/scatter_plot_1.png -------------------------------------------------------------------------------- /graphs/scatter_plot_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sayannath/Identical-Image-Retrieval/cd7d280ac0b874f62a09646065db31f9d6677420/graphs/scatter_plot_2.png -------------------------------------------------------------------------------- /graphs/train_graph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sayannath/Identical-Image-Retrieval/cd7d280ac0b874f62a09646065db31f9d6677420/graphs/train_graph.png -------------------------------------------------------------------------------- /logs/train_bit.csv: -------------------------------------------------------------------------------- 1 | epoch,loss,val_loss 2 | 0,0.9410563707351685,0.9188923835754395 3 | 1,0.9183816909790039,0.8837606310844421 4 | 2,0.8732835650444031,0.8407838940620422 5 | 3,0.8290039300918579,0.7836573719978333 6 | 4,0.7875127792358398,0.7251712679862976 7 | 5,0.7308125495910645,0.6643760204315186 8 | 6,0.6452625393867493,0.5985415577888489 9 | 7,0.5789654850959778,0.5468040704727173 10 | 8,0.5538362264633179,0.49445679783821106 11 | 9,0.48956769704818726,0.46749311685562134 12 | 10,0.43246206641197205,0.42672187089920044 13 | 11,0.40241581201553345,0.42068928480148315 14 | 12,0.425405889749527,0.4166845679283142 15 | 13,0.4003874659538269,0.39071059226989746 16 | 14,0.334881991147995,0.37016475200653076 17 | 15,0.36466914415359497,0.39089760184288025 18 | 16,0.2904554605484009,0.3555121421813965 19 | 17,0.3095793128013611,0.30122634768486023 20 | 18,0.27950116991996765,0.2966121733188629 21 | 19,0.33982399106025696,0.3077373802661896 22 | 20,0.23142686486244202,0.30521172285079956 23 | -------------------------------------------------------------------------------- /result/ouput_four.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sayannath/Identical-Image-Retrieval/cd7d280ac0b874f62a09646065db31f9d6677420/result/ouput_four.png -------------------------------------------------------------------------------- /result/ouput_three.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sayannath/Identical-Image-Retrieval/cd7d280ac0b874f62a09646065db31f9d6677420/result/ouput_three.png -------------------------------------------------------------------------------- /result/outout_one.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sayannath/Identical-Image-Retrieval/cd7d280ac0b874f62a09646065db31f9d6677420/result/outout_one.png -------------------------------------------------------------------------------- /result/output_five.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sayannath/Identical-Image-Retrieval/cd7d280ac0b874f62a09646065db31f9d6677420/result/output_five.png -------------------------------------------------------------------------------- /result/output_six.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sayannath/Identical-Image-Retrieval/cd7d280ac0b874f62a09646065db31f9d6677420/result/output_six.png -------------------------------------------------------------------------------- /result/output_two.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sayannath/Identical-Image-Retrieval/cd7d280ac0b874f62a09646065db31f9d6677420/result/output_two.png -------------------------------------------------------------------------------- /result/result.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sayannath/Identical-Image-Retrieval/cd7d280ac0b874f62a09646065db31f9d6677420/result/result.png --------------------------------------------------------------------------------