├── README.md ├── __init__.py ├── lib ├── __init__.py ├── __init__.pyc ├── neighbor_list_recom.pickle ├── search.py └── search.pyc └── server ├── image_vectorizer.py ├── rest-server.py ├── static ├── images │ ├── Predict_GREY.png │ ├── Train_GREY.png │ ├── ajax-loader.gif │ ├── predict.png │ ├── result1.jpg │ ├── result2.jpg │ ├── search.png │ └── train.png └── result │ ├── image_20171204154121.0.jpg │ ├── image_20171204154121.1.jpg │ ├── image_20171204154121.2.jpg │ ├── image_20171204154121.3.jpg │ ├── image_20171204154121.4.jpg │ ├── image_20171204154121.5.jpg │ ├── image_20171204154121.6.jpg │ ├── image_20171204154121.7.jpg │ └── image_20171204154121.8.jpg ├── templates └── recommend.html └── uploads └── 101026.3.jpg /README.md: -------------------------------------------------------------------------------- 1 | # Image similarity search using deep learning 2 | This project demonstrates, how we can make use of deep learning to do state-of-the-art image similarity search. I have used tensorflow and some publicly available datasets. 3 | 4 | ## Results 5 | ![alt text](https://cdn-images-1.medium.com/max/800/1*dd0QDhLZAjBKD5JZgJCI9w.png) 6 | ![alt text](https://cdn-images-1.medium.com/max/800/0*ziXWbDIrdW_qJMCL.png) 7 | ![alt text](https://cdn-images-1.medium.com/max/800/0*gL9UZQFdwFk-9smY.png) 8 | 9 | ## How to run 10 | 1. Download [imagenet](https://drive.google.com/open?id=1UOyZ8166qM3SzxGvaUeWpBzSUfoQLUjJ) folder, extraxt and keep it in server directory 11 | 2. Download datasets for [footwares](http://vision.cs.utexas.edu/projects/finegrained/utzap50k/), [apparels](http://mmlab.ie.cuhk.edu.hk/projects/DeepFashion/InShopRetrieval.html) keep them inside a directory under upload folder. Final folder strcture will be as below 12 | ``` 13 | root folder 14 | │ 15 | └───lib 16 | │ 17 | └───server 18 | | │───rest-server.py 19 | | │───imagenet 20 | | │───static 21 | | │───templates 22 | | │───uploads 23 | | │────dogs_and_cats 24 | | │────shoes 25 | | │────sandals 26 | | │────slippers 27 | | │────boots 28 | | │────apparels 29 | ``` 30 | 3. Run image vectorizer which passes each data through an inception-v3 model and collects the bottleneck layer vectors and stores in disc. Edit dataset paths accordingly indide the image_vectorizer.py 31 | ``` 32 | python server/image_vectorizer.py 33 | ``` 34 | This will generate two files namely, image_list.pickle and saved_features.txt. Keep them inside lib folder where search.py script is available. 35 | 36 | 4. Start the server by running rest-server.py. This project uses flask based REST implementation for UI 37 | ``` 38 | python server/rest-server.py 39 | ``` 40 | 5. Once the server starts up, access the url 127.0.0.1:5000 to get the UI. Now upload any file and see 9 similar images. You can change the value of K from 9 to any values, but dont foreget to update the html file accordingly for displaying. 41 | 42 | One interesting application of this project is a recommendation engine based on image features.Here is an example of similar project of mine. Here instead of a web UI i have used an android UI. Once the user clicks a product image, the image will go to the server and k-number of similar product images can be displayed on UI as product recommendations. Theses rescommendations are purely based on image similarity. This kind of recommendations have high potentials in fashion-based ecommerce industry. 43 | 44 | ![Example Results](server/static/images/result1.jpg) 45 | ![Example Results](server/static/images/result2.jpg) 46 | 47 | -------------------------------------------------------------------------------- /__init__.py: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /lib/__init__.py: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /lib/__init__.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/lib/__init__.pyc -------------------------------------------------------------------------------- /lib/search.py: -------------------------------------------------------------------------------- 1 | ################################################################################################################################ 2 | # This function implements the image search/retrieval . 3 | # inputs: Input location of uploaded image, extracted vectors 4 | # 5 | ################################################################################################################################ 6 | import random 7 | import tensorflow as tf 8 | import numpy as np 9 | import os 10 | import scipy.io 11 | import time 12 | from datetime import datetime 13 | from scipy import ndimage 14 | from scipy.misc import imsave 15 | from scipy.spatial.distance import cosine 16 | #import matplotlib.pyplot as plt 17 | from sklearn.neighbors import NearestNeighbors 18 | import pickle 19 | from PIL import Image 20 | import gc 21 | import os 22 | from tempfile import TemporaryFile 23 | from tensorflow.python.platform import gfile 24 | BOTTLENECK_TENSOR_NAME = 'pool_3/_reshape:0' 25 | BOTTLENECK_TENSOR_SIZE = 2048 26 | MODEL_INPUT_WIDTH = 299 27 | MODEL_INPUT_HEIGHT = 299 28 | MODEL_INPUT_DEPTH = 3 29 | JPEG_DATA_TENSOR_NAME = 'DecodeJpeg/contents:0' 30 | RESIZED_INPUT_TENSOR_NAME = 'ResizeBilinear:0' 31 | MAX_NUM_IMAGES_PER_CLASS = 2 ** 27 - 1 # ~134M 32 | 33 | #show_neighbors(random.randint(0, len(extracted_features)), indices, neighbor_list) 34 | 35 | def get_top_k_similar(image_data, pred, pred_final, k=9): 36 | print("total data",len(pred)) 37 | print(image_data.shape) 38 | #for i in pred: 39 | #print(i.shape) 40 | #break 41 | os.mkdir('static/result') 42 | 43 | # cosine calculates the cosine distance, not similiarity. Hence no need to reverse list 44 | top_k_ind = np.argsort([cosine(image_data, pred_row) \ 45 | for ith_row, pred_row in enumerate(pred)])[:k] 46 | print(top_k_ind) 47 | 48 | for i, neighbor in enumerate(top_k_ind): 49 | image = ndimage.imread(pred_final[neighbor]) 50 | 51 | timestr = datetime.now().strftime("%Y%m%d%H%M%S") 52 | name= timestr+"."+str(i) 53 | print(name) 54 | name = 'static/result/image'+"_"+name+'.jpg' 55 | imsave(name, image) 56 | 57 | 58 | def create_inception_graph(): 59 | """"Creates a graph from saved GraphDef file and returns a Graph object. 60 | 61 | Returns: 62 | Graph holding the trained Inception network, and various tensors we'll be 63 | manipulating. 64 | """ 65 | with tf.Session() as sess: 66 | model_filename = os.path.join( 67 | 'imagenet', 'classify_image_graph_def.pb') 68 | with gfile.FastGFile(model_filename, 'rb') as f: 69 | graph_def = tf.GraphDef() 70 | graph_def.ParseFromString(f.read()) 71 | bottleneck_tensor, jpeg_data_tensor, resized_input_tensor = ( 72 | tf.import_graph_def(graph_def, name='', return_elements=[ 73 | BOTTLENECK_TENSOR_NAME, JPEG_DATA_TENSOR_NAME, 74 | RESIZED_INPUT_TENSOR_NAME])) 75 | return sess.graph, bottleneck_tensor, jpeg_data_tensor, resized_input_tensor 76 | 77 | def run_bottleneck_on_image(sess, image_data, image_data_tensor, 78 | bottleneck_tensor): 79 | 80 | bottleneck_values = sess.run( 81 | bottleneck_tensor, 82 | {image_data_tensor: image_data}) 83 | bottleneck_values = np.squeeze(bottleneck_values) 84 | return bottleneck_values 85 | 86 | def recommend(imagePath, extracted_features): 87 | 88 | 89 | tf.reset_default_graph() 90 | 91 | config = tf.ConfigProto( 92 | device_count = {'GPU': 0} 93 | ) 94 | 95 | 96 | 97 | sess = tf.Session(config=config) 98 | graph, bottleneck_tensor, jpeg_data_tensor, resized_image_tensor = (create_inception_graph()) 99 | image_data = gfile.FastGFile(imagePath, 'rb').read() 100 | features = run_bottleneck_on_image(sess, image_data, jpeg_data_tensor, bottleneck_tensor) 101 | 102 | with open('../lib/neighbor_list_recom.pickle','rb') as f: 103 | neighbor_list = pickle.load(f) 104 | print("loaded images") 105 | get_top_k_similar(features, extracted_features, neighbor_list, k=9) 106 | 107 | -------------------------------------------------------------------------------- /lib/search.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/lib/search.pyc -------------------------------------------------------------------------------- /server/image_vectorizer.py: -------------------------------------------------------------------------------- 1 | ################################################################################################################################ 2 | # This file is used to extract features from dataset and save it on disc 3 | # inputs: 4 | # outputs: 5 | ################################################################################################################################ 6 | 7 | import random 8 | import tensorflow as tf 9 | import numpy as np 10 | import os 11 | from scipy import ndimage 12 | from scipy.spatial.distance import cosine 13 | import matplotlib.pyplot as plt 14 | from sklearn.neighbors import NearestNeighbors 15 | import pickle 16 | from tensorflow.python.platform import gfile 17 | 18 | BOTTLENECK_TENSOR_NAME = 'pool_3/_reshape:0' 19 | BOTTLENECK_TENSOR_SIZE = 2048 20 | MODEL_INPUT_WIDTH = 299 21 | MODEL_INPUT_HEIGHT = 299 22 | MODEL_INPUT_DEPTH = 3 23 | JPEG_DATA_TENSOR_NAME = 'DecodeJpeg/contents:0' 24 | RESIZED_INPUT_TENSOR_NAME = 'ResizeBilinear:0' 25 | MAX_NUM_IMAGES_PER_CLASS = 2 ** 27 - 1 # ~134M 26 | 27 | def create_inception_graph(): 28 | """"Creates a graph from saved GraphDef file and returns a Graph object. 29 | 30 | Returns: 31 | Graph holding the trained Inception network, and various tensors we'll be 32 | manipulating. 33 | """ 34 | with tf.Session() as sess: 35 | model_filename = os.path.join( 36 | 'imagenet', 'classify_image_graph_def.pb') 37 | with gfile.FastGFile(model_filename, 'rb') as f: 38 | graph_def = tf.GraphDef() 39 | graph_def.ParseFromString(f.read()) 40 | bottleneck_tensor, jpeg_data_tensor, resized_input_tensor = ( 41 | tf.import_graph_def(graph_def, name='', return_elements=[ 42 | BOTTLENECK_TENSOR_NAME, JPEG_DATA_TENSOR_NAME, 43 | RESIZED_INPUT_TENSOR_NAME])) 44 | return sess.graph, bottleneck_tensor, jpeg_data_tensor, resized_input_tensor 45 | 46 | def run_bottleneck_on_image(sess, image_data, image_data_tensor, 47 | bottleneck_tensor): 48 | 49 | bottleneck_values = sess.run( 50 | bottleneck_tensor, 51 | {image_data_tensor: image_data}) 52 | bottleneck_values = np.squeeze(bottleneck_values) 53 | return bottleneck_values 54 | 55 | # Get outputs from second-to-last layer in pre-built model 56 | 57 | 58 | boots_files = [ 59 | 'uploads/dogs_and_cats/Boots/' + f 60 | for 61 | f 62 | in 63 | os.listdir('uploads/dogs_and_cats/Boots') 64 | ] 65 | sandals_files = [ 66 | 'uploads/dogs_and_cats/Sandals/' + f 67 | for 68 | f 69 | in 70 | os.listdir('uploads/dogs_and_cats/Sandals') 71 | ] 72 | shoes_files = [ 73 | 'uploads/dogs_and_cats/Shoes/' + f 74 | for 75 | f 76 | in 77 | os.listdir('uploads/dogs_and_cats/Shoes') 78 | ] 79 | slippers_files = [ 80 | 'uploads/dogs_and_cats/Slippers/' + f 81 | for 82 | f 83 | in 84 | os.listdir('uploads/dogs_and_cats/Slippers') 85 | ] 86 | apparel_files = [ 87 | 'uploads/dogs_and_cats/apparel/' + f 88 | for 89 | f 90 | in 91 | os.listdir('uploads/dogs_and_cats/apparel') 92 | ] 93 | 94 | 95 | all_files = boots_files + shoes_files + slippers_files + sandals_files+ apparel_files 96 | 97 | random.shuffle(all_files) 98 | 99 | num_images = 10000 100 | neighbor_list = all_files[:num_images] 101 | with open('neighbor_list_recom.pickle','wb') as f: 102 | pickle.dump(neighbor_list,f) 103 | print("saved neighbour list") 104 | 105 | extracted_features = np.ndarray((num_images, 2048)) 106 | sess = tf.Session() 107 | graph, bottleneck_tensor, jpeg_data_tensor, resized_image_tensor = (create_inception_graph()) 108 | 109 | 110 | for i, filename in enumerate(neighbor_list): 111 | 112 | image_data = gfile.FastGFile(filename, 'rb').read() 113 | features = run_bottleneck_on_image(sess, image_data, jpeg_data_tensor, bottleneck_tensor) 114 | 115 | extracted_features[i:i+1] = features 116 | 117 | if i % 250 == 0: 118 | print(i) 119 | 120 | 121 | np.savetxt("saved_features_recom.txt", extracted_features) 122 | print("saved exttracted features") 123 | 124 | 125 | 126 | 127 | 128 | 129 | -------------------------------------------------------------------------------- /server/rest-server.py: -------------------------------------------------------------------------------- 1 | #!flask/bin/python 2 | ################################################################################################################################ 3 | #------------------------------------------------------------------------------------------------------------------------------ 4 | # This file implements the REST layer. It uses flask micro framework for server implementation. Calls from front end reaches 5 | # here as json and being branched out to each projects. Basic level of validation is also being done in this file. # 6 | #------------------------------------------------------------------------------------------------------------------------------- 7 | ################################################################################################################################ 8 | from flask import Flask, jsonify, abort, request, make_response, url_for,redirect, render_template 9 | from flask.ext.httpauth import HTTPBasicAuth 10 | from werkzeug.utils import secure_filename 11 | import os 12 | import sys 13 | sys.path.append('../') 14 | import shutil 15 | import numpy as np 16 | from lib.search import recommend 17 | import tarfile 18 | from datetime import datetime 19 | from scipy import ndimage 20 | from scipy.misc import imsave 21 | UPLOAD_FOLDER = 'uploads' 22 | ALLOWED_EXTENSIONS = set(['png', 'jpg', 'jpeg']) 23 | from tensorflow.python.platform import gfile 24 | app = Flask(__name__, static_url_path = "") 25 | app.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER 26 | auth = HTTPBasicAuth() 27 | 28 | #============================================================================================================================== 29 | # 30 | # Loading the extracted feature vectors for image retrieval 31 | # 32 | # 33 | #============================================================================================================================== 34 | extracted_features=np.zeros((10000,2048),dtype=np.float32) 35 | with open('../lib/saved_features_recom.txt') as f: 36 | for i,line in enumerate(f): 37 | extracted_features[i,:]=line.split() 38 | print("loaded extracted_features") 39 | 40 | 41 | #============================================================================================================================== 42 | # 43 | # This function is used to do the image search/image retrieval 44 | # 45 | # 46 | #============================================================================================================================== 47 | @app.route('/imgUpload', methods=['GET', 'POST']) 48 | def upload_img(): 49 | print("image upload") 50 | result = 'static/result' 51 | if not gfile.Exists(result): 52 | os.mkdir(result) 53 | shutil.rmtree(result) 54 | 55 | if request.method == 'POST' or request.method == 'GET': 56 | print(request.method) 57 | # check if the post request has the file part 58 | if 'file' not in request.files: 59 | 60 | print('No file part') 61 | return redirect(request.url) 62 | 63 | file = request.files['file'] 64 | print(file.filename) 65 | # if user does not select file, browser also 66 | # submit a empty part without filename 67 | if file.filename == '': 68 | 69 | print('No selected file') 70 | return redirect(request.url) 71 | if file and allowed_file(file.filename): 72 | 73 | filename = secure_filename(file.filename) 74 | file.save(os.path.join(app.config['UPLOAD_FOLDER'], filename)) 75 | inputloc = os.path.join(app.config['UPLOAD_FOLDER'], filename) 76 | #outputloc = '/home/vinayak/todo-api/out' 77 | recommend(inputloc, extracted_features) 78 | os.remove(inputloc) 79 | #label = label1.replace("\n", "") 80 | image_path = "/result" 81 | image_list =[os.path.join(image_path,file) for file in os.listdir(result) 82 | if not file.startswith('.')] 83 | images = { 84 | 'image0':image_list[0], 85 | 'image1':image_list[1], 86 | 'image2':image_list[2], 87 | 'image3':image_list[3], 88 | 'image4':image_list[4], 89 | 'image5':image_list[5], 90 | 'image6':image_list[6], 91 | 'image7':image_list[7], 92 | 'image8':image_list[8] 93 | } 94 | return jsonify(images) 95 | 96 | #============================================================================================================================== 97 | # 98 | # Main function # 99 | # 100 | #============================================================================================================================== 101 | @app.route("/") 102 | def main(): 103 | 104 | return render_template("main.html") 105 | if __name__ == '__main__': 106 | app.run(debug = True, host= '0.0.0.0') 107 | -------------------------------------------------------------------------------- /server/static/images/Predict_GREY.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/images/Predict_GREY.png -------------------------------------------------------------------------------- /server/static/images/Train_GREY.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/images/Train_GREY.png -------------------------------------------------------------------------------- /server/static/images/ajax-loader.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/images/ajax-loader.gif -------------------------------------------------------------------------------- /server/static/images/predict.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/images/predict.png -------------------------------------------------------------------------------- /server/static/images/result1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/images/result1.jpg -------------------------------------------------------------------------------- /server/static/images/result2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/images/result2.jpg -------------------------------------------------------------------------------- /server/static/images/search.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/images/search.png -------------------------------------------------------------------------------- /server/static/images/train.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/images/train.png -------------------------------------------------------------------------------- /server/static/result/image_20171204154121.0.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.0.jpg -------------------------------------------------------------------------------- /server/static/result/image_20171204154121.1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.1.jpg -------------------------------------------------------------------------------- /server/static/result/image_20171204154121.2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.2.jpg -------------------------------------------------------------------------------- /server/static/result/image_20171204154121.3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.3.jpg -------------------------------------------------------------------------------- /server/static/result/image_20171204154121.4.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.4.jpg -------------------------------------------------------------------------------- /server/static/result/image_20171204154121.5.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.5.jpg -------------------------------------------------------------------------------- /server/static/result/image_20171204154121.6.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.6.jpg -------------------------------------------------------------------------------- /server/static/result/image_20171204154121.7.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.7.jpg -------------------------------------------------------------------------------- /server/static/result/image_20171204154121.8.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.8.jpg -------------------------------------------------------------------------------- /server/templates/recommend.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | Recommendation Engine 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 27 | 33 | 34 |




35 |
36 | 37 | 38 |
39 | 40 | 41 |
42 | 44 | 45 | 51 | 52 | 53 | 54 | 55 | 56 |
Chose your file to upload
46 | 47 | 48 |                49 |                        50 |
57 | 58 | 59 |
60 |
61 | 62 |
63 |
64 | 65 | 91 | 92 | 119 | 145 | 146 | 172 | 198 | 224 | 225 | 251 | 277 | 303 | 304 | 305 | 306 | 307 | 308 |
309 | 310 | 311 | 312 | 313 | 314 | 369 | 370 | 371 | -------------------------------------------------------------------------------- /server/uploads/101026.3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/uploads/101026.3.jpg --------------------------------------------------------------------------------