├── README.md
├── __init__.py
├── lib
├── __init__.py
├── __init__.pyc
├── neighbor_list_recom.pickle
├── search.py
└── search.pyc
└── server
├── image_vectorizer.py
├── rest-server.py
├── static
├── images
│ ├── Predict_GREY.png
│ ├── Train_GREY.png
│ ├── ajax-loader.gif
│ ├── predict.png
│ ├── result1.jpg
│ ├── result2.jpg
│ ├── search.png
│ └── train.png
└── result
│ ├── image_20171204154121.0.jpg
│ ├── image_20171204154121.1.jpg
│ ├── image_20171204154121.2.jpg
│ ├── image_20171204154121.3.jpg
│ ├── image_20171204154121.4.jpg
│ ├── image_20171204154121.5.jpg
│ ├── image_20171204154121.6.jpg
│ ├── image_20171204154121.7.jpg
│ └── image_20171204154121.8.jpg
├── templates
└── recommend.html
└── uploads
└── 101026.3.jpg
/README.md:
--------------------------------------------------------------------------------
1 | # Image similarity search using deep learning
2 | This project demonstrates, how we can make use of deep learning to do state-of-the-art image similarity search. I have used tensorflow and some publicly available datasets.
3 |
4 | ## Results
5 | 
6 | 
7 | 
8 |
9 | ## How to run
10 | 1. Download [imagenet](https://drive.google.com/open?id=1UOyZ8166qM3SzxGvaUeWpBzSUfoQLUjJ) folder, extraxt and keep it in server directory
11 | 2. Download datasets for [footwares](http://vision.cs.utexas.edu/projects/finegrained/utzap50k/), [apparels](http://mmlab.ie.cuhk.edu.hk/projects/DeepFashion/InShopRetrieval.html) keep them inside a directory under upload folder. Final folder strcture will be as below
12 | ```
13 | root folder
14 | │
15 | └───lib
16 | │
17 | └───server
18 | | │───rest-server.py
19 | | │───imagenet
20 | | │───static
21 | | │───templates
22 | | │───uploads
23 | | │────dogs_and_cats
24 | | │────shoes
25 | | │────sandals
26 | | │────slippers
27 | | │────boots
28 | | │────apparels
29 | ```
30 | 3. Run image vectorizer which passes each data through an inception-v3 model and collects the bottleneck layer vectors and stores in disc. Edit dataset paths accordingly indide the image_vectorizer.py
31 | ```
32 | python server/image_vectorizer.py
33 | ```
34 | This will generate two files namely, image_list.pickle and saved_features.txt. Keep them inside lib folder where search.py script is available.
35 |
36 | 4. Start the server by running rest-server.py. This project uses flask based REST implementation for UI
37 | ```
38 | python server/rest-server.py
39 | ```
40 | 5. Once the server starts up, access the url 127.0.0.1:5000 to get the UI. Now upload any file and see 9 similar images. You can change the value of K from 9 to any values, but dont foreget to update the html file accordingly for displaying.
41 |
42 | One interesting application of this project is a recommendation engine based on image features.Here is an example of similar project of mine. Here instead of a web UI i have used an android UI. Once the user clicks a product image, the image will go to the server and k-number of similar product images can be displayed on UI as product recommendations. Theses rescommendations are purely based on image similarity. This kind of recommendations have high potentials in fashion-based ecommerce industry.
43 |
44 | 
45 | 
46 |
47 |
--------------------------------------------------------------------------------
/__init__.py:
--------------------------------------------------------------------------------
1 |
2 |
--------------------------------------------------------------------------------
/lib/__init__.py:
--------------------------------------------------------------------------------
1 |
2 |
--------------------------------------------------------------------------------
/lib/__init__.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/lib/__init__.pyc
--------------------------------------------------------------------------------
/lib/search.py:
--------------------------------------------------------------------------------
1 | ################################################################################################################################
2 | # This function implements the image search/retrieval .
3 | # inputs: Input location of uploaded image, extracted vectors
4 | #
5 | ################################################################################################################################
6 | import random
7 | import tensorflow as tf
8 | import numpy as np
9 | import os
10 | import scipy.io
11 | import time
12 | from datetime import datetime
13 | from scipy import ndimage
14 | from scipy.misc import imsave
15 | from scipy.spatial.distance import cosine
16 | #import matplotlib.pyplot as plt
17 | from sklearn.neighbors import NearestNeighbors
18 | import pickle
19 | from PIL import Image
20 | import gc
21 | import os
22 | from tempfile import TemporaryFile
23 | from tensorflow.python.platform import gfile
24 | BOTTLENECK_TENSOR_NAME = 'pool_3/_reshape:0'
25 | BOTTLENECK_TENSOR_SIZE = 2048
26 | MODEL_INPUT_WIDTH = 299
27 | MODEL_INPUT_HEIGHT = 299
28 | MODEL_INPUT_DEPTH = 3
29 | JPEG_DATA_TENSOR_NAME = 'DecodeJpeg/contents:0'
30 | RESIZED_INPUT_TENSOR_NAME = 'ResizeBilinear:0'
31 | MAX_NUM_IMAGES_PER_CLASS = 2 ** 27 - 1 # ~134M
32 |
33 | #show_neighbors(random.randint(0, len(extracted_features)), indices, neighbor_list)
34 |
35 | def get_top_k_similar(image_data, pred, pred_final, k=9):
36 | print("total data",len(pred))
37 | print(image_data.shape)
38 | #for i in pred:
39 | #print(i.shape)
40 | #break
41 | os.mkdir('static/result')
42 |
43 | # cosine calculates the cosine distance, not similiarity. Hence no need to reverse list
44 | top_k_ind = np.argsort([cosine(image_data, pred_row) \
45 | for ith_row, pred_row in enumerate(pred)])[:k]
46 | print(top_k_ind)
47 |
48 | for i, neighbor in enumerate(top_k_ind):
49 | image = ndimage.imread(pred_final[neighbor])
50 |
51 | timestr = datetime.now().strftime("%Y%m%d%H%M%S")
52 | name= timestr+"."+str(i)
53 | print(name)
54 | name = 'static/result/image'+"_"+name+'.jpg'
55 | imsave(name, image)
56 |
57 |
58 | def create_inception_graph():
59 | """"Creates a graph from saved GraphDef file and returns a Graph object.
60 |
61 | Returns:
62 | Graph holding the trained Inception network, and various tensors we'll be
63 | manipulating.
64 | """
65 | with tf.Session() as sess:
66 | model_filename = os.path.join(
67 | 'imagenet', 'classify_image_graph_def.pb')
68 | with gfile.FastGFile(model_filename, 'rb') as f:
69 | graph_def = tf.GraphDef()
70 | graph_def.ParseFromString(f.read())
71 | bottleneck_tensor, jpeg_data_tensor, resized_input_tensor = (
72 | tf.import_graph_def(graph_def, name='', return_elements=[
73 | BOTTLENECK_TENSOR_NAME, JPEG_DATA_TENSOR_NAME,
74 | RESIZED_INPUT_TENSOR_NAME]))
75 | return sess.graph, bottleneck_tensor, jpeg_data_tensor, resized_input_tensor
76 |
77 | def run_bottleneck_on_image(sess, image_data, image_data_tensor,
78 | bottleneck_tensor):
79 |
80 | bottleneck_values = sess.run(
81 | bottleneck_tensor,
82 | {image_data_tensor: image_data})
83 | bottleneck_values = np.squeeze(bottleneck_values)
84 | return bottleneck_values
85 |
86 | def recommend(imagePath, extracted_features):
87 |
88 |
89 | tf.reset_default_graph()
90 |
91 | config = tf.ConfigProto(
92 | device_count = {'GPU': 0}
93 | )
94 |
95 |
96 |
97 | sess = tf.Session(config=config)
98 | graph, bottleneck_tensor, jpeg_data_tensor, resized_image_tensor = (create_inception_graph())
99 | image_data = gfile.FastGFile(imagePath, 'rb').read()
100 | features = run_bottleneck_on_image(sess, image_data, jpeg_data_tensor, bottleneck_tensor)
101 |
102 | with open('../lib/neighbor_list_recom.pickle','rb') as f:
103 | neighbor_list = pickle.load(f)
104 | print("loaded images")
105 | get_top_k_similar(features, extracted_features, neighbor_list, k=9)
106 |
107 |
--------------------------------------------------------------------------------
/lib/search.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/lib/search.pyc
--------------------------------------------------------------------------------
/server/image_vectorizer.py:
--------------------------------------------------------------------------------
1 | ################################################################################################################################
2 | # This file is used to extract features from dataset and save it on disc
3 | # inputs:
4 | # outputs:
5 | ################################################################################################################################
6 |
7 | import random
8 | import tensorflow as tf
9 | import numpy as np
10 | import os
11 | from scipy import ndimage
12 | from scipy.spatial.distance import cosine
13 | import matplotlib.pyplot as plt
14 | from sklearn.neighbors import NearestNeighbors
15 | import pickle
16 | from tensorflow.python.platform import gfile
17 |
18 | BOTTLENECK_TENSOR_NAME = 'pool_3/_reshape:0'
19 | BOTTLENECK_TENSOR_SIZE = 2048
20 | MODEL_INPUT_WIDTH = 299
21 | MODEL_INPUT_HEIGHT = 299
22 | MODEL_INPUT_DEPTH = 3
23 | JPEG_DATA_TENSOR_NAME = 'DecodeJpeg/contents:0'
24 | RESIZED_INPUT_TENSOR_NAME = 'ResizeBilinear:0'
25 | MAX_NUM_IMAGES_PER_CLASS = 2 ** 27 - 1 # ~134M
26 |
27 | def create_inception_graph():
28 | """"Creates a graph from saved GraphDef file and returns a Graph object.
29 |
30 | Returns:
31 | Graph holding the trained Inception network, and various tensors we'll be
32 | manipulating.
33 | """
34 | with tf.Session() as sess:
35 | model_filename = os.path.join(
36 | 'imagenet', 'classify_image_graph_def.pb')
37 | with gfile.FastGFile(model_filename, 'rb') as f:
38 | graph_def = tf.GraphDef()
39 | graph_def.ParseFromString(f.read())
40 | bottleneck_tensor, jpeg_data_tensor, resized_input_tensor = (
41 | tf.import_graph_def(graph_def, name='', return_elements=[
42 | BOTTLENECK_TENSOR_NAME, JPEG_DATA_TENSOR_NAME,
43 | RESIZED_INPUT_TENSOR_NAME]))
44 | return sess.graph, bottleneck_tensor, jpeg_data_tensor, resized_input_tensor
45 |
46 | def run_bottleneck_on_image(sess, image_data, image_data_tensor,
47 | bottleneck_tensor):
48 |
49 | bottleneck_values = sess.run(
50 | bottleneck_tensor,
51 | {image_data_tensor: image_data})
52 | bottleneck_values = np.squeeze(bottleneck_values)
53 | return bottleneck_values
54 |
55 | # Get outputs from second-to-last layer in pre-built model
56 |
57 |
58 | boots_files = [
59 | 'uploads/dogs_and_cats/Boots/' + f
60 | for
61 | f
62 | in
63 | os.listdir('uploads/dogs_and_cats/Boots')
64 | ]
65 | sandals_files = [
66 | 'uploads/dogs_and_cats/Sandals/' + f
67 | for
68 | f
69 | in
70 | os.listdir('uploads/dogs_and_cats/Sandals')
71 | ]
72 | shoes_files = [
73 | 'uploads/dogs_and_cats/Shoes/' + f
74 | for
75 | f
76 | in
77 | os.listdir('uploads/dogs_and_cats/Shoes')
78 | ]
79 | slippers_files = [
80 | 'uploads/dogs_and_cats/Slippers/' + f
81 | for
82 | f
83 | in
84 | os.listdir('uploads/dogs_and_cats/Slippers')
85 | ]
86 | apparel_files = [
87 | 'uploads/dogs_and_cats/apparel/' + f
88 | for
89 | f
90 | in
91 | os.listdir('uploads/dogs_and_cats/apparel')
92 | ]
93 |
94 |
95 | all_files = boots_files + shoes_files + slippers_files + sandals_files+ apparel_files
96 |
97 | random.shuffle(all_files)
98 |
99 | num_images = 10000
100 | neighbor_list = all_files[:num_images]
101 | with open('neighbor_list_recom.pickle','wb') as f:
102 | pickle.dump(neighbor_list,f)
103 | print("saved neighbour list")
104 |
105 | extracted_features = np.ndarray((num_images, 2048))
106 | sess = tf.Session()
107 | graph, bottleneck_tensor, jpeg_data_tensor, resized_image_tensor = (create_inception_graph())
108 |
109 |
110 | for i, filename in enumerate(neighbor_list):
111 |
112 | image_data = gfile.FastGFile(filename, 'rb').read()
113 | features = run_bottleneck_on_image(sess, image_data, jpeg_data_tensor, bottleneck_tensor)
114 |
115 | extracted_features[i:i+1] = features
116 |
117 | if i % 250 == 0:
118 | print(i)
119 |
120 |
121 | np.savetxt("saved_features_recom.txt", extracted_features)
122 | print("saved exttracted features")
123 |
124 |
125 |
126 |
127 |
128 |
129 |
--------------------------------------------------------------------------------
/server/rest-server.py:
--------------------------------------------------------------------------------
1 | #!flask/bin/python
2 | ################################################################################################################################
3 | #------------------------------------------------------------------------------------------------------------------------------
4 | # This file implements the REST layer. It uses flask micro framework for server implementation. Calls from front end reaches
5 | # here as json and being branched out to each projects. Basic level of validation is also being done in this file. #
6 | #-------------------------------------------------------------------------------------------------------------------------------
7 | ################################################################################################################################
8 | from flask import Flask, jsonify, abort, request, make_response, url_for,redirect, render_template
9 | from flask.ext.httpauth import HTTPBasicAuth
10 | from werkzeug.utils import secure_filename
11 | import os
12 | import sys
13 | sys.path.append('../')
14 | import shutil
15 | import numpy as np
16 | from lib.search import recommend
17 | import tarfile
18 | from datetime import datetime
19 | from scipy import ndimage
20 | from scipy.misc import imsave
21 | UPLOAD_FOLDER = 'uploads'
22 | ALLOWED_EXTENSIONS = set(['png', 'jpg', 'jpeg'])
23 | from tensorflow.python.platform import gfile
24 | app = Flask(__name__, static_url_path = "")
25 | app.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER
26 | auth = HTTPBasicAuth()
27 |
28 | #==============================================================================================================================
29 | #
30 | # Loading the extracted feature vectors for image retrieval
31 | #
32 | #
33 | #==============================================================================================================================
34 | extracted_features=np.zeros((10000,2048),dtype=np.float32)
35 | with open('../lib/saved_features_recom.txt') as f:
36 | for i,line in enumerate(f):
37 | extracted_features[i,:]=line.split()
38 | print("loaded extracted_features")
39 |
40 |
41 | #==============================================================================================================================
42 | #
43 | # This function is used to do the image search/image retrieval
44 | #
45 | #
46 | #==============================================================================================================================
47 | @app.route('/imgUpload', methods=['GET', 'POST'])
48 | def upload_img():
49 | print("image upload")
50 | result = 'static/result'
51 | if not gfile.Exists(result):
52 | os.mkdir(result)
53 | shutil.rmtree(result)
54 |
55 | if request.method == 'POST' or request.method == 'GET':
56 | print(request.method)
57 | # check if the post request has the file part
58 | if 'file' not in request.files:
59 |
60 | print('No file part')
61 | return redirect(request.url)
62 |
63 | file = request.files['file']
64 | print(file.filename)
65 | # if user does not select file, browser also
66 | # submit a empty part without filename
67 | if file.filename == '':
68 |
69 | print('No selected file')
70 | return redirect(request.url)
71 | if file and allowed_file(file.filename):
72 |
73 | filename = secure_filename(file.filename)
74 | file.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
75 | inputloc = os.path.join(app.config['UPLOAD_FOLDER'], filename)
76 | #outputloc = '/home/vinayak/todo-api/out'
77 | recommend(inputloc, extracted_features)
78 | os.remove(inputloc)
79 | #label = label1.replace("\n", "")
80 | image_path = "/result"
81 | image_list =[os.path.join(image_path,file) for file in os.listdir(result)
82 | if not file.startswith('.')]
83 | images = {
84 | 'image0':image_list[0],
85 | 'image1':image_list[1],
86 | 'image2':image_list[2],
87 | 'image3':image_list[3],
88 | 'image4':image_list[4],
89 | 'image5':image_list[5],
90 | 'image6':image_list[6],
91 | 'image7':image_list[7],
92 | 'image8':image_list[8]
93 | }
94 | return jsonify(images)
95 |
96 | #==============================================================================================================================
97 | #
98 | # Main function #
99 | #
100 | #==============================================================================================================================
101 | @app.route("/")
102 | def main():
103 |
104 | return render_template("main.html")
105 | if __name__ == '__main__':
106 | app.run(debug = True, host= '0.0.0.0')
107 |
--------------------------------------------------------------------------------
/server/static/images/Predict_GREY.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/images/Predict_GREY.png
--------------------------------------------------------------------------------
/server/static/images/Train_GREY.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/images/Train_GREY.png
--------------------------------------------------------------------------------
/server/static/images/ajax-loader.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/images/ajax-loader.gif
--------------------------------------------------------------------------------
/server/static/images/predict.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/images/predict.png
--------------------------------------------------------------------------------
/server/static/images/result1.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/images/result1.jpg
--------------------------------------------------------------------------------
/server/static/images/result2.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/images/result2.jpg
--------------------------------------------------------------------------------
/server/static/images/search.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/images/search.png
--------------------------------------------------------------------------------
/server/static/images/train.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/images/train.png
--------------------------------------------------------------------------------
/server/static/result/image_20171204154121.0.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.0.jpg
--------------------------------------------------------------------------------
/server/static/result/image_20171204154121.1.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.1.jpg
--------------------------------------------------------------------------------
/server/static/result/image_20171204154121.2.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.2.jpg
--------------------------------------------------------------------------------
/server/static/result/image_20171204154121.3.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.3.jpg
--------------------------------------------------------------------------------
/server/static/result/image_20171204154121.4.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.4.jpg
--------------------------------------------------------------------------------
/server/static/result/image_20171204154121.5.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.5.jpg
--------------------------------------------------------------------------------
/server/static/result/image_20171204154121.6.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.6.jpg
--------------------------------------------------------------------------------
/server/static/result/image_20171204154121.7.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.7.jpg
--------------------------------------------------------------------------------
/server/static/result/image_20171204154121.8.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/static/result/image_20171204154121.8.jpg
--------------------------------------------------------------------------------
/server/templates/recommend.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 | Recommendation Engine
6 |
7 |
8 |
9 |
10 |
11 |
12 |
13 |
14 |
15 |
16 |
17 |
18 |
19 |
20 |
21 |
22 |
23 |
24 |
27 |
28 |
IMAGE SEARCH DEMO
29 |
30 |
31 |
32 |
33 |
34 |
35 |
36 |
37 |
38 |
39 |
40 |
41 |
42 |
44 | Chose your file to upload | |
45 | | |
51 |
52 |
53 | | |
54 |
55 |
56 |
57 |
58 |
59 |
60 |
61 |
62 |
63 |
309 |
310 |
311 |
312 |
313 |
314 |
369 |
370 |
371 |
--------------------------------------------------------------------------------
/server/uploads/101026.3.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/vinayakarannil/Deeplearning_Image_Similarity/223bc5c4670bc12a358305529f306c2ef677597a/server/uploads/101026.3.jpg
--------------------------------------------------------------------------------