![](\"https://openaiassets.blob.core.windows.net/$web/clip/draft/20210104b/overview-b.svg\")
\n",
171 | "\n",
172 | "CLIP allows us to link images with their description and map them to the same embedding space.\n",
173 | "\n",
174 | "### Similarity search\n",
175 | "\n",
176 | "Once the embedding is done, we need to run a nearest-neighbor search using the `cosine` similarity measure.\n",
177 | "\n",
178 | "The products that are closest to the query vector should (hopefully) be similar to the customer's intentions.\n",
179 | "\n",
180 | "The query vector could be a result of either `CLIP` image embedding or `CLIP` textual embedding.\n",
181 | "\n",
182 | "We will use the [vecsim](https://github.com/argmaxml/vecsim) module to do the similarity search.\n",
183 | "\n",
184 | "### Serving\n",
185 | "\n",
186 | "We used [Flask](https://flask.palletsprojects.com/en/2.2.x/) to implement the web-server, the code is at `server.py`.\n",
187 | "\n",
188 | "**Note**: The server code cotains several `TODO:` comments, you will need to implement. The server is currently functional and it outputs random results.\n",
189 | "\n",
190 | "# Submission\n",
191 | "\n",
192 | "\n",
193 | "1. Please clone this repo to a **private** repo on your github account.\n",
194 | "1. Implement the missing parts.\n",
195 | "1. Please fill in [this form](https://forms.gle/apMr8zPLbBf9pQY7A).\n",
196 | "1. Once done, please schdule an interview with Uri to review the code\n",
197 | "\n",
198 | "## Submission deadline:\n",
199 | "December 21st, 2022\n",
200 | "\n",
201 | "## Good luck !\n"
202 | ]
203 | },
204 | {
205 | "cell_type": "code",
206 | "execution_count": null,
207 | "metadata": {},
208 | "outputs": [],
209 | "source": []
210 | }
211 | ],
212 | "metadata": {
213 | "kernelspec": {
214 | "display_name": "Python 3",
215 | "language": "python",
216 | "name": "python3"
217 | },
218 | "language_info": {
219 | "codemirror_mode": {
220 | "name": "ipython",
221 | "version": 3
222 | },
223 | "file_extension": ".py",
224 | "mimetype": "text/x-python",
225 | "name": "python",
226 | "nbconvert_exporter": "python",
227 | "pygments_lexer": "ipython3",
228 | "version": "3.8.10"
229 | }
230 | },
231 | "nbformat": 4,
232 | "nbformat_minor": 4
233 | }
234 |
--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | pandas>=1.4.1
2 | numpy>=1.22.2
3 | vecsim>=0.0.2
4 | git+https://github.com/openai/CLIP.git
5 | Flask>=2.0.1
6 | pyarrow==7.0.0
7 |
--------------------------------------------------------------------------------
/server.py:
--------------------------------------------------------------------------------
1 | import sys, json, collections, random
2 | sys.path.append("src")
3 | from pathlib import Path
4 | from datetime import datetime, timedelta
5 | from dataclasses import dataclass
6 | from operator import itemgetter as at
7 | from operator import attrgetter as dot
8 | import pandas as pd
9 | import numpy as np
10 | from flask import Flask, request, send_from_directory, render_template, redirect, url_for, jsonify, Response, flash
11 | from werkzeug.utils import secure_filename
12 | from typing import List, Dict, Tuple, Optional
13 | from vecsim import SciKitIndex, RedisIndex
14 |
15 | __dir__ = Path(__file__).absolute().parent
16 | upload_dir = __dir__ / "upload"
17 | data_dir = __dir__ / "data"
18 | upload_dir.mkdir(exist_ok=True)
19 | NUMBER_OF_RESULTS = 12
20 | app = Flask(__name__)
21 |
22 | @dataclass
23 | class Recommendation:
24 | id: int
25 | image: str
26 | title: str
27 | highlight: bool
28 | distance: float
29 |
30 | @dataclass
31 | class Item:
32 | id : int
33 | image: str
34 | title : str
35 |
36 |
37 |
38 | @app.route('/favicon.ico')
39 | def favicon():
40 | return send_from_directory('assets', 'favicon.ico')
41 |
42 | @app.route('/assets/
16 | Argmax Search Demo
5 |
6 |
7 |
8 |
9 |
10 |
11 |
12 |
13 |
24 |
25 |
26 |
17 |
24 |
25 |
26 |
--------------------------------------------------------------------------------
/templates/index.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
18 | 
19 |
20 |
19 |
21 |
23 | Page not found
22 |
27 |
63 |
28 |
35 |
29 |
30 |
31 |
30 |
32 |
34 | Text+Image Search Demo
33 |
36 |
47 |
60 |
61 | {% include 'results.html' %}
62 |
64 |
68 |
69 |
70 |
--------------------------------------------------------------------------------
/templates/results.html:
--------------------------------------------------------------------------------
1 |
65 |
66 |
67 |
2 | {%for rec in recommendations%}
3 |
4 | {% if rec.highlight %}
5 |
--------------------------------------------------------------------------------
6 | {% else %}
7 |
17 | {%endfor%}
18 |
8 | {% endif %}
9 |
16 |
10 |
12 | {{rec.title}}
11 |
13 | 
14 |
15 |