├── .gitignore ├── README.md ├── app ├── app.py ├── data │ ├── GradREC_gif.gif │ ├── GradREC_gif_mini.gif │ ├── GradREC_traversal_vector_gif.gif │ ├── GradREC_traversal_vector_gif_mini.gif │ ├── demo_eg.gif │ ├── grad_rec_examples.json │ ├── tsne_fclip_blue_shirt.csv │ ├── tsne_fclip_shoes.csv │ └── tsne_fclip_skirt_length.csv ├── grad_rec_demo.py ├── gradient_rec.py ├── intro.py └── utils.py ├── grad_rec_api_demo.ipynb ├── main.py └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | pip-wheel-metadata/ 24 | share/python-wheels/ 25 | *.egg-info/ 26 | .installed.cfg 27 | *.egg 28 | MANIFEST 29 | 30 | # PyInstaller 31 | # Usually these files are written by a python script from a template 32 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 33 | *.manifest 34 | *.spec 35 | 36 | # Installer logs 37 | pip-log.txt 38 | pip-delete-this-directory.txt 39 | 40 | # Unit test / coverage reports 41 | htmlcov/ 42 | .tox/ 43 | .nox/ 44 | .coverage 45 | .coverage.* 46 | .cache 47 | nosetests.xml 48 | coverage.xml 49 | *.cover 50 | *.py,cover 51 | .hypothesis/ 52 | .pytest_cache/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | target/ 76 | 77 | # Jupyter Notebook 78 | .ipynb_checkpoints 79 | 80 | # IPython 81 | profile_default/ 82 | ipython_config.py 83 | 84 | # pyenv 85 | .python-version 86 | 87 | # pipenv 88 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 89 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 90 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 91 | # install all needed dependencies. 92 | #Pipfile.lock 93 | 94 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow 95 | __pypackages__/ 96 | 97 | # Celery stuff 98 | celerybeat-schedule 99 | celerybeat.pid 100 | 101 | # SageMath parsed files 102 | *.sage.py 103 | 104 | # Environments 105 | .env 106 | .venv 107 | env/ 108 | venv/ 109 | ENV/ 110 | env.bak/ 111 | venv.bak/ 112 | 113 | # Spyder project settings 114 | .spyderproject 115 | .spyproject 116 | 117 | # Rope project settings 118 | .ropeproject 119 | 120 | # mkdocs documentation 121 | /site 122 | 123 | # mypy 124 | .mypy_cache/ 125 | .dmypy.json 126 | dmypy.json 127 | 128 | # Pyre type checker 129 | .pyre/ 130 | 131 | # Ignore PyCharm 132 | .idea/ -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # `GradREC` 2 | 3 | __NB: Repo is still WIP!__ 4 | 5 | Link to Paper : [https://aclanthology.org/2022.ecnlp-1.22/](https://aclanthology.org/2022.ecnlp-1.22/) 6 | 7 | ## Overview 8 | 9 | Comparative recommendations answer questions of the form "Can I have something 10 | _darker/longer/warmer_". Most methods require extensive labelled data and employ 11 | learning-to-rank approaches. `GradREC` is a zero-shot approach toward generating comparative 12 | recommendations. We achieve this by framing comparative recommendations as latent 13 | space traversal -- a key component of our approach is leveraging the latent space learnt by 14 | [`FashionCLIP`](https://github.com/patrickjohncyh/fashion-clip), a CLIP-like model 15 | fine-tuned for fashion concepts. 16 | 17 | We postulated that like other self-supervised approaches 18 | for learning representations ([Mikolov et al., 2012](https://aclanthology.org/N13-1090/)), the contrastive 19 | learning approach employed by CLIP-like models should also encode certain concepts (e.g. analogies) despite 20 | being unsupervised/self-supervised. Of particular interest is the idea that it should be possible 21 | to traverse the latent space in a manner whereby we can discover products that vary along a certain 22 | attribute/dimension of interest (e.g. shoes of _increasing heel height_). 23 | 24 | Our preliminary investigation of this hypothesis established that we are indeed able to extract 25 | such comparative knowledge/concepts from CLIP-like models, allowing us to make comparative recommendations 26 | in a zero-shot fashion. 27 | 28 | We provide in this repo our implementation of our method, and an interactive demo that explores how 29 | `GradREC` works. 30 | 31 | 32 | 33 | ## API & Demo 34 | 35 | ### Pre-requisites 36 | 37 | To access the private bucket necessary to retrieve model weights and dataset, be sure to include an `.env` 38 | file containing the following: 39 | 40 | ``` 41 | AWS_ACCESS_KEY_ID 42 | AWS_SECRET_KEY 43 | ``` 44 | 45 | Critically, `GradREC` builds upon the fashion-clip package found 46 | [here](http://www.github.com/patrickjohncyh/fashion-clip). 47 | 48 | ### GradREC API 49 | 50 | The `GradREC` API provides an implementation of the latent space traversal approach described in our paper. 51 | It requires for initialization an instance of a `FashionCLIP` object (see 52 | fashion-clip [repo](http://www.github.com/patrickjohncyh/fashion-clip)). The API provides helper methods 53 | for each step in our methodology such as traversal vector construction and traversal function, with the ability 54 | to control the various parameters. 55 | 56 | ##### Usage 57 | A `GradREC` object is initialized as follows, 58 | ``` 59 | from gradient_rec import GracREC 60 | 61 | # we assume fclip is a FashionCLIP object 62 | gradrec = GradREC(fclip) 63 | ``` 64 | 65 | Several important methods are: 66 | 67 | - `direction_vector(self, start_query: str, end_query: str, 68 | start_N=100, end_N=1000)` 69 | - start_query: text for starting point to generate the semantic difference vector 70 | - end_query: text for ending point to generate the semantic difference vector 71 | - start_N: number of products to retrieve for start query 72 | - end_N: number of products to retrieve for end query 73 | 74 | - `traverse_space(self, 75 | start_point: np.ndarray, 76 | search_space: np.ndarray, 77 | v_dir: np.ndarray, 78 | step_size: float, 79 | steps: int, 80 | reg_space: np.ndarray, 81 | reg_weight: float, 82 | reg_k: int = 100, 83 | k=10)` 84 | 85 | - `start_point`: initial seed product vector or any point in space 86 | - `search_space`: latent representation of products to search 87 | - `v_dir`: traversal vector 88 | - `step_size`: size for each step taken 89 | - `steps`: number of steps to take 90 | - `reg_space`: latent representation of products to use for regularization 91 | - `reg_weight`: weight for amount of regularization to use 92 | - `reg_k`: number of nearest neighbors for use in regularization 93 | - `k`: number of products to return for each traversal step 94 | 95 | 96 | Please see the notebook inlcuded in this repo for a worked-out example of how 97 | to use `GradREC`! 98 | 99 | 100 | ### GradREC Demo 101 | 102 | The demo is built using streamlit, with further instructions and explanations included 103 | inside. 104 | 105 | Running the app requires access to the dataset/fine-tuned model. See below for a preview 106 | of the demo. Stay tuned for more updates! 107 | 108 | ![Alt Text](app/data/demo_eg.gif) 109 | 110 | 111 | #### How to run 112 | ``` 113 | $ cd app 114 | $ streamlit run app.py 115 | ``` 116 | 117 | 118 | ## Citation 119 | ```angular2html 120 | @inproceedings{chia-etal-2022-come, 121 | title = "{``}Does it come in black?{''} {CLIP}-like models are zero-shot recommenders", 122 | author = "Chia, Patrick John and 123 | Tagliabue, Jacopo and 124 | Bianchi, Federico and 125 | Greco, Ciro and 126 | Goncalves, Diogo", 127 | booktitle = "Proceedings of The Fifth Workshop on e-Commerce and NLP (ECNLP 5)", 128 | month = may, 129 | year = "2022", 130 | address = "Dublin, Ireland", 131 | publisher = "Association for Computational Linguistics", 132 | url = "https://aclanthology.org/2022.ecnlp-1.22", 133 | doi = "10.18653/v1/2022.ecnlp-1.22", 134 | pages = "191--198", 135 | abstract = "Product discovery is a crucial component for online shopping. However, item-to-item recommendations today do not allow users to explore changes along selected dimensions: given a query item, can a model suggest something similar but in a different color? We consider item recommendations of the comparative nature (e.g. {``}something darker{''}) and show how CLIP-based models can support this use case in a zero-shot manner. Leveraging a large model built for fashion, we introduce GradREC and its industry potential, and offer a first rounded assessment of its strength and weaknesses.", 136 | } 137 | ``` -------------------------------------------------------------------------------- /app/app.py: -------------------------------------------------------------------------------- 1 | import streamlit as st 2 | import grad_rec_demo 3 | import intro 4 | 5 | from fashion_clip.fashion_clip import FCLIPDataset, FashionCLIP 6 | from gradient_rec import GradREC 7 | from dotenv import load_dotenv 8 | 9 | load_dotenv("../.env") 10 | 11 | PAGES = { 12 | "Intro": intro, 13 | "Demo": grad_rec_demo, 14 | } 15 | 16 | @st.cache(allow_output_mutation=True) 17 | def load_model(): 18 | dataset = FCLIPDataset('FF', 19 | image_source_path='s3://farfetch-images-ztapq86olwi6kub2p79d/images/', 20 | image_source_type='S3') 21 | fclip_model = FashionCLIP('FCLIP', dataset) 22 | grad_rec_model = GradREC(fclip_model) 23 | print('DONE LOADING MODEL') 24 | return dataset, fclip_model, grad_rec_model 25 | 26 | page = st.sidebar.selectbox("", list(PAGES.keys())) 27 | if page == "Intro": 28 | PAGES[page].app() 29 | else: 30 | DATASET, FCLIP, GRADREC = load_model() 31 | PAGES[page].app(DATASET, FCLIP, GRADREC) 32 | -------------------------------------------------------------------------------- /app/data/GradREC_gif.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patrickjohncyh/gradient-recs/16f3d7f960a3beeb6fbb2b46a490be12500ed54e/app/data/GradREC_gif.gif -------------------------------------------------------------------------------- /app/data/GradREC_gif_mini.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patrickjohncyh/gradient-recs/16f3d7f960a3beeb6fbb2b46a490be12500ed54e/app/data/GradREC_gif_mini.gif -------------------------------------------------------------------------------- /app/data/GradREC_traversal_vector_gif.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patrickjohncyh/gradient-recs/16f3d7f960a3beeb6fbb2b46a490be12500ed54e/app/data/GradREC_traversal_vector_gif.gif -------------------------------------------------------------------------------- /app/data/GradREC_traversal_vector_gif_mini.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patrickjohncyh/gradient-recs/16f3d7f960a3beeb6fbb2b46a490be12500ed54e/app/data/GradREC_traversal_vector_gif_mini.gif -------------------------------------------------------------------------------- /app/data/demo_eg.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/patrickjohncyh/gradient-recs/16f3d7f960a3beeb6fbb2b46a490be12500ed54e/app/data/demo_eg.gif -------------------------------------------------------------------------------- /app/data/grad_rec_examples.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "title": "Shirt Color Luminance", 4 | "seed_sku": "16333728", 5 | "product_description": "Dark Blue Polo", 6 | "start_query": "Dark Blue Polo T-Shirt", 7 | "end_query": "Blue Polo T-Shirt", 8 | "step_size": 3.5, 9 | "is_example": true 10 | }, 11 | { 12 | "title": "Heel Height", 13 | "seed_sku": "16384528", 14 | "product_description": "Red Heels", 15 | "start_query": "Women High Heels", 16 | "end_query": "Women Flats", 17 | "step_size": 2.5, 18 | "is_example": true 19 | }, 20 | { 21 | "title": "Pants Length", 22 | "seed_sku": "16149236", 23 | "product_description": "Pink Joggers", 24 | "start_query": "Long Pants", 25 | "end_query": "Shorts", 26 | "step_size": 4.0, 27 | "is_example": true 28 | }, 29 | { 30 | "seed_sku": "16476835", 31 | "product_description": "Brown Mule Sandals", 32 | "is_example": false 33 | }, 34 | { 35 | "seed_sku": "15497006", 36 | "product_description": "Winter Coat", 37 | "is_example": false 38 | } 39 | ] -------------------------------------------------------------------------------- /app/data/tsne_fclip_blue_shirt.csv: -------------------------------------------------------------------------------- 1 | ,x,y,Category 2 | 0,-17.05594,15.277751,Dark Blue Polo T-Shirt 3 | 1,-15.643158,17.4543,Dark Blue Polo T-Shirt 4 | 2,-10.70574,14.93919,Dark Blue Polo T-Shirt 5 | 3,-9.584839,13.565543,Dark Blue Polo T-Shirt 6 | 4,-10.142312,14.703189,Dark Blue Polo T-Shirt 7 | 5,-8.242736,16.578497,Dark Blue Polo T-Shirt 8 | 6,-14.304222,17.249172,Dark Blue Polo T-Shirt 9 | 7,-17.132658,6.3409615,Dark Blue Polo T-Shirt 10 | 8,-14.29445,9.085008,Dark Blue Polo T-Shirt 11 | 9,-13.641582,16.02821,Dark Blue Polo T-Shirt 12 | 10,-15.595378,12.665338,Dark Blue Polo T-Shirt 13 | 11,-18.556253,17.663223,Dark Blue Polo T-Shirt 14 | 12,-15.19815,11.218774,Dark Blue Polo T-Shirt 15 | 13,-11.716849,8.709025,Dark Blue Polo T-Shirt 16 | 14,-12.053751,10.561135,Dark Blue Polo T-Shirt 17 | 15,-19.740578,15.581885,Dark Blue Polo T-Shirt 18 | 16,-14.241628,14.288636,Dark Blue Polo T-Shirt 19 | 17,-15.132595,8.551839,Dark Blue Polo T-Shirt 20 | 18,-13.278137,13.680922,Dark Blue Polo T-Shirt 21 | 19,-12.885351,7.9905562,Dark Blue Polo T-Shirt 22 | 20,-19.297028,16.943922,Dark Blue Polo T-Shirt 23 | 21,-12.694382,17.024336,Dark Blue Polo T-Shirt 24 | 22,-15.811055,16.209599,Dark Blue Polo T-Shirt 25 | 23,-16.123823,11.018021,Dark Blue Polo T-Shirt 26 | 24,-14.156363,10.467665,Dark Blue Polo T-Shirt 27 | 25,-13.923556,15.581078,Dark Blue Polo T-Shirt 28 | 26,-13.443067,8.381433,Dark Blue Polo T-Shirt 29 | 27,-16.784594,16.443354,Dark Blue Polo T-Shirt 30 | 28,-14.236089,6.6945143,Dark Blue Polo T-Shirt 31 | 29,-12.937062,15.591791,Dark Blue Polo T-Shirt 32 | 30,-15.537483,8.380934,Dark Blue Polo T-Shirt 33 | 31,-12.688154,14.984562,Dark Blue Polo T-Shirt 34 | 32,-13.058635,12.65595,Dark Blue Polo T-Shirt 35 | 33,-12.271379,10.891189,Dark Blue Polo T-Shirt 36 | 34,-15.888038,14.705737,Dark Blue Polo T-Shirt 37 | 35,-17.844387,12.678989,Dark Blue Polo T-Shirt 38 | 36,-14.295393,12.424546,Dark Blue Polo T-Shirt 39 | 37,-15.907895,13.410493,Dark Blue Polo T-Shirt 40 | 38,-8.071729,10.919954,Dark Blue Polo T-Shirt 41 | 39,-15.525278,8.711081,Dark Blue Polo T-Shirt 42 | 40,-15.772931,9.796755,Dark Blue Polo T-Shirt 43 | 41,-17.522184,10.012255,Dark Blue Polo T-Shirt 44 | 42,-10.435285,17.819914,Dark Blue Polo T-Shirt 45 | 43,-16.388685,7.406678,Dark Blue Polo T-Shirt 46 | 44,-10.611413,9.860755,Dark Blue Polo T-Shirt 47 | 45,-14.317795,13.009608,Dark Blue Polo T-Shirt 48 | 46,-11.921459,13.787853,Dark Blue Polo T-Shirt 49 | 47,-17.52907,7.249553,Dark Blue Polo T-Shirt 50 | 48,-13.275575,13.441176,Dark Blue Polo T-Shirt 51 | 49,-17.097307,13.545274,Dark Blue Polo T-Shirt 52 | 50,-12.146305,9.436496,Dark Blue Polo T-Shirt 53 | 51,-10.9286,10.668259,Dark Blue Polo T-Shirt 54 | 52,-14.062079,11.0776005,Dark Blue Polo T-Shirt 55 | 53,-10.382474,6.219318,Dark Blue Polo T-Shirt 56 | 54,-15.037696,15.439652,Dark Blue Polo T-Shirt 57 | 55,-17.1172,11.112967,Dark Blue Polo T-Shirt 58 | 56,-10.253845,8.66193,Dark Blue Polo T-Shirt 59 | 57,-19.139032,11.569437,Dark Blue Polo T-Shirt 60 | 58,11.766625,5.0960135,Dark Blue Polo T-Shirt 61 | 59,-12.610386,12.366625,Dark Blue Polo T-Shirt 62 | 60,11.728579,5.2060065,Dark Blue Polo T-Shirt 63 | 61,-10.71256,17.1263,Dark Blue Polo T-Shirt 64 | 62,-17.98023,14.258044,Dark Blue Polo T-Shirt 65 | 63,-11.366644,7.367574,Dark Blue Polo T-Shirt 66 | 64,-19.594458,13.618346,Dark Blue Polo T-Shirt 67 | 65,-10.865177,13.319805,Dark Blue Polo T-Shirt 68 | 66,-16.381113,12.372597,Dark Blue Polo T-Shirt 69 | 67,-15.910462,13.961919,Dark Blue Polo T-Shirt 70 | 68,-11.335881,11.825872,Dark Blue Polo T-Shirt 71 | 69,-14.2285595,5.2366877,Dark Blue Polo T-Shirt 72 | 70,-12.397056,13.337077,Dark Blue Polo T-Shirt 73 | 71,-13.478248,6.5808883,Dark Blue Polo T-Shirt 74 | 72,-10.358045,12.842708,Dark Blue Polo T-Shirt 75 | 73,-14.810872,14.349881,Dark Blue Polo T-Shirt 76 | 74,-7.8416348,14.245423,Dark Blue Polo T-Shirt 77 | 75,1.7978792,-3.0796556,Blue Polo T-Shirt 78 | 76,-1.9113743,-5.164208,Blue Polo T-Shirt 79 | 77,2.1397438,-1.3019812,Blue Polo T-Shirt 80 | 78,4.117008,-2.030548,Blue Polo T-Shirt 81 | 79,10.552501,1.9683052,Blue Polo T-Shirt 82 | 80,1.2742043,-0.006384938,Blue Polo T-Shirt 83 | 81,5.959586,1.2894058,Blue Polo T-Shirt 84 | 82,3.8033156,-0.54187745,Blue Polo T-Shirt 85 | 83,2.5379994,-3.555234,Light Blue Polo T-Shirt 86 | 84,7.4577394,1.5411556,Blue Polo T-Shirt 87 | 85,2.679446,-1.9844432,Blue Polo T-Shirt 88 | 86,2.575045,1.4465767,Blue Polo T-Shirt 89 | 87,7.9083443,1.0809798,Blue Polo T-Shirt 90 | 88,-6.387135,3.5212479,Blue Polo T-Shirt 91 | 89,9.48638,0.38817102,Blue Polo T-Shirt 92 | 90,5.029437,0.088591345,Blue Polo T-Shirt 93 | 91,0.25928894,3.6567242,Blue Polo T-Shirt 94 | 92,12.731136,1.9867386,Blue Polo T-Shirt 95 | 93,0.5248418,0.91884786,Blue Polo T-Shirt 96 | 94,-3.0164309,-1.9383484,Blue Polo T-Shirt 97 | 95,15.195465,0.13590832,Light Blue Polo T-Shirt 98 | 96,1.7619165,4.0841403,Blue Polo T-Shirt 99 | 97,-3.5190628,-0.52195066,Blue Polo T-Shirt 100 | 98,11.3164625,-5.2527966,Blue Polo T-Shirt 101 | 99,-3.064936,1.6436622,Blue Polo T-Shirt 102 | 100,3.4139686,3.3642724,Blue Polo T-Shirt 103 | 101,-2.0710993,-9.803748,Blue Polo T-Shirt 104 | 102,-9.171863,10.154752,Blue Polo T-Shirt 105 | 103,1.6775417,0.90059996,Blue Polo T-Shirt 106 | 104,-0.090172656,0.24068272,Blue Polo T-Shirt 107 | 105,-0.02461648,-4.122649,Blue Polo T-Shirt 108 | 106,14.057573,1.0632899,Blue Polo T-Shirt 109 | 107,0.056659043,0.86577946,Blue Polo T-Shirt 110 | 108,-1.450534,4.172601,Blue Polo T-Shirt 111 | 109,-0.59292364,-2.1486316,Blue Polo T-Shirt 112 | 110,-3.5138917,2.0417697,Blue Polo T-Shirt 113 | 111,3.8711116,1.8895642,Blue Polo T-Shirt 114 | 112,-4.6301975,-0.24473786,Blue Polo T-Shirt 115 | 113,0.48808154,-4.115183,Blue Polo T-Shirt 116 | 114,11.707916,-1.3426782,Blue Polo T-Shirt 117 | 115,-0.39320493,-2.5363505,Blue Polo T-Shirt 118 | 116,-2.6868284,-4.121857,Blue Polo T-Shirt 119 | 117,1.5933579,-0.67973167,Blue Polo T-Shirt 120 | 118,-0.74713516,-1.0576816,Blue Polo T-Shirt 121 | 119,1.1562034,1.4909724,Blue Polo T-Shirt 122 | 120,4.3243227,1.8636594,Blue Polo T-Shirt 123 | 121,2.9032438,-1.0701271,Blue Polo T-Shirt 124 | 122,-5.692723,2.5376086,Blue Polo T-Shirt 125 | 123,11.254777,-5.1648693,Blue Polo T-Shirt 126 | 124,-5.7966957,-1.9076961,Blue Polo T-Shirt 127 | 125,-0.6702564,2.4370542,Blue Polo T-Shirt 128 | 126,-2.4656546,4.8276052,Blue Polo T-Shirt 129 | 127,2.6282601,0.22948503,Blue Polo T-Shirt 130 | 128,9.67079,2.4279256,Blue Polo T-Shirt 131 | 129,-8.09793,7.4803534,Blue Polo T-Shirt 132 | 130,1.8525237,2.538708,Blue Polo T-Shirt 133 | 131,13.997181,1.2397349,Blue Polo T-Shirt 134 | 132,-4.9019585,2.5539343,Blue Polo T-Shirt 135 | 133,-4.075349,-1.396714,Blue Polo T-Shirt 136 | 134,9.591652,-2.4179718,Blue Polo T-Shirt 137 | 135,2.859379,-0.06713286,Blue Polo T-Shirt 138 | 136,11.485649,-5.6944904,Blue Polo T-Shirt 139 | 137,4.395482,-3.7330778,Blue Polo T-Shirt 140 | 138,-0.37331858,5.323092,Blue Polo T-Shirt 141 | 139,-1.2606287,0.2660657,Blue Polo T-Shirt 142 | 140,-6.149285,4.724071,Blue Polo T-Shirt 143 | 141,2.1190834,1.4232506,Blue Polo T-Shirt 144 | 142,7.061392,-0.6572423,Blue Polo T-Shirt 145 | 143,1.1701138,-1.4712037,Blue Polo T-Shirt 146 | 144,10.066358,1.8806269,Blue Polo T-Shirt 147 | 145,4.930096,-1.495009,Blue Polo T-Shirt 148 | 146,-1.854929,-1.1853125,Blue Polo T-Shirt 149 | 147,0.461492,2.8523757,Blue Polo T-Shirt 150 | 148,3.419839,-5.855317,Light Blue Polo T-Shirt 151 | 149,-5.985063,0.022657322,Blue Polo T-Shirt 152 | 150,15.835698,-0.60327256,Light Blue Polo T-Shirt 153 | 151,15.131699,-3.0788143,Light Blue Polo T-Shirt 154 | 152,11.407346,-8.730622,Light Blue Polo T-Shirt 155 | 153,9.629819,-8.655423,Light Blue Polo T-Shirt 156 | 154,3.361909,-13.522162,Light Blue Polo T-Shirt 157 | 155,6.115406,-9.743488,Light Blue Polo T-Shirt 158 | 156,13.087893,-1.5554916,Light Blue Polo T-Shirt 159 | 157,13.476703,-5.9489183,Light Blue Polo T-Shirt 160 | 158,4.0623055,-13.57294,Light Blue Polo T-Shirt 161 | 159,4.5344815,-6.6934996,Light Blue Polo T-Shirt 162 | 160,5.07922,-10.842265,Light Blue Polo T-Shirt 163 | 161,1.9844146,-11.68071,Light Blue Polo T-Shirt 164 | 162,13.973948,-3.0897403,Light Blue Polo T-Shirt 165 | 163,15.910545,-11.32654,Light Blue Polo T-Shirt 166 | 164,15.5776005,-11.560812,Light Blue Polo T-Shirt 167 | 165,14.142944,-1.8166269,Light Blue Polo T-Shirt 168 | 166,7.805079,-8.6163225,Light Blue Polo T-Shirt 169 | 167,8.56889,-12.252618,Light Blue Polo T-Shirt 170 | 168,3.8990533,-12.534136,Light Blue Polo T-Shirt 171 | 169,8.259746,-9.4874115,Light Blue Polo T-Shirt 172 | 170,7.096725,-13.54799,Light Blue Polo T-Shirt 173 | 171,14.698552,-4.4052615,Light Blue Polo T-Shirt 174 | 172,12.238621,-5.751015,Light Blue Polo T-Shirt 175 | 173,5.677124,-13.767665,Light Blue Polo T-Shirt 176 | 174,14.912369,-3.3205094,Light Blue Polo T-Shirt 177 | 175,15.687846,-12.133735,Light Blue Polo T-Shirt 178 | 176,8.541653,-11.752457,Light Blue Polo T-Shirt 179 | 177,3.5430849,-11.309573,Light Blue Polo T-Shirt 180 | 178,1.2463125,-15.447924,Light Blue Polo T-Shirt 181 | 179,0.48139215,-11.728397,Light Blue Polo T-Shirt 182 | 180,2.7918904,-10.940899,Light Blue Polo T-Shirt 183 | 181,15.203463,-12.5939245,Light Blue Polo T-Shirt 184 | 182,4.793413,-12.6467285,Light Blue Polo T-Shirt 185 | 183,4.6483564,-13.856709,Light Blue Polo T-Shirt 186 | 184,7.8284283,-9.450815,Light Blue Polo T-Shirt 187 | 185,11.958345,-8.194777,Light Blue Polo T-Shirt 188 | 186,13.921505,-1.4527653,Light Blue Polo T-Shirt 189 | 187,16.933151,-1.2487923,Light Blue Polo T-Shirt 190 | 188,2.5806274,-8.502298,Light Blue Polo T-Shirt 191 | 189,16.317753,-2.2231162,Light Blue Polo T-Shirt 192 | 190,3.777024,-10.274745,Light Blue Polo T-Shirt 193 | 191,3.419839,-5.855317,Light Blue Polo T-Shirt 194 | 192,5.7629457,-11.79425,Light Blue Polo T-Shirt 195 | 193,4.827107,-10.03825,Light Blue Polo T-Shirt 196 | 194,2.7187557,-12.366612,Light Blue Polo T-Shirt 197 | 195,2.3099973,-5.791154,Light Blue Polo T-Shirt 198 | 196,15.195465,0.13590832,Light Blue Polo T-Shirt 199 | 197,-1.6430299,-11.17896,Light Blue Polo T-Shirt 200 | 198,16.270191,0.29444999,Light Blue Polo T-Shirt 201 | 199,17.782986,-2.0559554,Light Blue Polo T-Shirt 202 | 200,3.2738166,-8.590168,Light Blue Polo T-Shirt 203 | 201,12.749478,-3.3384902,Light Blue Polo T-Shirt 204 | 202,14.925398,-0.76347625,Light Blue Polo T-Shirt 205 | 203,7.2751307,-11.619801,Light Blue Polo T-Shirt 206 | 204,2.5379994,-3.555234,Light Blue Polo T-Shirt 207 | 205,11.997183,-7.844407,Light Blue Polo T-Shirt 208 | 206,4.729904,-11.642058,Light Blue Polo T-Shirt 209 | 207,6.616534,-11.797326,Light Blue Polo T-Shirt 210 | 208,16.659012,-2.5156105,Light Blue Polo T-Shirt 211 | 209,5.491069,-10.636182,Light Blue Polo T-Shirt 212 | 210,10.694689,-2.2808435,Light Blue Polo T-Shirt 213 | 211,-1.0874171,-11.413772,Light Blue Polo T-Shirt 214 | 212,5.306825,-8.052219,Light Blue Polo T-Shirt 215 | 213,5.3156567,-6.0726995,Light Blue Polo T-Shirt 216 | 214,15.156926,-12.596803,Light Blue Polo T-Shirt 217 | 215,1.3650204,-9.673095,Light Blue Polo T-Shirt 218 | 216,14.100738,-7.727695,Light Blue Polo T-Shirt 219 | 217,13.078316,-0.81863177,Light Blue Polo T-Shirt 220 | 218,0.7820657,-8.976066,Light Blue Polo T-Shirt 221 | 219,15.00869,-5.527535,Light Blue Polo T-Shirt 222 | 220,1.309861,-15.509447,Light Blue Polo T-Shirt 223 | 221,-1.4009761,-10.84253,Light Blue Polo T-Shirt 224 | -------------------------------------------------------------------------------- /app/data/tsne_fclip_shoes.csv: -------------------------------------------------------------------------------- 1 | ,x,y,Category 2 | 0,-15.010797,4.5890284,Sandals 3 | 1,-14.779888,4.234241,Sandals 4 | 2,-21.448315,3.9063215,Sandals 5 | 3,-19.082443,6.3363147,Sandals 6 | 4,-17.669138,3.5243328,Sandals 7 | 5,-11.658825,5.5322375,Sandals 8 | 6,-16.166521,7.229133,Sandals 9 | 7,-19.427534,8.25375,Sandals 10 | 8,-12.094136,8.7716675,Sandals 11 | 9,-19.018745,4.574078,Sandals 12 | 10,-17.468784,3.9641213,Sandals 13 | 11,-19.23921,7.1553726,Sandals 14 | 12,-10.8289995,4.28813,Sandals 15 | 13,-20.526102,4.189574,Sandals 16 | 14,-21.018005,6.50643,Sandals 17 | 15,-12.263525,9.505483,Sandals 18 | 16,-16.48029,5.660445,Sandals 19 | 17,-14.349368,8.77419,Sandals 20 | 18,-17.611841,5.0055223,Sandals 21 | 19,-16.255745,8.214719,Sandals 22 | 20,-13.08252,4.388797,Sandals 23 | 21,-13.789499,6.0066504,Sandals 24 | 22,-17.8132,3.0019898,Sandals 25 | 23,-11.983147,5.3021374,Sandals 26 | 24,-16.705385,9.577329,Sandals 27 | 25,-17.743366,8.826171,Sandals 28 | 26,-16.368603,11.08414,Sandals 29 | 27,-7.563687,6.775731,Sandals 30 | 28,-11.254203,8.155114,Sandals 31 | 29,-17.464874,5.886648,Sandals 32 | 30,-16.421238,6.343374,Sandals 33 | 31,-13.101817,3.341861,Sandals 34 | 32,-14.872627,6.763342,Sandals 35 | 33,-14.079145,8.915151,Sandals 36 | 34,-12.619409,2.602977,Sandals 37 | 35,-14.562682,5.8747897,Sandals 38 | 36,-14.1100025,4.5948234,Sandals 39 | 37,-11.90553,8.706348,Sandals 40 | 38,-7.4280386,4.513593,Sandals 41 | 39,-12.765278,7.9202766,Sandals 42 | 40,-14.70195,2.5236664,Sandals 43 | 41,-14.116882,3.8434193,Sandals 44 | 42,-17.97435,6.865042,Sandals 45 | 43,-16.57776,11.769702,Sandals 46 | 44,-20.71222,3.61445,Sandals 47 | 45,-14.294954,7.0348496,Sandals 48 | 46,-20.545107,9.041846,Sandals 49 | 47,-15.747871,6.918538,Sandals 50 | 48,-8.227158,10.289023,Sandals 51 | 49,-18.224144,4.9087877,Sandals 52 | 50,-7.6130543,4.772893,Sandals 53 | 51,-16.472088,11.562887,Sandals 54 | 52,-21.584593,4.3881354,Sandals 55 | 53,-20.16226,5.6966624,Sandals 56 | 54,-18.891293,5.1047473,Sandals 57 | 55,-20.33278,2.531126,Sandals 58 | 56,-10.89614,4.336592,Sandals 59 | 57,-14.6906805,7.6927576,Sandals 60 | 58,-8.296406,10.366029,Sandals 61 | 59,-17.365143,4.7221637,Sandals 62 | 60,-9.339321,6.3140883,Sandals 63 | 61,-19.52823,5.577203,Sandals 64 | 62,-11.9771805,5.2840943,Sandals 65 | 63,-8.309052,10.382038,Sandals 66 | 64,-13.48266,3.6936564,Sandals 67 | 65,-17.263218,7.6564765,Sandals 68 | 66,-7.596642,6.738202,Sandals 69 | 67,-16.211557,5.889729,Sandals 70 | 68,-16.722729,2.8886507,Sandals 71 | 69,-18.28589,5.867668,Sandals 72 | 70,-15.928436,9.557389,Sandals 73 | 71,-16.725048,4.55006,Sandals 74 | 72,-17.9068,6.816841,Sandals 75 | 73,-15.368489,8.001443,Sandals 76 | 74,-16.078938,4.118269,Sandals 77 | 75,-5.890847,1.0341287,Footwear 78 | 76,-0.7606894,1.3060572,Footwear 79 | 77,-3.9994743,-2.8643782,Footwear 80 | 78,-0.9953849,-0.64097285,Footwear 81 | 79,-3.9513607,6.461034,Footwear 82 | 80,-0.9664325,4.4266076,Footwear 83 | 81,-7.0125403,-1.8412695,Footwear 84 | 82,-3.9904869,1.2489719,Footwear 85 | 83,0.1907751,1.4441205,Footwear 86 | 84,-3.4301717,2.634459,Footwear 87 | 85,-1.423993,5.347801,Footwear 88 | 86,-3.5869164,6.1638775,Footwear 89 | 87,-7.8831153,0.38160524,Footwear 90 | 88,-6.9785123,-6.980046,Footwear 91 | 89,-0.75412893,0.75768137,Footwear 92 | 90,-6.291008,-7.5697775,Footwear 93 | 91,-0.20329134,1.2487705,Footwear 94 | 92,-2.71936,2.6076007,Footwear 95 | 93,-7.630792,-6.2619066,Footwear 96 | 94,-7.332044,-2.681316,Footwear 97 | 95,-5.255707,-0.27927327,Footwear 98 | 96,-6.789331,-4.0794125,Footwear 99 | 97,-6.863856,-5.538023,Footwear 100 | 98,-2.35813,-4.506202,Footwear 101 | 99,-5.2755938,-6.5925746,Footwear 102 | 100,-0.8421282,3.8635845,Footwear 103 | 101,-4.146997,6.616141,Footwear 104 | 102,-8.084199,-4.9437904,Footwear 105 | 103,-0.73746973,1.4299024,Footwear 106 | 104,-2.2493672,1.3156996,Footwear 107 | 105,-6.3891835,-5.7665443,Footwear 108 | 106,-3.1836658,-0.9396402,Footwear 109 | 107,-4.3362103,-6.6692457,Footwear 110 | 108,-7.9144607,-3.330374,Footwear 111 | 109,-5.5676374,3.2859263,Footwear 112 | 110,-7.717628,-1.345243,Footwear 113 | 111,-0.2750072,-0.16855389,Footwear 114 | 112,-5.6699443,-0.7754393,Footwear 115 | 113,-4.0534163,-0.99313337,Footwear 116 | 114,-4.013871,-2.9185567,Footwear 117 | 115,-3.9051387,-0.31553105,Footwear 118 | 116,-8.433369,-7.388967,Footwear 119 | 117,-2.850527,1.0615159,Footwear 120 | 118,-0.83573836,0.20519885,Footwear 121 | 119,-7.9139686,-1.0921752,Footwear 122 | 120,1.8291273,1.6185493,Footwear 123 | 121,-3.915837,-7.66731,Footwear 124 | 122,12.307138,-6.385041,Footwear 125 | 123,0.52429456,-2.4949458,Footwear 126 | 124,0.356044,-0.3337288,Footwear 127 | 125,12.752764,-6.4705305,Footwear 128 | 126,-0.08035428,0.46245727,Footwear 129 | 127,0.26106083,-3.3523314,Footwear 130 | 128,-6.7671466,-4.348802,Footwear 131 | 129,-6.442276,-5.798387,Footwear 132 | 130,11.507452,-6.268568,Footwear 133 | 131,-9.278063,-2.286605,Footwear 134 | 132,-1.7860016,2.21864,Footwear 135 | 133,-5.5284133,-0.34560636,Footwear 136 | 134,-6.486552,-7.470903,Footwear 137 | 135,1.0845321,0.73830056,Footwear 138 | 136,-4.216113,-0.104978256,Footwear 139 | 137,-7.405845,10.152274,Footwear 140 | 138,-2.4953666,1.1872301,Footwear 141 | 139,-3.9475749,-5.9924655,Footwear 142 | 140,-6.019023,-0.025732208,Footwear 143 | 141,1.9100375,0.10497865,Footwear 144 | 142,-1.0363196,-2.000445,Footwear 145 | 143,0.5286608,2.747639,Footwear 146 | 144,-0.96146494,2.6594663,Footwear 147 | 145,-7.650643,-7.697514,Footwear 148 | 146,-6.5497804,-7.455147,Footwear 149 | 147,1.184314,-1.4943849,Footwear 150 | 148,-2.784697,3.4622207,Footwear 151 | 149,-2.571337,2.9373362,Footwear 152 | 150,23.78686,-7.970794,Dress Shoes 153 | 151,17.203669,-4.5239096,Dress Shoes 154 | 152,23.133728,-8.195884,Dress Shoes 155 | 153,23.197952,-10.246191,Dress Shoes 156 | 154,16.78945,-7.6165314,Dress Shoes 157 | 155,25.360865,-1.8317837,Dress Shoes 158 | 156,19.504297,-3.8389866,Dress Shoes 159 | 157,21.52182,-7.838454,Dress Shoes 160 | 158,24.631731,-9.4999275,Dress Shoes 161 | 159,18.390951,-6.3138566,Dress Shoes 162 | 160,24.735037,-7.8021564,Dress Shoes 163 | 161,20.406637,-6.8381424,Dress Shoes 164 | 162,24.918756,-1.5733966,Dress Shoes 165 | 163,20.997469,-8.784358,Dress Shoes 166 | 164,21.786514,-8.370912,Dress Shoes 167 | 165,23.511477,-2.6126657,Dress Shoes 168 | 166,22.202671,-3.6189034,Dress Shoes 169 | 167,19.017883,-9.366005,Dress Shoes 170 | 168,22.485228,-8.013358,Dress Shoes 171 | 169,22.593061,-9.195555,Dress Shoes 172 | 170,21.163675,-6.013562,Dress Shoes 173 | 171,22.655039,-7.0980215,Dress Shoes 174 | 172,19.612139,-10.882344,Dress Shoes 175 | 173,25.004799,-4.0273705,Dress Shoes 176 | 174,22.340714,-7.307522,Dress Shoes 177 | 175,24.975508,-2.728119,Dress Shoes 178 | 176,26.51306,-6.693518,Dress Shoes 179 | 177,17.846231,-8.756912,Dress Shoes 180 | 178,23.402422,-6.717202,Dress Shoes 181 | 179,20.28459,-4.9932194,Dress Shoes 182 | 180,21.114756,-4.2198124,Dress Shoes 183 | 181,24.29695,-1.3208741,Dress Shoes 184 | 182,17.42131,-7.36276,Dress Shoes 185 | 183,19.144278,-5.2324557,Dress Shoes 186 | 184,21.00589,-7.917274,Dress Shoes 187 | 185,21.538681,-5.6539927,Dress Shoes 188 | 186,26.089823,-8.880847,Dress Shoes 189 | 187,23.627455,-11.821115,Dress Shoes 190 | 188,15.910412,-7.439475,Dress Shoes 191 | 189,24.169641,-5.90484,Dress Shoes 192 | 190,25.685797,-1.2066505,Dress Shoes 193 | 191,22.973923,-9.28762,Dress Shoes 194 | 192,25.8228,-2.947845,Dress Shoes 195 | 193,24.249304,-2.5429883,Dress Shoes 196 | 194,24.740255,-6.964723,Dress Shoes 197 | 195,19.901325,-8.4666,Dress Shoes 198 | 196,26.98074,-3.8917763,Dress Shoes 199 | 197,15.637216,-7.233885,Dress Shoes 200 | 198,23.35721,-5.316696,Dress Shoes 201 | 199,24.042387,-12.234557,Dress Shoes 202 | 200,25.506968,-0.9503193,Dress Shoes 203 | 201,20.031054,-6.4441657,Dress Shoes 204 | 202,19.287388,-7.3902807,Dress Shoes 205 | 203,23.973783,-4.992083,Dress Shoes 206 | 204,21.211023,-10.006322,Dress Shoes 207 | 205,16.733698,-9.999649,Dress Shoes 208 | 206,21.690018,-10.012011,Dress Shoes 209 | 207,21.556175,-6.1005263,Dress Shoes 210 | 208,23.621498,-12.051014,Dress Shoes 211 | 209,18.654903,-5.835736,Dress Shoes 212 | 210,21.433767,-11.331738,Dress Shoes 213 | 211,26.29603,-3.3510804,Dress Shoes 214 | 212,21.39414,-2.2944584,Dress Shoes 215 | 213,21.163506,-6.6633754,Dress Shoes 216 | 214,21.138666,-11.107319,Dress Shoes 217 | 215,16.885315,-4.7549314,Dress Shoes 218 | 216,21.704767,-6.8381696,Dress Shoes 219 | 217,20.828218,-9.370696,Dress Shoes 220 | 218,22.406063,-4.748797,Dress Shoes 221 | 219,23.988714,-6.219578,Dress Shoes 222 | 220,20.690653,-4.4443445,Dress Shoes 223 | 221,20.354881,-7.623167,Dress Shoes 224 | 222,25.784485,-0.38331857,Dress Shoes 225 | 223,16.036495,-8.67778,Dress Shoes 226 | 224,22.581488,-5.849419,Dress Shoes 227 | -------------------------------------------------------------------------------- /app/data/tsne_fclip_skirt_length.csv: -------------------------------------------------------------------------------- 1 | ,x,y,Category 2 | 0,9.731072,-6.6777296,Short Skirt 3 | 1,18.114798,-3.576326,Short Skirt 4 | 2,11.570949,-9.971787,Short Skirt 5 | 3,12.915526,-15.30006,Short Skirt 6 | 4,14.036375,-2.3231027,Short Skirt 7 | 5,9.883474,-4.2550426,Short Skirt 8 | 6,14.236753,-13.165866,Short Skirt 9 | 7,14.832459,-14.116144,Short Skirt 10 | 8,14.100344,-12.706402,Short Skirt 11 | 9,16.454262,-6.9644876,Short Skirt 12 | 10,9.109681,-4.76347,Short Skirt 13 | 11,13.085969,-6.4553657,Short Skirt 14 | 12,15.815956,-6.521428,Short Skirt 15 | 13,9.261693,-7.3976264,Short Skirt 16 | 14,16.500017,-3.1612675,Short Skirt 17 | 15,13.506629,-14.041532,Short Skirt 18 | 16,4.723845,-11.582674,Short Skirt 19 | 17,16.633038,-3.166199,Short Skirt 20 | 18,12.353477,-11.034141,Short Skirt 21 | 19,18.223507,-9.506669,Short Skirt 22 | 20,16.417768,-10.655224,Short Skirt 23 | 21,20.234125,-10.688416,Short Skirt 24 | 22,8.867842,-8.429001,Short Skirt 25 | 23,6.723511,-10.773395,Short Skirt 26 | 24,10.328525,-12.002694,Short Skirt 27 | 25,9.157858,-12.851906,Short Skirt 28 | 26,11.566632,-13.399378,Short Skirt 29 | 27,11.633548,-3.2495134,Short Skirt 30 | 28,11.534396,-9.248008,Short Skirt 31 | 29,12.58178,-7.4358096,Short Skirt 32 | 30,13.019913,-9.219881,Short Skirt 33 | 31,13.836194,-6.9778733,Short Skirt 34 | 32,11.501793,-12.585853,Short Skirt 35 | 33,12.244869,-12.427067,Short Skirt 36 | 34,14.787611,-11.070805,Short Skirt 37 | 35,11.222081,-14.408208,Short Skirt 38 | 36,15.336505,-8.758395,Short Skirt 39 | 37,13.390233,-15.77444,Short Skirt 40 | 38,11.877284,-3.0418942,Short Skirt 41 | 39,7.864597,0.09095836,Short Skirt 42 | 40,11.1582365,-11.972071,Short Skirt 43 | 41,14.994577,-8.136281,Short Skirt 44 | 42,10.19535,-11.52894,Short Skirt 45 | 43,8.891427,-9.58832,Short Skirt 46 | 44,11.83646,-10.567488,Short Skirt 47 | 45,10.457828,-10.64288,Short Skirt 48 | 46,16.283735,-2.6852872,Short Skirt 49 | 47,19.371815,-10.4844675,Short Skirt 50 | 48,13.142316,-13.68074,Short Skirt 51 | 49,6.7108603,-13.88908,Short Skirt 52 | 50,12.140423,-5.387679,Short Skirt 53 | 51,11.996662,-14.722118,Short Skirt 54 | 52,13.317009,-12.714342,Short Skirt 55 | 53,9.682962,-13.758051,Short Skirt 56 | 54,12.815478,-10.536143,Short Skirt 57 | 55,12.557284,-13.48206,Short Skirt 58 | 56,5.125611,-13.805622,Short Skirt 59 | 57,11.291044,-8.260778,Short Skirt 60 | 58,12.280951,-16.507416,Short Skirt 61 | 59,18.657112,-10.576109,Short Skirt 62 | 60,10.965022,-8.973792,Short Skirt 63 | 61,8.309714,-11.721736,Short Skirt 64 | 62,13.547128,-12.03413,Short Skirt 65 | 63,14.429685,-10.370545,Short Skirt 66 | 64,14.726533,-15.240793,Short Skirt 67 | 65,17.079329,-3.6285126,Short Skirt 68 | 66,16.353123,-4.233526,Short Skirt 69 | 67,14.378712,-2.3583171,Short Skirt 70 | 68,6.137485,-14.282259,Short Skirt 71 | 69,11.3719015,-16.155178,Short Skirt 72 | 70,6.1694736,-9.787726,Short Skirt 73 | 71,10.841411,-12.9333105,Short Skirt 74 | 72,9.691531,-15.272415,Short Skirt 75 | 73,9.026041,-12.449908,Short Skirt 76 | 74,16.527126,-6.8709173,Short Skirt 77 | 75,1.56282,-4.3783574,Midi Skirt 78 | 76,-4.872366,14.996627,Midi Skirt 79 | 77,-1.5861204,5.235099,Midi Skirt 80 | 78,1.2909815,-1.2954471,Midi Skirt 81 | 79,0.21423516,-4.8157225,Midi Skirt 82 | 80,1.4918376,-6.222082,Midi Skirt 83 | 81,3.9494603,-0.10427925,Midi Skirt 84 | 82,3.6768503,-1.3759061,Midi Skirt 85 | 83,-0.74004877,5.4660573,Midi Skirt 86 | 84,2.372471,0.80376285,Midi Skirt 87 | 85,4.933676,1.8476098,Midi Skirt 88 | 86,-4.741607,14.539544,Midi Skirt 89 | 87,-12.363253,14.682093,Midi Skirt 90 | 88,4.5787215,0.5578065,Midi Skirt 91 | 89,-10.798926,14.692745,Midi Skirt 92 | 90,-11.802737,1.5782601,Long Skirt 93 | 91,-5.0047603,15.776786,Midi Skirt 94 | 92,-5.1537704,13.174216,Midi Skirt 95 | 93,-1.1038619,-1.0598925,Midi Skirt 96 | 94,1.9858364,-7.130662,Midi Skirt 97 | 95,5.6359344,-0.6352919,Midi Skirt 98 | 96,-7.3314304,14.409073,Midi Skirt 99 | 97,2.45789,-4.5832086,Midi Skirt 100 | 98,3.2111392,-0.7468807,Midi Skirt 101 | 99,5.141329,-1.2823415,Midi Skirt 102 | 100,2.4965808,-0.8152632,Midi Skirt 103 | 101,-11.420962,14.586033,Midi Skirt 104 | 102,0.25221202,-3.9070764,Midi Skirt 105 | 103,-0.097258866,-4.923916,Midi Skirt 106 | 104,-7.209188,14.043155,Midi Skirt 107 | 105,0.6094677,-8.839394,Midi Skirt 108 | 106,-1.0919377,-1.1444012,Midi Skirt 109 | 107,-6.0064588,15.506646,Midi Skirt 110 | 108,-4.314995,-4.5097866,Midi Skirt 111 | 109,1.2567455,2.8194478,Midi Skirt 112 | 110,0.34918126,-2.658564,Midi Skirt 113 | 111,0.1724916,-7.507131,Midi Skirt 114 | 112,-15.533025,2.09415,Midi Skirt 115 | 113,3.4332259,-5.0580826,Midi Skirt 116 | 114,-18.234303,3.6272552,Midi Skirt 117 | 115,-4.5026293,14.96654,Midi Skirt 118 | 116,-0.5439495,-3.9950826,Midi Skirt 119 | 117,1.9785125,1.6391681,Midi Skirt 120 | 118,-8.01462,16.396889,Midi Skirt 121 | 119,-3.6960204,14.397169,Midi Skirt 122 | 120,-1.2806661,-6.407018,Midi Skirt 123 | 121,-10.611496,-4.075881,Midi Skirt 124 | 122,-5.543162,-2.0162973,Midi Skirt 125 | 123,1.6984189,2.9726853,Midi Skirt 126 | 124,1.2803323,-2.884188,Midi Skirt 127 | 125,-4.433107,11.208738,Midi Skirt 128 | 126,1.723239,-0.7748812,Midi Skirt 129 | 127,3.3874035,-7.461976,Midi Skirt 130 | 128,1.2494643,-5.1144934,Midi Skirt 131 | 129,-0.7736567,0.85639495,Midi Skirt 132 | 130,5.5019245,0.104163,Midi Skirt 133 | 131,-1.9325691,-4.7040243,Midi Skirt 134 | 132,-2.2893372,-0.29618716,Midi Skirt 135 | 133,-2.370317,-3.0937507,Long Skirt 136 | 134,2.3216224,-5.5888867,Midi Skirt 137 | 135,-7.4201,-3.0374248,Midi Skirt 138 | 136,-15.643473,1.7747955,Midi Skirt 139 | 137,-9.647762,15.734284,Midi Skirt 140 | 138,5.8054714,-1.5992832,Midi Skirt 141 | 139,0.7034816,-6.5695176,Midi Skirt 142 | 140,4.975691,-2.1683557,Midi Skirt 143 | 141,2.5025363,-3.8603003,Midi Skirt 144 | 142,3.6945195,-7.91369,Midi Skirt 145 | 143,4.1527143,-2.7911236,Midi Skirt 146 | 144,-12.667677,11.960879,Midi Skirt 147 | 145,1.7438921,1.041812,Midi Skirt 148 | 146,2.9426441,-7.335671,Midi Skirt 149 | 147,7.46896,-1.0000883,Midi Skirt 150 | 148,-4.88049,13.742134,Midi Skirt 151 | 149,0.6861427,-0.7592874,Midi Skirt 152 | 150,-7.486808,10.269816,Long Skirt 153 | 151,-7.2288284,4.899923,Long Skirt 154 | 152,-15.101378,3.427493,Long Skirt 155 | 153,-7.2797003,0.2862819,Long Skirt 156 | 154,-12.401872,4.4309993,Long Skirt 157 | 155,-9.9170475,10.441819,Long Skirt 158 | 156,-17.184319,2.6543982,Long Skirt 159 | 157,-7.0487003,1.7071338,Long Skirt 160 | 158,-9.114664,12.2172985,Long Skirt 161 | 159,-7.429446,10.074997,Long Skirt 162 | 160,-13.502958,6.7809258,Long Skirt 163 | 161,-9.220712,12.061296,Long Skirt 164 | 162,-8.686619,1.7877907,Long Skirt 165 | 163,-8.786627,4.6252613,Long Skirt 166 | 164,-8.057854,3.789531,Long Skirt 167 | 165,-14.7840185,7.104374,Long Skirt 168 | 166,-6.9540434,2.2445087,Long Skirt 169 | 167,-6.3536644,1.6969609,Long Skirt 170 | 168,-7.008644,3.3644004,Long Skirt 171 | 169,-2.821259,2.0003543,Long Skirt 172 | 170,-11.804617,10.648098,Long Skirt 173 | 171,-3.634942,0.23740497,Long Skirt 174 | 172,-6.097986,0.4027862,Long Skirt 175 | 173,-16.098188,3.399409,Long Skirt 176 | 174,-13.208172,10.134532,Long Skirt 177 | 175,-11.801921,1.5781081,Long Skirt 178 | 176,-8.427789,4.261424,Long Skirt 179 | 177,-5.8883963,8.10017,Long Skirt 180 | 178,-11.133307,4.0772853,Long Skirt 181 | 179,-5.4844527,1.6687071,Long Skirt 182 | 180,-11.5604925,11.35141,Long Skirt 183 | 181,-13.216326,2.6861758,Long Skirt 184 | 182,-7.7038403,2.5954912,Long Skirt 185 | 183,-6.5334277,5.1103406,Long Skirt 186 | 184,-14.345552,6.9346037,Long Skirt 187 | 185,-7.7583823,13.583935,Long Skirt 188 | 186,-8.992201,-0.38213167,Long Skirt 189 | 187,-4.7980123,4.4152493,Long Skirt 190 | 188,-5.7994175,3.1584272,Long Skirt 191 | 189,-13.8854265,1.2509996,Long Skirt 192 | 190,-6.2984014,11.496375,Long Skirt 193 | 191,-11.144497,13.084984,Long Skirt 194 | 192,-3.4818323,-1.8615625,Long Skirt 195 | 193,-7.5485806,1.41581,Long Skirt 196 | 194,-1.4754466,-3.6978362,Long Skirt 197 | 195,-7.9374,11.364464,Long Skirt 198 | 196,-8.361503,2.1441696,Long Skirt 199 | 197,-13.719901,8.857749,Long Skirt 200 | 198,-9.180294,15.002401,Long Skirt 201 | 199,-5.1322637,10.627503,Long Skirt 202 | 200,-9.8727,13.005935,Long Skirt 203 | 201,-7.442793,13.11157,Long Skirt 204 | 202,-9.80776,11.9457245,Long Skirt 205 | 203,-10.599536,11.630542,Long Skirt 206 | 204,-9.396894,13.4319725,Long Skirt 207 | 205,-11.076631,3.8614671,Long Skirt 208 | 206,-10.627693,-3.980306,Long Skirt 209 | 207,-12.011795,9.141751,Long Skirt 210 | 208,-17.704649,3.8346725,Long Skirt 211 | 209,-6.103835,2.337846,Long Skirt 212 | 210,-7.4371843,5.519689,Long Skirt 213 | 211,-9.614144,5.1579585,Long Skirt 214 | 212,-2.3702028,-3.0936558,Long Skirt 215 | 213,-6.0781946,8.123384,Long Skirt 216 | 214,-12.818332,5.0284376,Long Skirt 217 | 215,-8.853877,0.9687402,Long Skirt 218 | 216,-7.822114,-0.41576424,Long Skirt 219 | 217,-16.953157,3.303128,Long Skirt 220 | 218,-3.8415725,9.072427,Long Skirt 221 | 219,-5.012538,2.7922685,Long Skirt 222 | 220,-3.50646,-0.11556801,Long Skirt 223 | 221,-4.8739715,0.4365729,Long Skirt 224 | 222,-13.249024,1.9162967,Long Skirt 225 | -------------------------------------------------------------------------------- /app/grad_rec_demo.py: -------------------------------------------------------------------------------- 1 | import json 2 | import time 3 | import streamlit as st 4 | from fashion_clip.fashion_clip import FCLIPDataset, FashionCLIP 5 | from fashion_clip.utils import display_images_from_url 6 | from gradient_rec import GradREC 7 | 8 | @st.cache(allow_output_mutation=True, 9 | hash_funcs={ GradREC: lambda _ : _.fclip.model_name }) 10 | def get_direction_vector(gradrec_model: GradREC, start_query: str, end_query:str): 11 | return -gradrec_model.direction_vector(start_query, end_query) 12 | 13 | @st.cache(allow_output_mutation=True, 14 | hash_funcs={ GradREC: lambda _ : _.fclip.model_name }) 15 | def traverse_space(gradrec_model: GradREC, start_point, search_space, v_dir, step_size, steps, reg_space, reg_weight=0.9): 16 | @st.cache(allow_output_mutation=True) 17 | def traverse_space_rec(start_point, v_dir, step_size, steps, reg_weight=0.9): 18 | if steps == 0: 19 | return start_point 20 | return gradrec_model.traversal_fn(start_point=traverse_space_rec(start_point, v_dir, step_size, steps-1, reg_weight,), 21 | v_dir=v_dir, 22 | step_size=step_size, 23 | reg_space=reg_space, 24 | reg_k=100, 25 | reg_weight=reg_weight) 26 | 27 | end_point = traverse_space_rec(start_point, v_dir, step_size, steps, reg_weight) 28 | nn = gradrec_model._k_neighbors(end_point, search_space, k=100) 29 | return nn[:10] 30 | 31 | def expander_generator(dataset:FCLIPDataset, 32 | fclip_model: FashionCLIP, 33 | gradrec_model: GradREC, 34 | title: str, 35 | seed_sku: str, 36 | start_query:str = None, 37 | end_query:str = None, 38 | step_size:float = 2.0, 39 | is_example:bool = False, 40 | **kwargs): 41 | 42 | text_cols = st.columns([1, 1, 0.75, 0.3]) 43 | im_cols = st.columns([1, 1, 1, 1, 1]) 44 | # Text and Step Input 45 | with text_cols[0]: 46 | start_query = st.text_input("Start Query:", value=start_query or '', key=title+'_start_query', disabled=is_example) 47 | with text_cols[1]: 48 | end_query = st.text_input("End Query:", value=end_query or '', key=title+'_end_query', disabled=is_example) 49 | with text_cols[2]: 50 | step_size = st.number_input("Step Size:", 51 | min_value=0.25, 52 | max_value=10.0, 53 | step=0.25, 54 | value=step_size, 55 | key=title+'_step_size', 56 | disabled=is_example) 57 | # GO Button 58 | with text_cols[3]: 59 | st.markdown(''); 60 | st.markdown('') 61 | go = st.button('Go!', key=title+'_button') 62 | 63 | # display seed product 64 | with im_cols[0]: 65 | st.pyplot(display_images_from_url([dataset._retrieve_row(seed_sku)['product_photo_url']])) 66 | 67 | if not start_query and not end_query: 68 | return 69 | if not go: 70 | return 71 | 72 | # Computation 73 | v_dir = get_direction_vector(gradrec_model, start_query, end_query) 74 | seen_prods = [seed_sku] 75 | for i in range(1,len(im_cols)): 76 | with im_cols[i]: 77 | next_product = traverse_space(gradrec_model=gradrec_model, 78 | start_point=fclip_model.image_vectors[dataset.id_to_idx[seed_sku]], 79 | search_space=fclip_model.image_vectors, 80 | v_dir=v_dir, 81 | step_size=step_size, 82 | steps=i, 83 | reg_space=fclip_model.image_vectors, 84 | reg_weight=0.9) 85 | next_prod_skus = [dataset.ids[idx] for idx in next_product if dataset.ids[idx] not in seen_prods] 86 | seen_prods.append(next_prod_skus[0]) 87 | product_info = dataset._retrieve_row(next_prod_skus[0]) 88 | print(product_info['category_level_3']) 89 | st.pyplot(display_images_from_url([product_info['product_photo_url']])) 90 | time.sleep(0.5) 91 | 92 | 93 | 94 | def app(dataset: FCLIPDataset, fclip_model: FashionCLIP, gradrec_model: GradREC): 95 | st.write(""" 96 | # GradREC DEMO 97 | 98 | In this interactive demo, explore the kinds of comparative recommendations `GradREC` is capable of. 99 | 100 | In general, you will need to supply a `start query` and an `end query` whose _semantic difference_ represents 101 | the attribute/dimension you want to vary. For example, the difference between "Dark Blue Polo T-Shirt" and 102 | "Blue Polo T-Shirt" represents the _color luminance_ dimension. In addition, a step size needs to provided, which 103 | controls the strength of attribute change on each step. 104 | 105 | Expand the various sections to view recommendations generated by `GradREC` for various attributes. These come with 106 | pre-filled values to give you an idea of how `GradREC` works. 107 | 108 | In the last section you can select various products and supply your own parameters for generating comparative 109 | recommendations. 110 | 111 | """) 112 | with open('data/grad_rec_examples.json') as f: 113 | examples = json.load(f) 114 | 115 | expanders = [st.expander(eg['title']) for eg in examples if eg['is_example']] 116 | # examples 117 | for idx, (exp, ex)in enumerate(zip(expanders, examples)): 118 | with exp: 119 | expander_generator(dataset, fclip_model, gradrec_model, **ex) 120 | 121 | # try it out yourself 122 | diy_expander = st.expander("Try it yourself!") 123 | with diy_expander: 124 | prod_sel = st.selectbox('Select a Product', [ _['product_description'] for _ in examples]) 125 | seed_sku = list(filter(lambda eg: eg['product_description'] == prod_sel, examples))[0]['seed_sku'] 126 | expander_generator(dataset, fclip_model, gradrec_model, title='diy', seed_sku=seed_sku) -------------------------------------------------------------------------------- /app/gradient_rec.py: -------------------------------------------------------------------------------- 1 | from typing import List, Union 2 | from fashion_clip.fashion_clip import FashionCLIP 3 | import numpy as np 4 | from tqdm import tqdm 5 | 6 | _PROMPT_TEMPLATES = [ 7 | 'a bad photo of a {}.', 8 | 'a photo of many {}.', 9 | 'a photo of the hard to see {}.', 10 | 'a low resolution photo of the {}.', 11 | 'a rendering of a {}.', 12 | 'a bad photo of the {}.', 13 | 'a cropped photo of the {}.', 14 | 'a photo of a hard to see {}.', 15 | 'a bright photo of a {}.', 16 | 'a photo of a clean {}.', 17 | 'a photo of a dirty {}.', 18 | 'a dark photo of the {}.', 19 | 'a drawing of a {}.', 20 | 'a photo of my {}.', 21 | 'a close-up photo of a {}.', 22 | 'a black and white photo of the {}.', 23 | 'a painting of the {}.', 24 | 'a painting of a {}.', 25 | 'a pixelated photo of the {}.', 26 | 'a bright photo of the {}.', 27 | 'a cropped photo of a {}.', 28 | 'a photo of the dirty {}.', 29 | 'a jpeg corrupted photo of a {}.', 30 | 'a blurry photo of the {}.', 31 | 'a photo of the {}.', 32 | 'a good photo of the {}.', 33 | 'a rendering of the {}.', 34 | 'a {} in a video game.', 35 | 'a photo of one {}.', 36 | 'a doodle of a {}.', 37 | 'a close-up photo of the {}.', 38 | 'a photo of a {}.', 39 | 'the origami {}.', 40 | 'the {} in a video game.', 41 | 'a sketch of a {}.', 42 | 'a doodle of the {}.', 43 | 'a origami {}.', 44 | 'a low resolution photo of a {}.', 45 | 'the toy {}.', 46 | 'a rendition of the {}.', 47 | 'a photo of the clean {}.', 48 | 'a photo of a large {}.', 49 | 'a rendition of a {}.', 50 | 'a photo of a nice {}.', 51 | 'a photo of a weird {}.', 52 | 'a blurry photo of a {}.', 53 | 'a cartoon {}.', 54 | 'art of a {}.', 55 | 'a sketch of the {}.', 56 | 'a embroidered {}.', 57 | 'a pixelated photo of a {}.', 58 | 'a jpeg corrupted photo of the {}.', 59 | 'a good photo of a {}.', 60 | 'a photo of the nice {}.', 61 | 'a photo of the small {}.', 62 | 'a photo of the weird {}.', 63 | 'the cartoon {}.', 64 | 'art of the {}.', 65 | 'a drawing of the {}.', 66 | 'a photo of the large {}.', 67 | 'a black and white photo of a {}.', 68 | 'a dark photo of a {}.', 69 | 'graffiti of the {}.', 70 | 'a toy {}.', 71 | 'a photo of a cool {}.', 72 | 'a photo of a small {}.', 73 | ] 74 | 75 | class GradREC: 76 | 77 | def __init__(self, fclip: FashionCLIP): 78 | self.fclip = fclip 79 | 80 | def _encode_queries(self, queries: List[str]): 81 | """ 82 | Encode queries into vector representations whilst applying prompt templates 83 | 84 | :param queries: list of queries to encode 85 | :return: textual vector representation for queries 86 | """ 87 | # TODO: offload prompt template computation to FashionCLIP 88 | # apply prompt templates to query 89 | queries_with_prompt = [[prompt.format(q) for prompt in _PROMPT_TEMPLATES] for q in queries] 90 | # flatten queries with prompt and encode in batch 91 | text_vectors_flat = self.fclip.encode_text([ _ for q in queries_with_prompt for _ in q], batch_size=32) 92 | # un-flatten vectors to correct shape 93 | text_vectors = text_vectors_flat.reshape(len(queries), len(_PROMPT_TEMPLATES), -1) 94 | # compute average for each query 95 | query_vectors = text_vectors.mean(axis=1) 96 | return query_vectors 97 | 98 | def _product_retrieval(self, query_vectors: np.ndarray, N:int =100): 99 | """ 100 | Retrieve product image vectors for query vectors using cosine similarity as the distance measure 101 | 102 | :param query_vectors: vectors used as retrieval keys 103 | :param N: number of products to retrieve 104 | :return: image vector representation 105 | """ 106 | cosine_sim = self.fclip._cosine_similarity(query_vectors, self.fclip.image_vectors, normalize=True) 107 | indices = cosine_sim.argsort()[:, -N:][:, ::-1] 108 | return self.fclip.image_vectors[indices] 109 | 110 | def direction_vector(self, start_query: str, end_query: str, start_N=100, end_N=1000): 111 | """ 112 | Computes direction vector given start and end query 113 | 114 | :param start_query: start query text 115 | :param end_query: end query text 116 | :param start_N: number of products to retrieve for start query 117 | :param end_N: number of products to retrieve for end query 118 | :return: direction vector for traversal 119 | """ 120 | # TODO: Simplify computation by leveraging FashionCLIP methods 121 | # TODO: Generalize to any latent space 122 | # encode start and end queries 123 | query_vectors = self._encode_queries([start_query, end_query]) 124 | # retrieve nearest image vectors 125 | query_im_vectors = self._product_retrieval(query_vectors, N=max(start_N, end_N)) 126 | return self._direction_vector_from_vectors(query_im_vectors[0][:start_N], query_im_vectors[1][:end_N]) 127 | 128 | def _direction_vector_from_vectors(self, 129 | exemplar_vectors: np.ndarray, 130 | pop_vectors: np.ndarray, 131 | normalize:bool = True): 132 | """ 133 | Computes the channel-wise SNR given exemplar and population vectors 134 | 135 | :param exemplar_vectors: exemplar vectors 136 | :param pop_vectors: population vectors 137 | :param normalize: flag to normalize SNR vector 138 | :return: channel-wise SNR vector 139 | """ 140 | 141 | population_mean = np.mean(pop_vectors, axis=0, keepdims=True) 142 | population_std = np.std(pop_vectors, axis=0, keepdims=True) 143 | 144 | deltas = (exemplar_vectors - population_mean) / population_std 145 | delta_mean = np.mean(deltas, axis=0, keepdims=False) 146 | delta_std = np.std(deltas, axis=0, keepdims=False) 147 | theta = delta_mean / delta_std 148 | v_dir = theta / np.linalg.norm(theta, ord=2, axis=-1) if normalize else theta 149 | 150 | return v_dir 151 | 152 | def _k_neighbors(self, 153 | point: np.ndarray, 154 | space, k=10): 155 | """ 156 | Compute nearest neighbor given point in latent space 157 | 158 | :param point: seed point in space 159 | :param space: latent space to perform NN retrieval on 160 | :param k: number of neighbors to retrieve 161 | :return: k neighbors of a given point 162 | """ 163 | # TODO: Simplify or remove method by using FashonCLIP methods directly 164 | cosine_sim = self.fclip._nearest_neighbours(k, [point], space)[0] 165 | return cosine_sim 166 | 167 | def traversal_fn(self, 168 | start_point: np.ndarray, 169 | v_dir: np.ndarray, 170 | step_size: float, 171 | reg_space: np.ndarray, 172 | reg_weight: float, 173 | reg_k: int, 174 | nearest_neighbors=None): 175 | """ 176 | Single-step latent space traversal 177 | 178 | :param start_point: starting point for traversal 179 | :param v_dir: direction vector used for traversal 180 | :param step_size: size of step 181 | :param reg_space: latent space used for regularization 182 | :param reg_weight: regularization weight 183 | :param reg_k: number of neighbors used for regularization 184 | :param nearest_neighbors: option to pass in pre-computed nearest neighbors for regularization 185 | :return: new point in space after traversal 186 | """ 187 | start_point = start_point / np.linalg.norm(start_point, ord=2) 188 | if nearest_neighbors is None: 189 | nearest_neighbors = self._k_neighbors(start_point, reg_space, k=reg_k) 190 | neighborhood_mean = reg_space[nearest_neighbors].mean(axis=0) 191 | return start_point + (1 - reg_weight) * step_size * v_dir + reg_weight * neighborhood_mean 192 | 193 | def traverse_space(self, 194 | start_point: np.ndarray, 195 | search_space: np.ndarray, 196 | v_dir: np.ndarray, 197 | step_size: float, 198 | steps: int, 199 | reg_space: np.ndarray, 200 | reg_weight: float, 201 | reg_k: int = 100, 202 | k=10): 203 | """ 204 | 205 | :param start_point: starting point for traversal 206 | :param search_space: latent space to traverse 207 | :param v_dir: direction vector used for traversal 208 | :param step_size: size of step 209 | :param steps: number of traversal steps 210 | :param reg_space: latent space used for regularization 211 | :param reg_weight: regularization weight 212 | :param reg_k: number of neighbors used for regularization 213 | :param k: number of products to return for each traversal step 214 | :return: list of list of product indices for each step in traversal 215 | """ 216 | nearest_neighbors = self._k_neighbors(start_point, search_space, k=reg_k) 217 | traversal_path = [nearest_neighbors[:k]] 218 | for _ in tqdm(range(steps)): 219 | # take a single step 220 | start_point = self.traversal_fn( 221 | start_point=start_point, 222 | v_dir=v_dir, 223 | step_size=step_size, 224 | reg_space=reg_space, 225 | reg_weight=reg_weight, 226 | reg_k=reg_k, 227 | nearest_neighbors=nearest_neighbors) 228 | # get nearest products 229 | nearest_neighbors = self._k_neighbors(start_point, search_space, k=reg_k) 230 | # store products 231 | traversal_path.append(nearest_neighbors[:k]) 232 | return traversal_path 233 | 234 | 235 | 236 | -------------------------------------------------------------------------------- /app/intro.py: -------------------------------------------------------------------------------- 1 | import streamlit as st 2 | import pandas as pd 3 | from utils import plotly_tsne, resize_gif 4 | from fashion_clip.fashion_clip import FCLIPDataset 5 | from fashion_clip.utils import display_images_from_s3 6 | import base64 7 | 8 | 9 | 10 | def app(): 11 | st.write(""" 12 | # Introduction 13 | 14 | Item-to-item comparative recommendations of the form "Can I get something _darker_/_longer_/_warmer_?" traditionally 15 | require fine-grained supervised data and employ "learning-to-rank" approaches. In this demo we introduce a zero-shot 16 | approach toward generating comparative recommendations by leveraging the linguistic capabilites of `FashionCLIP`, a 17 | `CLIP`-like model fine-tuned for fashion concepts. 18 | 19 | 20 | #### `FashionCLIP` latent space 21 | 22 | We begin by first exploring the latent space of `FashionCLIP`. By selecting different options in the drop down below 23 | you can visualize the TSNE projects of the image embeddings for various _comparative concepts_. 24 | """) 25 | 26 | # Latent Space Visualization 27 | tsne_plots = { 28 | 'Shirt Color Luminance' : pd.read_csv('data/tsne_fclip_blue_shirt.csv'), 29 | 'Skirt Length': pd.read_csv('data/tsne_fclip_skirt_length.csv'), 30 | 'Footwear Formality': pd.read_csv('data/tsne_fclip_shoes.csv') 31 | } 32 | tsne_sel = st.selectbox(label='', options=list(tsne_plots.keys())) 33 | st.plotly_chart(plotly_tsne(tsne_plots[tsne_sel], 34 | title=tsne_sel, 35 | enable_legend=True)) 36 | 37 | st.write(""" 38 | We observe that `FashionCLIP` organizes the various intensities for each comparative concept into separate clusters, 39 | suggesting that it is possible to trace a path in the latent space from one cluster to another. For example for the 40 | comparative concept of ___shirt color luminance___, three distinct clusters are formed for 41 | __light blue polo t-shirt__, __blue polo t-shirt__, and __dark blue polo t-shirt__. Similar patterns are observed 42 | for the other examples. 43 | 44 | The _goal_ of `GradREC` is to traverse such a path, in order to discover products along a certain comparative dimension. 45 | """) 46 | 47 | # Method 48 | 49 | st.write(""" 50 | #### Method 51 | 52 | We visualize here the overarching approach toward traversing the latent space. 53 | 54 | The two main ingredients are: 55 | 56 | 1. __Traversal Function__: Takes in an existing point in space (1) and a traversal vector (2), and returns a new point in space 57 | which is closer to a product (3) with increasing/decreasing attribute intensity (e.g. decreasing skirt length). Repeated application 58 | allows for traversal of the space. See illustration below. 59 | """) 60 | gif_path = "data/GradREC_gif.gif" 61 | gif_mini_path = "data/GradREC_gif_mini.gif" 62 | # resize_gif(gif_path, gif_mini_path, scale=3.5) 63 | with open(gif_mini_path, "rb") as f: 64 | contents = f.read() 65 | data_url = base64.b64encode(contents).decode("utf-8") 66 | 67 | exp_func = st.expander("Traversal Function") 68 | exp_func.markdown( 69 | """ 70 | cat gif 71 |   72 |   73 | """.format(data_url), 74 | unsafe_allow_html=True) 75 | 76 | st.write(""" 77 | 2. __Traversal Vector__: We construct the traversal vector by borrowing ideas from literature. The intuition behind 78 | the approach is to find channels which encode the attribute of interest. 79 | """) 80 | 81 | gif_path = "data/GradREC_traversal_vector_gif.gif" 82 | gif_mini_path = "data/GradREC_traversal_vector_gif_mini.gif" 83 | # resize_gif(gif_path, gif_mini_path, scale=5.5) 84 | with open(gif_mini_path, "rb") as f: 85 | contents = f.read() 86 | data_url = base64.b64encode(contents).decode("utf-8") 87 | 88 | exp_vector = st.expander("Traversal Vector") 89 | exp_vector.markdown( 90 | """ 91 | cat gif 92 | """.format(data_url), 93 | unsafe_allow_html=True) 94 | 95 | 96 | 97 | 98 | -------------------------------------------------------------------------------- /app/utils.py: -------------------------------------------------------------------------------- 1 | from typing import List 2 | import plotly.express as px 3 | from plotly.subplots import make_subplots 4 | import os 5 | from PIL import Image 6 | 7 | 8 | def plotly_tsne(tsne_results_df, title, enable_legend=True, width=None): 9 | if isinstance(tsne_results_df, List): 10 | assert isinstance(title, List) 11 | fig = make_subplots(rows=1,cols=len(tsne_results_df), subplot_titles=title, 12 | horizontal_spacing = 0.01) 13 | 14 | else: 15 | fig = make_subplots(rows=1,cols=1, subplot_titles=[title]) 16 | tsne_results_df = [tsne_results_df] 17 | 18 | for idx, df in enumerate(tsne_results_df): 19 | fig_scatter = px.scatter(df, 20 | x='x', 21 | y='y', 22 | color="Category", 23 | color_discrete_sequence=px.colors.qualitative.Bold, 24 | hover_data={ 25 | 'x': False, 26 | 'y': False, 27 | 'Category': True 28 | },) 29 | 30 | for _ in fig_scatter['data']: 31 | if idx == 0 and len(tsne_results_df)!=1: 32 | _.update(showlegend=False) 33 | fig.add_trace(_, row=1, col=idx+1) 34 | full_fig_scatter = fig_scatter.full_figure_for_development() 35 | xrange = full_fig_scatter.layout.xaxis.range 36 | yrange = full_fig_scatter.layout.yaxis.range 37 | fig.update_xaxes(range=xrange, row=1, col=idx + 1) 38 | fig.update_yaxes(range=yrange, row=1, col=idx + 1) 39 | 40 | 41 | fig.update_layout( 42 | margin={ 43 | 'b':0, 'l':0 ,'r':0, 't':30 44 | }, 45 | legend={ 46 | 'title':'category' 47 | }, 48 | paper_bgcolor='rgba(0,0,0,0)', 49 | autosize=False, 50 | showlegend=enable_legend, 51 | height=400, 52 | width=width or 700, 53 | ) 54 | fig.update_yaxes(title=None, visible=True, showticklabels=False, showgrid=True, fixedrange=True) 55 | fig.update_xaxes(title=None, visible=True, showticklabels=False, showgrid=True, fixedrange=True) 56 | return fig 57 | 58 | 59 | # https://stackoverflow.com/questions/41718892/pillow-resizing-a-gif 60 | def resize_gif(path, save_as=None, scale=1): 61 | """ 62 | Resizes the GIF to a given length: 63 | 64 | Args: 65 | path: the path to the GIF file 66 | save_as (optional): Path of the resized gif. If not set, the original gif will be overwritten. 67 | resize_to (optional): new size of the gif. Format: (int, int). If not set, the original GIF will be resized to 68 | half of its size. 69 | """ 70 | all_frames = extract_and_resize_frames(path, scale) 71 | 72 | if not save_as: 73 | save_as = path 74 | 75 | if len(all_frames) == 1: 76 | print("Warning: only 1 frame found") 77 | all_frames[0].save(save_as, optimize=True) 78 | else: 79 | all_frames[0].save(save_as, optimize=True, save_all=True, append_images=all_frames[1:], loop=1000) 80 | 81 | 82 | def analyseImage(path): 83 | """ 84 | Pre-process pass over the image to determine the mode (full or additive). 85 | Necessary as assessing single frames isn't reliable. Need to know the mode 86 | before processing all frames. 87 | """ 88 | im = Image.open(path) 89 | results = { 90 | 'size': im.size, 91 | 'mode': 'full', 92 | } 93 | try: 94 | while True: 95 | if im.tile: 96 | tile = im.tile[0] 97 | update_region = tile[1] 98 | update_region_dimensions = update_region[2:] 99 | if update_region_dimensions != im.size: 100 | results['mode'] = 'partial' 101 | break 102 | im.seek(im.tell() + 1) 103 | except EOFError: 104 | pass 105 | return results 106 | 107 | 108 | def extract_and_resize_frames(path, scale): 109 | """ 110 | Iterate the GIF, extracting each frame and resizing them 111 | 112 | Returns: 113 | An array of all frames 114 | """ 115 | mode = analyseImage(path)['mode'] 116 | 117 | im = Image.open(path) 118 | 119 | resize_to = (im.size[0] // scale, im.size[1] // scale) 120 | 121 | i = 0 122 | p = im.getpalette() 123 | last_frame = im.convert('RGBA') 124 | 125 | all_frames = [] 126 | 127 | try: 128 | while True: 129 | # print("saving %s (%s) frame %d, %s %s" % (path, mode, i, im.size, im.tile)) 130 | 131 | ''' 132 | If the GIF uses local colour tables, each frame will have its own palette. 133 | If not, we need to apply the global palette to the new frame. 134 | ''' 135 | 136 | new_frame = Image.new('RGBA', im.size) 137 | 138 | ''' 139 | Is this file a "partial"-mode GIF where frames update a region of a different size to the entire image? 140 | If so, we need to construct the new frame by pasting it on top of the preceding frames. 141 | ''' 142 | if mode == 'partial': 143 | new_frame.paste(last_frame) 144 | 145 | new_frame.paste(im, (0, 0), im.convert('RGBA')) 146 | 147 | new_frame.thumbnail(resize_to, Image.ANTIALIAS) 148 | all_frames.append(new_frame) 149 | 150 | i += 1 151 | last_frame = new_frame 152 | im.seek(im.tell() + 1) 153 | except EOFError: 154 | pass 155 | 156 | return all_frames 157 | -------------------------------------------------------------------------------- /grad_rec_api_demo.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%load_ext autoreload\n", 12 | "%autoreload 2" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": null, 18 | "outputs": [], 19 | "source": [ 20 | "%reload_ext autoreload" 21 | ], 22 | "metadata": { 23 | "collapsed": false, 24 | "pycharm": { 25 | "name": "#%%\n" 26 | } 27 | } 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": null, 32 | "outputs": [], 33 | "source": [ 34 | "# Load some envs since we are using a private bucket for now\n", 35 | "from dotenv import load_dotenv\n", 36 | "load_dotenv('.env')\n", 37 | "from fashion_clip.fashion_clip import FCLIPDataset, FashionCLIP\n", 38 | "from fashion_clip.utils import display_images_from_url\n", 39 | "from app.gradient_rec import GradREC" 40 | ], 41 | "metadata": { 42 | "collapsed": false, 43 | "pycharm": { 44 | "name": "#%%\n" 45 | } 46 | } 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": null, 51 | "outputs": [], 52 | "source": [ 53 | "# load a FashionCLIP modfel\n", 54 | "dataset = FCLIPDataset(name='FF',\n", 55 | " image_source_type='s3',\n", 56 | " image_source_path ='s3://farfetch-images-ztapq86olwi6kub2p79d/images/')\n", 57 | "fclip = FashionCLIP('FCLIP', dataset)" 58 | ], 59 | "metadata": { 60 | "collapsed": false, 61 | "pycharm": { 62 | "name": "#%%\n" 63 | } 64 | } 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": null, 69 | "outputs": [], 70 | "source": [ 71 | "# instantiate GradREC object\n", 72 | "gradrec = GradREC(fclip)" 73 | ], 74 | "metadata": { 75 | "collapsed": false, 76 | "pycharm": { 77 | "name": "#%%\n" 78 | } 79 | } 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": null, 84 | "outputs": [], 85 | "source": [ 86 | "# Supply a start and end query to obtain semantic direction of traversal\n", 87 | "start_query = 'long skirt'\n", 88 | "end_query = 'short skirt'\n", 89 | "v_dir = gradrec.direction_vector(start_query=start_query, end_query=end_query)" 90 | ], 91 | "metadata": { 92 | "collapsed": false, 93 | "pycharm": { 94 | "name": "#%%\n" 95 | } 96 | } 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": null, 101 | "outputs": [], 102 | "source": [ 103 | "# Use FashionCLIP retrieval to get some initial seed products\n", 104 | "start_points = fclip.retrieval([start_query])" 105 | ], 106 | "metadata": { 107 | "collapsed": false, 108 | "pycharm": { 109 | "name": "#%%\n" 110 | } 111 | } 112 | }, 113 | { 114 | "cell_type": "code", 115 | "execution_count": null, 116 | "outputs": [], 117 | "source": [ 118 | "path = gradrec.traverse_space(start_point=fclip.image_vectors[start_points[0][1]],\n", 119 | " search_space=fclip.image_vectors,\n", 120 | " v_dir=v_dir,\n", 121 | " step_size=-2.0,\n", 122 | " steps=20,\n", 123 | " reg_space=fclip.image_vectors,\n", 124 | " reg_weight=0.9)\n", 125 | "# convert path from index to id\n", 126 | "path = [[dataset.ids[idx] for idx in p] for p in path]" 127 | ], 128 | "metadata": { 129 | "collapsed": false, 130 | "pycharm": { 131 | "name": "#%%\n" 132 | } 133 | } 134 | }, 135 | { 136 | "cell_type": "code", 137 | "execution_count": null, 138 | "outputs": [], 139 | "source": [ 140 | "for p in path:\n", 141 | " dataset.display_products(p, fields=tuple(), columns=10)" 142 | ], 143 | "metadata": { 144 | "collapsed": false, 145 | "pycharm": { 146 | "name": "#%%\n" 147 | } 148 | } 149 | }, 150 | { 151 | "cell_type": "code", 152 | "execution_count": null, 153 | "outputs": [], 154 | "source": [ 155 | "print(path)\n" 156 | ], 157 | "metadata": { 158 | "collapsed": false, 159 | "pycharm": { 160 | "name": "#%%\n" 161 | } 162 | } 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": null, 167 | "outputs": [], 168 | "source": [], 169 | "metadata": { 170 | "collapsed": false, 171 | "pycharm": { 172 | "name": "#%%\n" 173 | } 174 | } 175 | } 176 | ], 177 | "metadata": { 178 | "kernelspec": { 179 | "display_name": "Python 3", 180 | "language": "python", 181 | "name": "python3" 182 | }, 183 | "language_info": { 184 | "codemirror_mode": { 185 | "name": "ipython", 186 | "version": 2 187 | }, 188 | "file_extension": ".py", 189 | "mimetype": "text/x-python", 190 | "name": "python", 191 | "nbconvert_exporter": "python", 192 | "pygments_lexer": "ipython2", 193 | "version": "2.7.6" 194 | } 195 | }, 196 | "nbformat": 4, 197 | "nbformat_minor": 0 198 | } -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | from fashion_clip.fashion_clip import FCLIPDataset, FashionCLIP 2 | from gradient_rec import GradREC 3 | 4 | 5 | if __name__ == "__main__": 6 | ff_dataset = FCLIPDataset('FF', image_folder='s3://farfetch-images-ztapq86olwi6kub2p79d/images/') 7 | fclip = FashionCLIP('FCLIP', ff_dataset) 8 | grec = GradREC(fclip) 9 | 10 | query_vectors = grec._encode_queries(['long red skirt', 'short red skirt']) 11 | v_dir = grec.direction_vector('long red skirt', 'short red skirt') 12 | start_points, p_info = grec._product_retrieval([query_vectors[0]]) 13 | print(p_info[0][:3]) 14 | path = grec.traverse_space(start_point=start_points[0][0], 15 | search_space=fclip.image_vectors, 16 | v_dir=v_dir, 17 | step_size=0.1, 18 | steps=10, 19 | reg_space=fclip.image_vectors, 20 | reg_weight=0.9) 21 | 22 | for p in path: 23 | print(fclip.dataset.catalog[p]) 24 | print('---') -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | fashion-clip @ git+https://github.com/patrickjohncyh/fashion-clip 2 | numpy==1.21.5 3 | streamlit==1.8.1 4 | pandas==1.3.5 5 | plotly==5.8.0 6 | kaleido==0.2.1 --------------------------------------------------------------------------------