├── .gitignore ├── LICENSE ├── README.md ├── color-extractor ├── color_extractor ├── __init__.py ├── back.py ├── cluster.py ├── exceptions.py ├── from_file.py ├── from_json.py ├── image_to_color.py ├── name.py ├── resize.py ├── selector.py ├── skin.py └── task.py ├── color_names.npz └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2016 Algolia 4 | http://www.algolia.com/ 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in 14 | all copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 22 | THE SOFTWARE. 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Color Extractor 2 | 3 | This project is both a library and a CLI tool to extract the dominant colors of 4 | the main object of an image. Most of the preprocessing steps assume that the 5 | images are related to e-commerce, meaning that the objects targeted by the 6 | algorithms are supposed to be mostly centered and with a fairly simple 7 | background (single color, gradient, low contrast, etc.). The algorithm may 8 | still perform if any of those two conditions is not met, but be aware that its 9 | precision will certainly be hindered. 10 | 11 | ***A blog post describing this experiment can be found [here](https://blog.algolia.com/how-we-handled-color-identification/).*** 12 | 13 | > Note: this project is released as-is, and is no longer maintained by us, 14 | > however feel free to edit the code and use as you see fit. 15 | 16 | ## Installation 17 | 18 | The script and the library are currently targeting python 3 and won't work with 19 | python 2. 20 | 21 | Most of the dependencies can be installed using 22 | 23 | ```sh 24 | pip install -r requirements.txt 25 | ``` 26 | 27 | Note that library and the CLI tool also depend on opencv 3.1.0 and its python 3 28 | bindings. 29 | For Linux users, the steps to install it are available 30 | [here](http://www.pyimagesearch.com/2015/07/20/install-opencv-3-0-and-python-3-4-on-ubuntu/). 31 | For OSX users, the steps to install it are available 32 | [here](http://www.pyimagesearch.com/2015/06/29/install-opencv-3-0-and-python-3-4-on-osx/). 33 | 34 | You then just have to ensure that this repository root is present in your 35 | `PYTHONPATH`. 36 | 37 | ## Color tagging 38 | 39 | Searching objects by color is a common practice while browsing e-commerce 40 | web sites and relying only on the description and the title of the object may 41 | not be enough to provide top-notch relevancy. 42 | We propose this tool to automatically associate color tags to an image by 43 | trying to guess the main object of the picture and extracting its dominant 44 | color(s). 45 | 46 | The design of the library can be viewed as a pipeline composed of several 47 | sequential processing. Each of these processings accepts several options in order 48 | to tune its behavior to better fit your catalog. 49 | Those processings are (in order): 50 | 51 | 1. Resizing and cropping 52 | 53 | 2. Background detection 54 | 55 | 3. Skin detection 56 | 57 | 4. Clustering of remaining pixels 58 | 59 | 5. Selection of the _best_ clusters 60 | 61 | 6. Giving color names to clusters 62 | 63 | ### Usage 64 | 65 | The library can be used as simply as this: 66 | 67 | ```python 68 | import cv2 69 | import numpy as np 70 | 71 | from color_extractor import ImageToColor 72 | 73 | npz = np.load('color_names.npz') 74 | img_to_color = ImageToColor(npz['samples'], npz['labels']) 75 | 76 | img = cv2.imread('image.jpg') 77 | print(img_to_color.get(img)) 78 | ``` 79 | 80 | The CLI tool as simply as this: 81 | 82 | ```sh 83 | ./color-extractor color_names.npz image.jpg 84 | > red,black 85 | ``` 86 | 87 | The file `color_names.pnz` can be found in this repository. 88 | 89 | ### Passing Settings 90 | 91 | All algorithms can be used right out of the box thanks to settings tweaked for 92 | the larger range of images possible. Because these settings don't target any 93 | special kind of catalog, changing them may cause a gain of precision. 94 | 95 | Settings can be passed at three different levels. 96 | 97 | The lowest level is at the algorithm-level. Each algorithm is embodied by a 98 | python class which accepts a `settings` dictionary. This dictionary is then 99 | merged with its default settings. The given settings have precedence over the 100 | default one. 101 | 102 | A slightly higher level still concerns the library users. The process of chaining 103 | all those algorithms together is also embedded in 3 classes called `FromJson`, 104 | `FromFile` and `ImageToColor`. Those three classes also take a `settings` 105 | parameter, composed of several dictionary to be forwarded to each algorithm. 106 | 107 | The higher level is to pass those settings to the CLI tool. When passing the 108 | `--settings` option with a JSON file the latter is parsed as a dictionary and 109 | giving to the underlying `FromJson` or `FromFile` object (which in turn will 110 | forward to the individual algorithms). 111 | 112 | ### Resizing and Cropping 113 | 114 | This step is available as the `Resize` class. 115 | 116 | Pictures with a too high resolution have too much details that can be considered 117 | as noise when the goal is to find the most dominant colors. Moreover, smaller 118 | images mean faster processing time. Most of the testing has been done on 119 | `100x100` images, and it is usually the best compromise between precision and 120 | speed. 121 | Most of the time the object of the picture is centered, cropping can make sense 122 | in order to reduce the quantity of background and ease its removal. 123 | 124 | The available settings are: 125 | 126 | - `'crop'` sets the cropping ratio. A ratio of `1.` means no cropping. 127 | Default is `0.9`. 128 | 129 | - `'rows'` gives the number of rows to reduce the image to. The columns are 130 | computed to keep the same ratio. 131 | Default is `100`. 132 | 133 | ### Background Detection 134 | 135 | This step is available as the `Back` class. 136 | 137 | This algorithm tries to discard the background from the foreground by combining 138 | two simple algorithms. 139 | 140 | The first algorithm takes the colors of the four corners of the image and treat 141 | as background all pixels _close_ to those colors. 142 | 143 | The second algorithm uses a Sobel filter to detect edges and then runs a 144 | flood fill algorithm from all four corners. All pixels touched by the flood fill 145 | are considered background. 146 | 147 | The masks created by the two algorithms are then combined together with a 148 | logical `or`. 149 | 150 | The available settings are: 151 | 152 | - `'max_distance'` sets the maximum distance for two colors to be considered 153 | close by the first algorithm. A higher value means more pixels will be 154 | considered as background. 155 | Default is `5`. 156 | 157 | - `'use_lab'` converts pixels to the LAB color space before using the first 158 | algorithm. The conversion makes the process a bit more expensive but the 159 | computed distances are closer to human perception. 160 | Default is `True`. 161 | 162 | ### Skin Detection 163 | 164 | This step is available as the `Skin` class. 165 | 166 | When working with fashion pictures models are usually present in the picture. 167 | The main problem is that their skin color can be confused with the object color 168 | and yield to incorrect tags. One way to avoid that is to ignore ranges of colors 169 | corresponding to common color skins. 170 | 171 | The available settings are: 172 | 173 | - `'skin_type'` The skin type to target. At the moment only `'general'` and 174 | `'none'` are supported. `'none'` returns an empty mask every time, 175 | deactivating skin detection. 176 | Default is `'general'`. 177 | 178 | 179 | ### Clustering 180 | 181 | This step is available as the `Cluster` class. 182 | 183 | As we want to find the most dominant color(s) of an object, grouping them into 184 | buckets allows us to retain only a few ones and to have a sense of which are the 185 | most present. 186 | The clustering is done using the K-Means algorithm. K-Means doesn't result 187 | in the most accurate clusterings (compared to Mean Shift for example) but its 188 | speed certainly compensate. Before all images are different, it's hard to 189 | use a fixed number of clusters for the entire catalog. We implemented a method 190 | that tries to find an optimal number of clusters called the 191 | [jump](https://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set#An_Information_Theoretic_Approach) 192 | method. 193 | 194 | The available settings are: 195 | 196 | - `'min_k'` The minimum number of clusters to consider. 197 | Default is `2`. 198 | 199 | - `'max_k'` The maximum number of clusters to consider. Allowing more clusters 200 | results in greater computing times. 201 | Default is `7`. 202 | 203 | ### Selection of Clusters 204 | 205 | This step is available as the `Selector` class. 206 | 207 | Once clusters are made, all of them may not be worth a color tag: some may be 208 | very tiny for example. The purpose of this step is to only keep the clusters 209 | that are worth it. 210 | We implemented different way of selecting clusters: 211 | 212 | - `'all'` keeps all clusters. 213 | 214 | - `'largest'` keeps only the largest cluster. 215 | 216 | - `'ratio'` keeps the biggest clusters until their total number of pixels 217 | exceeds a certain percentage of all clustered pixels. 218 | 219 | While the outcome of `all` is quite obvious, the use of `largest` versus 220 | `ratio` is trickier. `largest` will yield very few colors, meaning the chance 221 | of assigning a tag not really relevant is greatly diminished. On the other 222 | hand objects with two colors in equal quantity will see one of them discarded. 223 | It's up to you to decide which one behaves the best with your catalog. 224 | 225 | The available settings are: 226 | 227 | - `'strategy'`: The strategy to used among `'all'`, `'largest'` and `'ratio'`. 228 | Default is `'largest'`. 229 | 230 | - `'ratio.threshold'`: The percentage of clustered pixels to target while 231 | selecting clusters with the `'ratio'` strategy. 232 | Default is `0.75`. 233 | 234 | ### Naming Color Values 235 | 236 | This step is available as the `Name` class. 237 | 238 | The last step is to give human readable color names to RGB values. To solve 239 | this last step we use a K Nearest Neighbors algorithm applied to a large 240 | dictionary of colors taken from the XKCD color survey. Because of the erratic 241 | distribution of colors (some colors are far more represented that others) a 242 | KNN behaves in most cases better than more statistical classifiers. 243 | The "learning" phase of the classifier is done when the object is built, and 244 | requires that two arrays are passed to its constructor: an array of BGR colors 245 | and an array of the corresponding names. When using the CLI tool, the path 246 | to an `.npz` numpy archive containing those two matrices must be given. 247 | 248 | Even if the algorithm used defaults to KNN, it's still possible to use a custom 249 | class to do it. The supplied class must support a `fit` method in lieu of 250 | training phase and a `predict` method for the actual classification. 251 | 252 | The available settings are: 253 | 254 | - `'algorithm'` The algorithm to use to perform the classification. Must be 255 | either `'knn'` or `'custom'`. If custom is given, `'classifier.class'` must 256 | also be given. 257 | Default is `'knn'` 258 | 259 | - `'hard_monochrome'` Monochrome colors (especially gray) may be hard to 260 | classify, this option makes use of a built in way of qualifying colors as 261 | "white", "gray" or "black". It uses the rejection of the color vector against 262 | the gray axis and uses a threshold to determine whether or not the color can 263 | be considered monochrome and the luminance to classify it as "black", "white" 264 | or "gray". 265 | Default is `True`. 266 | 267 | - `'{gray,white,black}_name'` When using `'hard_monochrome'` changes the name 268 | actually given to "gray", "white" and "black" respectively. Useful when 269 | wanting color names in another language. 270 | Default is `"gray"`, `"white"` and `"black"` 271 | 272 | - `'classifier.args'` Arguments passed to the classifier constructor. Default 273 | one are provided for `'knn'` being 274 | `{"n_neighbors": 50, "weights": "distance", "n_jobs": -1}`. The possible 275 | arguments are the ones available to the scikit-learn implementation of the 276 | `KNeighborsClassifier`. 277 | 278 | - `'classifier.scale'` Many classification algorithms make strong assumption 279 | regarding the distribution of the samples, and may need some kind of 280 | standardization of the data to behave better. This settings controls the 281 | application of such a standardization before training and prediction. 282 | Default is `True` but is ignored when using `'knn'`. 283 | 284 | ### Complete Processing 285 | 286 | Instead of instantiating each of the aforementioned classes, you can simply use 287 | `ImageToColor` or `FromFile`. Those two classes take the same 288 | arguments for their construction. 289 | 290 | - An array of BGR colors to learn how to associate color names to color values. 291 | 292 | - An array of strings corresponding to the labels of the previous array. 293 | 294 | - A dictionary of settings to be passed to each processing. 295 | 296 | The dictionary can have the following keys: 297 | 298 | - `'resize'` settings to be given to the `Resize` object 299 | 300 | - `'back'` settings to be given to the `Back` object 301 | 302 | - `'skin'` settings to be given to the `Skin` object 303 | 304 | - `'cluster'` settings to be given to the `Cluster` object 305 | 306 | - `'selector'` settings to be given to the `Selector` object 307 | 308 | - `'name'` settings to be given to the `Name` object 309 | 310 | 311 | The main difference is the source of the image used. `ImageToColor` expects a 312 | numpy array while `FromFile` expects both a local path or a URL where the 313 | image can be (down)loaded from. 314 | 315 | ### Enriching JSON 316 | 317 | Because we want Algolia customers to be able to enrich their JSON records easily 318 | we provide a class able to stream JSON and add color tags on the fly. 319 | The object is initialized with the same arguments as `FromFile` plus the name 320 | of the field where the URI of the images can be found. While reading the JSON 321 | file if the given name is encountered the corresponding image is downloaded and 322 | its colors computed. Those colors are then added to the JSON object under the 323 | field `_color_tags`. The name of this field can be changed thanks to an optional 324 | parameter of the constructor. 325 | 326 | Enriching JSON can be used directly from the command line as this: 327 | 328 | ```sh 329 | ./color-extractor -j color_names.npz file.json 330 | ``` 331 | -------------------------------------------------------------------------------- /color-extractor: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | """Color Extractor 4 | 5 | Add color tags to your e-commerce pictures to improve the search experience! 6 | 7 | This command line tool needs the path to a numpy npz file containing two 8 | matrices allowing the script to learn how to give color names to color values. 9 | By default the two matrices are expected to be named `samples` and `labels`. 10 | 11 | Following the npz archive are expected local paths or URLs to images. Those 12 | images will then be downloaded and their associated color computed. Colors will 13 | be printed as a comma-separated list, a line per image. 14 | A JSON can be used to parameterize the colors' computation using the 15 | `--settings` option. For details about the available options, refer to the 16 | README.md file. 17 | 18 | Instead of paths to images, paths to JSON files can be given by supplying 19 | `--enrich-json`. The script will then proceed to retrieve the 20 | images found in the JSON objects and compute their associated colors. 21 | The whole JSON objects are then written to the standard output with an 22 | additional attribute containing the color tags. 23 | By default the images are retrieved from the 'image' attribute and the colors 24 | written to the `_color_tags` attribute. 25 | 26 | Usage: 27 | color-extractor.py [options] ... 28 | 29 | Options: 30 | -h --help Show this message. 31 | 32 | -s, --settings Read configuration of the pipeline from the given 33 | JSON file. 34 | 35 | --npz-samples Name of the samples matrix in the npz archive. 36 | [default: samples] 37 | 38 | --npz-labels Name of the labels matrix in the npz archive. 39 | [default: labels] 40 | 41 | -j, --enrich-json Expect JSON files and enrich them with color tags. 42 | [default: False] 43 | 44 | --image-field Use to retrieve images from JSON files. 45 | Must be used with `--enrich-json`. 46 | [default: image] 47 | 48 | --colors-field Write found colors in JSON field. 49 | Must be used with `--enrich-json`. 50 | [default: _color_tags] 51 | 52 | """ 53 | 54 | import json 55 | from sys import stdout, stderr 56 | 57 | import numpy as np 58 | 59 | from color_extractor import FromJson, FromFile 60 | from docopt import docopt 61 | 62 | 63 | def _load_matrices(args): 64 | try: 65 | npz = np.load(args['']) 66 | except Exception as e: 67 | stderr.write('Failed to load npz archive: `{}`\n'.format(e)) 68 | exit(1) 69 | 70 | def load_matrix(key, name): 71 | try: 72 | return npz[key] 73 | except KeyError as e: 74 | stderr.write('Failed to load {} matrix: `{}`\n'.format(name, e)) 75 | exit(1) 76 | 77 | s = load_matrix(args['--npz-samples'], 'samples') 78 | l = load_matrix(args['--npz-labels'], 'labels') 79 | return s, l 80 | 81 | 82 | def _load_settings(file_): 83 | try: 84 | with open(file_, 'r') as f: 85 | return json.load(f) 86 | except Exception as e: 87 | stderr.write('Failed to load settings file: `{}`\n'.format(e)) 88 | exit(1) 89 | 90 | 91 | def _json_files(args, samples, labels, settings): 92 | ifield = args['--image-field'] 93 | cfield = args['--colors-field'] 94 | j = FromJson(ifield, samples, labels, cfield, settings) 95 | 96 | stdout.write('[') 97 | 98 | for i, file_ in enumerate(args['']): 99 | with open(file_, 'r') as f: 100 | j.get(f) 101 | 102 | if i < len(args['']) - 1: 103 | stdout.write(',') 104 | 105 | stdout.write(']') 106 | 107 | 108 | def _images_files(args, samples, labels, settings): 109 | f = FromFile(samples, labels, settings) 110 | for file_ in args['']: 111 | try: 112 | colors = f.get(file_) 113 | if isinstance(colors, tuple): 114 | colors = colors[0] 115 | print(','.join(colors)) 116 | except Exception as e: 117 | m = 'Unable to find colors for {}: `{}`\n'.format(file_, e) 118 | stderr.write(m) 119 | print('') 120 | 121 | 122 | if __name__ == '__main__': 123 | args = docopt(__doc__, version='Color Extractor 1.0') 124 | samples, labels = _load_matrices(args) 125 | settings = {} 126 | if args['--settings'] is not None: 127 | settings = _load_settings(args['--settings']) 128 | 129 | if args['--enrich-json']: 130 | _json_files(args, samples, labels, settings) 131 | else: 132 | _images_files(args, samples, labels, settings) 133 | -------------------------------------------------------------------------------- /color_extractor/__init__.py: -------------------------------------------------------------------------------- 1 | from .resize import Resize 2 | from .back import Back 3 | from .skin import Skin 4 | from .cluster import Cluster 5 | from .selector import Selector 6 | from .name import Name 7 | from .image_to_color import ImageToColor 8 | from .from_file import FromFile 9 | from .from_json import FromJson 10 | from .exceptions import KMeansException 11 | 12 | __all__ = ['Resize', 'Back', 'Skin', 'Cluster', 'Selector', 'Name', 13 | 'ImageToColor', 'FromFile', 'FromJson', 'KMeansException'] 14 | -------------------------------------------------------------------------------- /color_extractor/back.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import skimage.filters as skf 3 | import skimage.color as skc 4 | import skimage.morphology as skm 5 | from skimage.measure import label 6 | 7 | from .task import Task 8 | 9 | 10 | class Back(Task): 11 | """ 12 | Two algorithms are used together to separate background and foreground. 13 | One consider as background all pixel whose color is close to the pixels 14 | in the corners. This part is impacted by the `max_distance' and 15 | `use_lab' settings. 16 | The second one computes the edges of the image and uses a flood fill 17 | starting from all corners. 18 | """ 19 | def __init__(self, settings=None): 20 | """ 21 | The possible settings are: 22 | - max_distance: The maximum distance for two colors to be 23 | considered closed. A higher value will yield to a more aggressive 24 | background removal. 25 | (default: 5) 26 | 27 | - use_lab: Whether to use the LAB color space to perform 28 | background removal. More expensive but closer to eye perception. 29 | (default: True) 30 | """ 31 | if settings is None: 32 | settings = {} 33 | 34 | super(Back, self).__init__(settings) 35 | 36 | def get(self, img): 37 | f = self._floodfill(img) 38 | g = self._global(img) 39 | m = f | g 40 | 41 | if np.count_nonzero(m) < 0.90 * m.size: 42 | return m 43 | 44 | ng = np.count_nonzero(g) 45 | nf = np.count_nonzero(f) 46 | 47 | if ng < 0.90 * g.size and nf < 0.90 * f.size: 48 | return g if ng > nf else f 49 | 50 | if ng < 0.90 * g.size: 51 | return g 52 | 53 | if nf < 0.90 * f.size: 54 | return f 55 | 56 | return np.zeros_like(m) 57 | 58 | def _global(self, img): 59 | h, w = img.shape[:2] 60 | mask = np.zeros((h, w), dtype=np.bool) 61 | max_distance = self._settings['max_distance'] 62 | 63 | if self._settings['use_lab']: 64 | img = skc.rgb2lab(img) 65 | 66 | # Compute euclidean distance of each corner against all other pixels. 67 | corners = [(0, 0), (-1, 0), (0, -1), (-1, -1)] 68 | for color in (img[i, j] for i, j in corners): 69 | norm = np.sqrt(np.sum(np.square(img - color), 2)) 70 | # Add to the mask pixels close to one of the corners. 71 | mask |= norm < max_distance 72 | 73 | return mask 74 | 75 | def _floodfill(self, img): 76 | back = Back._scharr(img) 77 | # Binary thresholding. 78 | back = back > 0.05 79 | 80 | # Thin all edges to be 1-pixel wide. 81 | back = skm.skeletonize(back) 82 | 83 | # Edges are not detected on the borders, make artificial ones. 84 | back[0, :] = back[-1, :] = True 85 | back[:, 0] = back[:, -1] = True 86 | 87 | # Label adjacent pixels of the same color. 88 | labels = label(back, background=-1, connectivity=1) 89 | 90 | # Count as background all pixels labeled like one of the corners. 91 | corners = [(1, 1), (-2, 1), (1, -2), (-2, -2)] 92 | for l in (labels[i, j] for i, j in corners): 93 | back[labels == l] = True 94 | 95 | # Remove remaining inner edges. 96 | return skm.opening(back) 97 | 98 | @staticmethod 99 | def _default_settings(): 100 | return { 101 | 'max_distance': 5, 102 | 'use_lab': True, 103 | } 104 | 105 | @staticmethod 106 | def _scharr(img): 107 | # Invert the image to ease edge detection. 108 | img = 1. - img 109 | grey = skc.rgb2grey(img) 110 | return skf.scharr(grey) 111 | -------------------------------------------------------------------------------- /color_extractor/cluster.py: -------------------------------------------------------------------------------- 1 | from sklearn.cluster import KMeans 2 | 3 | from .exceptions import KMeansException 4 | from .task import Task 5 | 6 | 7 | class Cluster(Task): 8 | """ 9 | Use the K-Means algorithm to group pixels by clusters. The algorithm tries 10 | to determine the optimal number of clusters for the given pixels. 11 | """ 12 | def __init__(self, settings=None): 13 | if settings is None: 14 | settings = {} 15 | 16 | super(Cluster, self).__init__(settings) 17 | self._kmeans_args = { 18 | 'max_iter': 50, 19 | 'tol': 1.0, 20 | } 21 | 22 | def get(self, img): 23 | a = self._settings['algorithm'] 24 | if a == 'kmeans': 25 | return self._jump(img) 26 | else: 27 | raise ValueError('Unknown algorithm {}'.format(a)) 28 | 29 | def _kmeans(self, img, k): 30 | kmeans = KMeans(n_clusters=k, **self._kmeans_args) 31 | try: 32 | kmeans.fit(img) 33 | except: 34 | raise KMeansException() 35 | 36 | return kmeans.inertia_, kmeans.labels_, kmeans.cluster_centers_ 37 | 38 | def _jump(self, img): 39 | npixels = img.size 40 | 41 | best = None 42 | prev_distorsion = 0 43 | largest_diff = float('-inf') 44 | 45 | for k in range(self._settings['min_k'], self._settings['max_k']): 46 | compact, labels, centers = self._kmeans(img, k) 47 | distorsion = Cluster._square_distorsion(npixels, compact, 1.5) 48 | diff = prev_distorsion - distorsion 49 | prev_distorsion = distorsion 50 | 51 | if diff > largest_diff: 52 | largest_diff = diff 53 | best = k, labels, centers 54 | 55 | return best 56 | 57 | @staticmethod 58 | def _default_settings(): 59 | return { 60 | 'min_k': 2, 61 | 'max_k': 7, 62 | 'algorithm': 'kmeans', 63 | } 64 | 65 | @staticmethod 66 | def _square_distorsion(npixels, compact, y): 67 | return pow(compact / npixels, -y) 68 | -------------------------------------------------------------------------------- /color_extractor/exceptions.py: -------------------------------------------------------------------------------- 1 | class KMeansException(Exception): 2 | def __init__(self): 3 | message = 'Not enough pixels left to perform clustering.' 4 | super(KMeansException, self).__init__(message) 5 | -------------------------------------------------------------------------------- /color_extractor/from_file.py: -------------------------------------------------------------------------------- 1 | from os.path import basename, join, splitext 2 | 3 | from skimage.io import imread, imsave 4 | from skimage.util import img_as_float 5 | from skimage.color import gray2rgb 6 | 7 | from .image_to_color import ImageToColor 8 | from .task import Task 9 | 10 | 11 | class FromFile(Task): 12 | def __init__(self, samples, labels, settings=None): 13 | if settings is None: 14 | settings = {} 15 | 16 | super(FromFile, self).__init__(settings) 17 | self._image_to_color = ImageToColor(samples, labels, self._settings) 18 | 19 | def get(self, uri): 20 | i = imread(uri) 21 | if len(i.shape) == 2: 22 | i = gray2rgb(i) 23 | else: 24 | i = i[:, :, :3] 25 | c = self._image_to_color.get(i) 26 | 27 | dbg = self._settings['debug'] 28 | if dbg is None: 29 | return c 30 | 31 | c, imgs = c 32 | b = splitext(basename(uri))[0] 33 | imsave(join(dbg, b + '-resized.jpg'), imgs['resized']) 34 | imsave(join(dbg, b + '-back.jpg'), img_as_float(imgs['back'])) 35 | imsave(join(dbg, b + '-skin.jpg'), img_as_float(imgs['skin'])) 36 | imsave(join(dbg, b + '-clusters.jpg'), imgs['clusters']) 37 | 38 | return c, { 39 | 'resized': join(dbg, b + '-resized.jpg'), 40 | 'back': join(dbg, b + '-back.jpg'), 41 | 'skin': join(dbg, b + '-skin.jpg'), 42 | 'clusters': join(dbg, b + '-clusters.jpg'), 43 | } 44 | 45 | @staticmethod 46 | def _default_settings(): 47 | return { 48 | 'debug': None, 49 | } 50 | -------------------------------------------------------------------------------- /color_extractor/from_json.py: -------------------------------------------------------------------------------- 1 | import json 2 | import sys 3 | 4 | import ijson 5 | 6 | from .from_file import FromFile 7 | from .task import Task 8 | 9 | 10 | class FromJson(Task): 11 | def __init__(self, image_field, samples, labels, 12 | colors_field='_color_tags', settings=None): 13 | if settings is None: 14 | settings = {} 15 | 16 | super(FromJson, self).__init__(settings) 17 | self._image_field = image_field 18 | self._colors_field = colors_field 19 | self._from_file = FromFile(samples, labels, self._settings) 20 | 21 | def get(self, handle, out=sys.stdout): 22 | prev_event = 'start_map' 23 | for prefix, event, value in ijson.parse(handle): 24 | FromJson._put_comma(event, prev_event, out) 25 | if event.startswith('start_'): 26 | out.write('{' if event == 'start_map' else '[') 27 | elif event.startswith('end_'): 28 | out.write('}' if event == 'end_map' else ']') 29 | elif event == 'map_key': 30 | out.write('{}:'.format(json.dumps(value))) 31 | elif event == 'number': 32 | out.write(str(value)) 33 | else: 34 | out.write(json.dumps(value)) 35 | 36 | if event == 'string' and prefix.endswith(self._image_field): 37 | self._add_colors_tags(value, out) 38 | 39 | prev_event = event 40 | 41 | def _add_colors_tags(self, uri, out): 42 | try: 43 | colors = self._from_file.get(uri) 44 | except Exception as e: 45 | colors = [] 46 | m = 'Unable to find colors for {}: `{}`\n'.format(uri, e) 47 | sys.stderr.write(m) 48 | 49 | out.write(',"{}":{}'.format(self._colors_field, json.dumps(colors))) 50 | 51 | @staticmethod 52 | def _put_comma(ev, prev, out): 53 | if (ev != 'end_array' and ev != 'end_map' and prev != 'start_map' and 54 | prev != 'start_array' and prev != 'map_key'): 55 | out.write(',') 56 | -------------------------------------------------------------------------------- /color_extractor/image_to_color.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | from .back import Back 4 | from .cluster import Cluster 5 | from .name import Name 6 | from .resize import Resize 7 | from .selector import Selector 8 | from .skin import Skin 9 | from .task import Task 10 | 11 | 12 | class ImageToColor(Task): 13 | def __init__(self, samples, labels, settings=None): 14 | 15 | if settings is None: 16 | settings = {} 17 | 18 | super(ImageToColor, self).__init__(settings) 19 | self._resize = Resize(self._settings['resize']) 20 | self._back = Back(self._settings['back']) 21 | self._skin = Skin(self._settings['skin']) 22 | self._cluster = Cluster(self._settings['cluster']) 23 | self._selector = Selector(self._settings['selector']) 24 | self._name = Name(samples, labels, self._settings['name']) 25 | 26 | def get(self, img): 27 | resized = self._resize.get(img) 28 | back_mask = self._back.get(resized) 29 | skin_mask = self._skin.get(resized) 30 | mask = back_mask | skin_mask 31 | k, labels, clusters_centers = self._cluster.get(resized[~mask]) 32 | centers = self._selector.get(k, labels, clusters_centers) 33 | colors = [self._name.get(c) for c in centers] 34 | flattened = list({c for l in colors for c in l}) 35 | 36 | if self._settings['debug'] is None: 37 | return flattened 38 | 39 | colored_labels = np.zeros((labels.shape[0], 3), np.float64) 40 | for i, c in enumerate(clusters_centers): 41 | colored_labels[labels == i] = c 42 | 43 | clusters = np.zeros(resized.shape, np.float64) 44 | clusters[~mask] = colored_labels 45 | 46 | return flattened, { 47 | 'resized': resized, 48 | 'back': back_mask, 49 | 'skin': skin_mask, 50 | 'clusters': clusters 51 | } 52 | 53 | @staticmethod 54 | def _default_settings(): 55 | return { 56 | 'resize': {}, 57 | 'back': {}, 58 | 'skin': {}, 59 | 'cluster': {}, 60 | 'selector': {}, 61 | 'name': {}, 62 | } 63 | -------------------------------------------------------------------------------- /color_extractor/name.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from numpy.linalg import norm 3 | from sklearn.neighbors import KNeighborsClassifier 4 | from sklearn.preprocessing import StandardScaler 5 | 6 | from .task import Task 7 | 8 | 9 | class Name(Task): 10 | """ 11 | Create a color classifier trained on samples and labels. Samples should 12 | represent the actual value of the color (RGB, HSV, etc.) and labels 13 | should be the name of the color ('red', 'blue'...). 14 | Samples must be a `numpy` array of shape `(n_colors, 3)`. 15 | Labels must be a `numpy` array of `str` of shape `(n_colors,)`. 16 | """ 17 | def __init__(self, samples, labels, settings=None): 18 | """ 19 | The possible settings are: 20 | - algorithm: The algorithm to use for training the classifier. 21 | Possible values are 'knn' and 'custom'. 22 | If custom is provided, the_setting `classifier.class` must be 23 | set. 24 | (default: 'knn') 25 | 26 | - hard_monochrome: Use hardcoded values for white, black and gray. 27 | The check is performed in the BGR color space. 28 | The name returned depends on the settings 'white_name', 29 | 'black_name' and 'gray_name'. 30 | (default: True) 31 | 32 | - {gray,white,black}_name: Name to give to the {gray,white,black} 33 | color when 'hard_monochrome' is used. 34 | (default: {'gray','white','black}) 35 | 36 | - classifier.args: Settings to pass to the scikit-learn 37 | algorithm used. The settings can be found on the scikit-learn 38 | documentation. Defaults are provided for the specific algorithm 39 | `knn` for an out-of-the-box experience. 40 | (default: {}) 41 | 42 | - classifier.class: The class to use to perform the classification. 43 | when using the 'custom' algorithm. The class must support the 44 | method `fit` to train the model and `predict` to classify 45 | samples. 46 | (default: None) 47 | 48 | - classifier.scale: Use scikit-learn `StandardScaler` prior to 49 | train the model and classifying samples. 50 | """ 51 | if settings is None: 52 | settings = {} 53 | 54 | super(Name, self).__init__(settings) 55 | 56 | algo = self._settings['algorithm'] 57 | if algo == 'knn': 58 | self._settings['classifier.scale'] = False 59 | args = self._settings['classifier.args'] or Name._knn_args() 60 | type_ = KNeighborsClassifier 61 | elif algo == 'custom': 62 | args = self._settings['classifier.args'] 63 | type_ = self._settings['classifier.class'] 64 | else: 65 | raise ValueError('Unknown algorithm {}'.format(algo)) 66 | 67 | self._classifier = type_(**args) 68 | self._names, labels = np.unique(labels, return_inverse=True) 69 | 70 | if self._settings['classifier.scale']: 71 | self._scaler = StandardScaler() 72 | samples = self._scaler.fit_transform(samples) 73 | 74 | self._classifier.fit(samples, labels) 75 | 76 | def get(self, sample): 77 | """Return the color names for `sample`""" 78 | labels = [] 79 | sample = sample * 255 80 | 81 | if self._settings['hard_monochrome']: 82 | labels = self._hard_monochrome(sample) 83 | if labels: 84 | return labels 85 | 86 | if self._settings['classifier.scale']: 87 | sample = self._scaler.transform(sample) 88 | 89 | sample = sample.reshape((1, -1)) 90 | labels += [self._names[i] for i in self._classifier.predict(sample)] 91 | return labels 92 | 93 | def _hard_monochrome(self, sample): 94 | """ 95 | Return the monochrome colors corresponding to `sample`, if any. 96 | A boolean is also returned, specifying whether or not the saturation is 97 | sufficient for non monochrome colors. 98 | """ 99 | gray_proj = np.inner(sample, Name._GRAY_UNIT) * Name._GRAY_UNIT 100 | gray_dist = norm(sample - gray_proj) 101 | 102 | if gray_dist > 15: 103 | return [] 104 | 105 | colors = [] 106 | luminance = np.sum(sample * Name._GRAY_COEFF) 107 | if luminance > 45 and luminance < 170: 108 | colors.append(self._settings['gray_name']) 109 | if luminance <= 50: 110 | colors.append(self._settings['black_name']) 111 | if luminance >= 170: 112 | colors.append(self._settings['white_name']) 113 | 114 | return colors 115 | 116 | # Normalized identity (BGR gray) vector. 117 | _GRAY_UNIT = np.array([1, 1, 1]) / norm(np.array([1, 1, 1])) 118 | 119 | # Coefficients for BGR -> luminance. 120 | _GRAY_COEFF = np.array([0.114, 0.587, 0.299], np.float32) 121 | 122 | @staticmethod 123 | def _knn_args(): 124 | """Return the default arguments used by the `KNeighborsClassifier`""" 125 | return { 126 | 'n_neighbors': 50, 127 | 'weights': 'distance', 128 | 'n_jobs': -1, 129 | } 130 | 131 | @staticmethod 132 | def _default_settings(): 133 | return { 134 | 'algorithm': 'knn', 135 | 136 | 'hard_monochrome': True, 137 | 'white_name': 'white', 138 | 'black_name': 'black', 139 | 'gray_name': 'gray', 140 | 141 | 'classifier.class': None, 142 | 'classifier.args': {}, 143 | 'classifier.scale': True, 144 | } 145 | -------------------------------------------------------------------------------- /color_extractor/resize.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from skimage.transform import resize 3 | 4 | from .task import Task 5 | 6 | 7 | class Resize(Task): 8 | """ 9 | Resizes and crops given images to the specified shape. As most fashion 10 | have the subject centered, cropping may help reducing the background 11 | and help discarding background from foreground. 12 | Note that the background detection algorithm relies heavily on the 13 | corners, if the cropping is too important, the object itself may be 14 | disregarded. 15 | """ 16 | def __init__(self, settings=None): 17 | """ 18 | The possible settings are: 19 | - crop: The crop ratio to use. `1' means no cropping. A floating 20 | point number between `0' and `1' is expected. 21 | (default: 0.90) 22 | 23 | - shape: The height of the resized image. The ratio between height 24 | and width is kept. 25 | (default: 100) 26 | """ 27 | if settings is None: 28 | settings = {} 29 | 30 | super(Resize, self).__init__(settings) 31 | 32 | def get(self, img): 33 | """Returns `img` cropped and resized.""" 34 | return self._resize(self._crop(img)) 35 | 36 | def _resize(self, img): 37 | src_h, src_w = img.shape[:2] 38 | dst_h = self._settings['rows'] 39 | dst_w = int((dst_h / src_h) * src_w) 40 | return resize(img, (dst_h, dst_w)) 41 | 42 | def _crop(self, img): 43 | src_h, src_w = img.shape[:2] 44 | c = self._settings['crop'] 45 | dst_h, dst_w = int(src_h * c), int(src_w * c) 46 | rm_h, rm_w = (src_h - dst_h) // 2, (src_w - dst_w) // 2 47 | return img[rm_h:rm_h + dst_h, rm_w:rm_w + dst_w].copy() 48 | 49 | @staticmethod 50 | def _default_settings(): 51 | return { 52 | 'crop': 0.90, 53 | 'rows': 100, 54 | } 55 | -------------------------------------------------------------------------------- /color_extractor/selector.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | from .task import Task 4 | 5 | 6 | class Selector(Task): 7 | def __init__(self, settings=None): 8 | if settings is None: 9 | settings = {} 10 | 11 | super(Selector, self).__init__(settings) 12 | 13 | def get(self, k, labels, centers): 14 | s = self._settings['strategy'] 15 | if s == 'largest': 16 | return Selector._largest(k, labels, centers) 17 | elif s == 'ratio': 18 | return self._ratio(k, labels, centers) 19 | elif s == 'all': 20 | return centers 21 | else: 22 | raise ValueError('Unknown strategy {}'.format(s)) 23 | 24 | def _ratio(self, k, labels, centers): 25 | counts = [np.count_nonzero(labels == l) for l in range(k)] 26 | counts = np.array(counts, np.uint32) 27 | total = np.sum(counts) 28 | sort_idx = np.argsort(counts)[::-1] 29 | cum_counts = np.cumsum(counts[sort_idx]) 30 | 31 | threshold = self._settings['ratio.threshold'] 32 | for idx_stop in range(k): 33 | if cum_counts[idx_stop] >= threshold * total: 34 | break 35 | sort_centers = centers[sort_idx] 36 | return sort_centers[:idx_stop + 1] 37 | 38 | @staticmethod 39 | def _largest(k, labels, centers): 40 | counts = [np.count_nonzero(labels == l) for l in range(k)] 41 | sort_idx = np.argsort(counts)[::-1] 42 | return [centers[sort_idx[0]]] 43 | 44 | @staticmethod 45 | def _default_settings(): 46 | return { 47 | 'strategy': 'largest', 48 | 'ratio.threshold': 0.75, 49 | } 50 | -------------------------------------------------------------------------------- /color_extractor/skin.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import skimage.morphology as skm 3 | from skimage.filters import gaussian 4 | from skimage.color import rgb2hsv 5 | from skimage.util import img_as_float 6 | 7 | from .task import Task 8 | 9 | 10 | class Skin(Task): 11 | def __init__(self, settings=None): 12 | """ 13 | Skin is detected using color ranges. 14 | 15 | The possible settings are: 16 | - skin_type: The type of skin most expected in the given images. 17 | The value can be 'general' or 'none'. If 'none' is given the 18 | an empty mask is returned. 19 | (default: 'general') 20 | """ 21 | if settings is None: 22 | settings = {} 23 | 24 | super(Skin, self).__init__(settings) 25 | self._k = skm.disk(1, np.bool) 26 | 27 | t = self._settings['skin_type'] 28 | if t == 'general': 29 | self._lo = np.array([0, 0.19, 0.31], np.float64) 30 | self._up = np.array([0.1, 1., 1.], np.float64) 31 | elif t != 'none': 32 | raise NotImplementedError('Only general type is implemented') 33 | 34 | def get(self, img): 35 | t = self._settings['skin_type'] 36 | if t == 'general': 37 | img = rgb2hsv(img) 38 | elif t == 'none': 39 | return np.zeros(img.shape[:2], np.bool) 40 | else: 41 | raise NotImplementedError('Only general type is implemented') 42 | 43 | return self._range_mask(img) 44 | 45 | def _range_mask(self, img): 46 | mask = np.all((img >= self._lo) & (img <= self._up), axis=2) 47 | 48 | # Smooth the mask. 49 | skm.binary_opening(mask, selem=self._k, out=mask) 50 | return gaussian(mask, 0.8, multichannel=True) != 0 51 | 52 | @staticmethod 53 | def _default_settings(): 54 | return { 55 | 'skin_type': 'general', 56 | } 57 | -------------------------------------------------------------------------------- /color_extractor/task.py: -------------------------------------------------------------------------------- 1 | class Task(object): 2 | def __init__(self, settings): 3 | self._settings = self._default_settings() 4 | self._settings.update(settings) 5 | 6 | def get(self, img): 7 | raise NotImplementedError 8 | 9 | @staticmethod 10 | def _default_settings(): 11 | return {} 12 | -------------------------------------------------------------------------------- /color_names.npz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/algolia/color-extractor/3f1d5753cb418a3461d9b350e907c4d1d693d590/color_names.npz -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | cycler==0.10.0 2 | dask==0.10.1 3 | decorator==4.0.10 4 | docopt==0.6.2 5 | ijson==2.3 6 | matplotlib==1.5.1 7 | networkx==1.11 8 | numpy==1.11.1 9 | Pillow==3.3.0 10 | pyparsing==2.1.5 11 | python-dateutil==2.5.3 12 | pytz==2016.6.1 13 | scikit-image==0.12.3 14 | scikit-learn==0.17.1 15 | scipy==0.17.1 16 | six==1.10.0 17 | toolz==0.8.0 18 | --------------------------------------------------------------------------------