This is an Oxford Visual Geometry Group computer vision practical, authored by Andrea Vedaldi and Andrew Zisserman (Release 2018a).
13 |
The goal of instance-level recognition is to match (recognize) a specific object or scene. Examples include recognizing a specific building, such as Notre Dame, or a specific painting, such as ''Starry Night'' by Van Gogh. The object is recognized despite changes in scale, camera viewpoint, illumination conditions and partial occlusion. An important application is image retrieval -- starting from an image of an object of interest (the query), search through an image dataset to obtain (or retrieve) those images that contain the target object.
15 |The goal of this session is to get basic practical experience with the methods that enable specific object recognition. It includes: (i) using SIFT features to obtain sparse matches between two images; (ii) using affine co-variant detectors to cover changes in viewpoint; (iii) vector quantizing the SIFT descriptors into visual words to enable large scale retrieval; and (iv) constructing and using an image retrieval system to identify objects.
16 |Read and understand the requirements and installation instructions. The download links for this practical are:
45 |After the installation is complete, open and edit the script exercise1.m in the MATLAB editor. The script contains commented code and a description for all steps of this exercise, relative to Part I of this document. You can cut and paste this code into the MATLAB window to run it, and will need to modify it as you go through the session. Other files exercise2.m, exercise3.m, and exercise4.m are given for Part II, III, and IV.
Note: the student packages contain only the code required to run the practical. The complete package, including code to preprocess the data, is available on GitHub.
53 |The SIFT feature has both a detector and a descriptor. We will start by computing and visualizing the SIFT feature detections for two images of the same object (a building facade). Load an image, rotate and scale it, and then display the original and transformed pair:
56 |% Load an image
57 | im1 = imread('data/oxbuild_lite/all_souls_000002.jpg') ;
58 | % Let the second image be a rotated and scaled version of the first
59 | im3 = imresize(imrotate(im1,35,'bilinear'),0.7) ;% Display the images
60 | subplot(1,2,1) ; imagesc(im1) ; axis equal off ; hold on ;
61 | subplot(1,2,2) ; imagesc(im3) ; axis equal off ;
62 | A SIFT frame is a circle with an orientation and is specified by four parameters: the center $t_x$, $t_y$, the scale $s$, and the rotation $\theta$ (in radians), resulting in a vector of four parameters $(s, \theta, t_x, t_y)$. Now compute and visualise the SIFT feature detections (frames):
65 |% Compute SIFT features for each
66 | [frames1, descrs1] = getFeatures(im1, 'peakThreshold', 0.01) ;
67 | [frames3, descrs3] = getFeatures(im3, 'peakThreshold', 0.01) ;
68 | subplot(1,2,1) ; imagesc(im1) ; axis equal off ; hold on ;
69 | vl_plotframe(frames1, 'linewidth', 2) ;
70 | subplot(1,2,2) ; imagesc(im3) ; axis equal off ; hold on ;
71 | vl_plotframe(frames3, 'linewidth', 2) ;
72 | Examine the second image and its rotated and scaled version and convince yourself that the detections overlap the same scene regions (even though the circles have moved their image position and changed radius). It is helpful to zoom into a smaller image area using the MATLAB magnifying glass tool. This demonstrates that the detection process transforms (is co-variant) with translations, rotations and isotropic scalings. This class of transformations is known as a similarity or equiform.
75 |76 |78 |Task: The number of detected features can be controlled by changing the
77 |peakThresholdoption. A larger value will select features that correspond to higher contrast structures in the image. Try this now: run again the same code, but increasepeakThresholdtwo or three times.
Now repeat the exercise with a pair of natural images. Start by loading the second one:
79 |% Load a second image
80 | im2 = imread('data/oxbuild_lite/all_souls_000015.jpg') ;
81 | and plot images and feature frames. Again you should see that many of the detections overlap the same scene region. Note that, while repeatability occurs for the pair of natural views, it is much betterfor the synthetically rotated pair.
84 |85 |88 |Question: Note the change in density of detections across the image. Why does it change? Will it be a problem for matching? How could it be avoided?
86 |Question: Occasionally, a feature is detected multiple times, with different orientations. This may happen when the orientation assignment is ambiguous. Which kind of image structure would result in ambiguous orientation assignment?
87 |
Next we will use the descriptor computed over each detection to match the detections between images. We will start with the simplest matching scheme (first nearest neighbour of descriptors) and then add more sophisticated methods to eliminate any mismatches.
90 |vl_plotsiftdescriptor. Then use vl_plotframe to overlay the corresponding frames.94 |96 |Question: Note the descriptors are computed over a much larger region (shown in blue) than the detection (shown in green). Why?
95 |
Compute first nearest neighbours matches - for each SIFT descriptor in the first image, compute its nearest neighbour in the second image with the function findNeighbours.
Visualize the correspondences using lines joining matched SIFT features with the function plotMatches.
105 |107 |Question: Notice that there are many mismatches. Examine some of the mismatches to understand why the mistakes are being made. For example, is the change in lighting a problem? What additional constraints can be applied to remove the mismatches?
106 |
Hint: You can visualize a subset of the matches using:
108 |figure; plotMatches(im1,im2,frames1,frames2,matches(:,3:200:end));
109 | Lowe introduced a second nearest neighbour (2nd NN) test to identify, and hence remove, ambiguous matches. The idea is to identify distinctive matches by a threshold on the ratio of first to second NN distances. In the MATLAB file, the ratio is nnThreshold = 1NN distance / 2NN distance.
nnThreshold in a range from 0.1 to 0.9, and examine how the number of matches and number of mismatches changes.nnThreshold = 0.8 is often a good compromise between losing too many matches and rejecting mismatches.118 |120 |Question: Examine some of the remaining mismatches to understand why they have occurred. How could they be removed?
119 |
In addition to the 2nd NN test, we can also require consistency between the matches and a geometric transformation between the images. For the moment we will look for matches that are consistent with a similarity transformation
122 |123 | 137 |
138 |which consists of a rotation by $\theta$, an isotropic scaling (i.e. same in all directions) by s, and a translation by a vector $(t_x, t_y)$. This transformation is specified by four parameters $(s,\theta,t_x,t_y)$ and can be computed from a single correspondence between SIFT detections in each image.
139 |140 |142 |Task: Work out how to compute this transformation from a single correspondence.
141 |
Hint: Recall from Stage I.A that a SIFT feature frame is an oriented circle and map one onto the other.
143 |The matches consistent with a similarity can then be found using a RANSAC inspired algorithm, implemented by the function geometricVerification:
RANSAC-like algorithm for geometric verification
145 |After this algorithm the inliers are consistent with the transformation and are retained, and most mismatches should now be removed.
158 |159 |161 |Task: The figure generated by
160 |plotMatchessupports the interactive visualisation of the transformation found bygeometricVerification. Try hovering with the mouse on the figure and check that corresponding image points are highlighted in the two images.
Skip to Part 2 on fast track
162 |163 |165 |Task: Test this procedure by varying the threshold distance (edit
164 |geometricVerificationand change theopts.tolerance1,opts.tolerance2, andopts.tolerance3parameters, where the last two thresholds are relative to the optional iterative fitting of an affine transformation or homography to the inliers). Note the number of inliers and number of mismatches.
If more matches are required the geometric transformation can be used alone, without also requiring the 2nd NN test. Indeed, since the 1st NN may not be the correct match, a list of potential (putative) matches can be generated for each SIFT descriptor by including the 1st NN, 2nd NN, 3rd NN etc. Investigate how the number of correct matches (and time for computation) grows as the potential match list is extended, and the geometric transformation is used to select inliers. To this end:
166 |167 |170 |Task: Change the code to include in the match list the 1st NN, 2nd NN, 3rd NN, … best matches for each feature. 168 | Task: Run geometric verification and check the number of verified matches using this expanded list.
169 |
Hint: You can use MATLAB’s tic and toc functions to measure the execution time of a snippet of code. For example
171 |tic ; pause(3) ; toc
172 | will pause MATLAB for three seconds and return an elapsed time approximately equal to 3. See help tic for details.
So far the change in viewpoint between images has been a similarity transformation. Now we consider more severe viewpoint changes - for example where an object is fronto-parallel in one view, and turns away from the camera in the other as in the graffiti wall images below:
177 |
In this case, there is foreshortening (anisotropic scaling) and perspective distortions between the images (as well as in-plane rotation, translation and scaling). A circle in one image cannot cover the same scene area as a circle in the other, but an ellipse can. Affine co-variant detectors are designed to find such regions.
179 |In the following we will compare the number of matches using a similarity and affine co-variant detector as the viewpoint becomes progressively more extreme. The detectors are SIFT (for similarity) and SIFT+affine adaptation (for affine), while the descriptor are in both cases SIFT.
180 |181 |183 |Task: Open and examine the script
182 |exercise2.min the MATLAB editor. Run the script.
Note the behaviour in the number of verified matches as the viewpoint becomes more extreme. Observe that the matches also identify the regions of the images that are in common.
184 |185 |187 |Question: The transformation between the images induced by the plane is a planar homography. The detections are only affine co-variant (not as general as a planar homography). So how can descriptors computed on these detections possibly match?
186 |
Note: There are many other detector variants that could be used for this task. These can be activated by the method option of getFeatures.m (see also help vl_covdet).
In large scale retrieval the goal is to match a query image to a large database of images (for example the WWW or Wikipedia). The quality of a match is measured as the number of geometrically verified feature correspondences between the query and a database image. While the techniques discussed in Part I and II are sufficient to do this, in practice they require too much memory to store the SIFT descriptors for all the detections in all the database images. We explore next two key ideas: one to reduce the memory footprint and pre-compute descriptor matches; the other to speed up image retrieval.
190 |191 |193 |Task: Open and edit the script
192 |exercise3.min the MATLAB editor, and cut and paste to work through the following stages.
Instead of matching feature descriptors directly as done in Part I and II, descriptors are usually mapped first to discrete symbols, also called visual words, by means of a clustering technique like K-Means. The descriptors that are assigned to the same visual word are considered matched. Each of the rows in the following figure illustrates image patches that are mapped to the same visual word, and are hence indistinguishable by the representation.
195 |
Then, matching two sets of feature descriptors (from two images) reduces to finding the intersection of two sets of symbols.
197 |198 |215 |Tasks:
199 |200 |
208 |- Load a visual word dictionary and an associated approximate nearest neighbour (ANN) matcher (the ANN matcher is used to determine the closest visual word to each descriptor and is based on a forest of KD trees).
201 |- Given SIFT descriptors for two images, quantise them (assign them) into the corresponding visual words.
202 |- Find corresponding features by looking for the same visual words in the two images and note the computation time.
203 |- Geometrically verify these initial correspondences and count the number of inlier matches found.
204 |- Find corresponding features by using the method of Part I and II, i.e. by comparing the descriptors directly, and note the computation time. Geometrically verify these initial correspondences and count the number of inlier matches found.
205 |- Compare the speed and number of inliers when using visual words vs raw SIFT descriptors by means of the function
206 |matchWords. Note, you should repeat the timing (by running the matching again) as the first time you run it there may be a delay as certain MATLAB components are loaded into memory.- Optional: compare the speed and number of matches over another pair of images (from part I and II).
207 |Questions:
209 |210 |
214 |- The size of the vocabulary (the number of clusters) is an important parameter in visual word algorithms. How does the size affect the number of inliers and the difficulty of computing the transformation?
211 |- In the above procedure the time required to convert the descriptors into visual words was not accounted for. Why?
212 |- What is the speedup in searching a large, fixed database of 10, 100, 1000 images?
213 |
Skip to Stage III.B on fast track
216 |Often multiple feature occurrences are mapped to the same visual word. In this case matchWords generates only one of the possible matches.
218 |225 |Tasks:
219 |220 |
224 |- Modify
221 |matchWordsto generate more than one match for cases in which multiple features are mapped to the same visual word.This can be achieved by increasing the value ofmaxNumMatches.- Most of these additional matches are incorrect. Filter them out by running
222 |geometricVerification.- Compare the number of inliers obtained before and after this modification.
223 |
While matching with visual words is much faster than doing so by comparing feature descriptors directly, scoring images directly based on the number of geometrically verified matches still entails fitting a geometric model, a relatively slow operation. Rather than scoring all the images in the database in this way, we are going to use an approximation and count the number of visual words shared between two images.
227 |To this end, one computes a histogram of the visual words in a query image and for each of the database images. Then the number of visual words in common can be computed from the intersection of the two histograms.
228 |The histogram intersection can be thought as a similarity measure between two histograms. In practice, this measure can be refined in several ways:
229 |Computing histogram similarities can be implemented extremely efficiently using an inverted file index. In this exercise, inner products between normalized histograms are computed quite efficiently using MATLAB's built-in sparse matrix engine.
234 |We now apply this retrieval method to search using a query image within a 660 image subset of the Oxford 5k building image set.
235 |236 |239 |Task: How many erroneously matched images do you count in the top results? 237 | Question: Why does the top image have a score of 1?
238 |
Histogram-based retrieval results are good but far from perfect. Given a short list of top ranked images from the previous step, we are now going to re-score them based on the number of inlier matches after a geometric verification step.
241 |242 |245 |Question: Why is the top score much larger than 1 now? 243 | Question: Are the retrieval results improved after geometric verification?
244 |
Now try the full system to retrieve matches to an unseen query image.
247 |
Skip and end here on fast track
250 |The images below are all details of paintings. The goal of this last part of the practical is to identify the paintings that they came from. For this we selected a set of 1734 images of paintings from Wikipedia.
251 |
To identify the details you can either:
253 |We follow route (3) here. Look through and run exercise4.m. This uses the techniques described in Part III in order to construct an index for 1734 Wikipedia images so that they may be searched quickly. Use the code to find from which paintings these details come from.
Note, although the index is stored locally, the matching images are downloaded from Wikipedia and displayed. Click on the image to reach the Wikipedia page for that painting (and hence identify it).
260 |261 |263 |Task: Use the code to visually search Wikipedia for further paintings from Van Gogh downloaded from the Internet.
262 |
Note: the code supports URL in place of filenames.
264 |Take note of the code output.
265 |266 |273 |Questions:
267 |268 |
272 |- How many features are there in the painting database?
269 |- How much memory does the image database take?
270 |- What are the stages of the search? And how long does each of the stages take for one of the query images?
271 |
That completes this practical.
274 |
6 |
7 | The goal of instance-level recognition is to match (recognize) a specific object or scene. Examples include recognizing a specific building, such as Notre Dame, or a specific painting, such as ''Starry Night'' by Van Gogh. The object is recognized despite changes in scale, camera viewpoint, illumination conditions and partial occlusion. An important application is image retrieval -- starting from an image of an object of interest (the query), search through an image dataset to obtain (or retrieve) those images that contain the target object.
8 |
9 | The goal of this session is to get basic practical experience with the methods that enable specific object recognition. It includes: (i) using SIFT features to obtain sparse matches between two images; (ii) using affine co-variant detectors to cover changes in viewpoint; (iii) vector quantizing the SIFT descriptors into visual words to enable large scale retrieval; and (iv) constructing and using an image retrieval system to identify objects.
10 |
11 | [TOC]
12 |
13 | ## Getting started
14 |
15 | Read and understand the [requirements and installation instructions](../overview/index.html#installation). The download links for this practical are:
16 |
17 | * Code and data: [practical-instance-recognition-2018a.tar.gz](http://www.robots.ox.ac.uk/~vgg/share/practical-instance-recognition-2018a.tar.gz) 560MB
18 | * Code only: [practical-instance-recognition-2018a-code-only.tar.gz](http://www.robots.ox.ac.uk/~vgg/share/practical-instance-recognition-2018a-code-only.tar.gz) 10MB
19 | * Data only: [practical-instance-recognition-2018a-data-only.tar.gz](http://www.robots.ox.ac.uk/~vgg/share/practical-instance-recognition-2018a-data-only.tar.gz) 550MB
20 | * [Git repository](https://github.com/vedaldi/practical-object-instance-recognition) (for lab setters and developers)
21 |
22 | After the installation is complete, open and edit the script `exercise1.m` in the MATLAB editor. The script contains commented code and a description for all steps of this exercise, relative to Part I of this document. You can cut and paste this code into the MATLAB window to run it, and will need to modify it as you go through the session. Other files `exercise2.m`, `exercise3.m`, and `exercise4.m` are given for Part II, III, and IV.
23 |
24 | **Note**: the student packages contain only the code required to run the practical. The complete package, including code to preprocess the data, is available on GitHub.
25 |
26 | ## Part I: Sparse features for matching object instances
27 |
28 | ### Stage I.A: SIFT features detector
29 |
30 | The SIFT feature has both a detector and a descriptor. We will start by computing and visualizing the SIFT feature detections for two images of the same object (a building facade). Load an image, rotate and scale it, and then display the original and transformed pair:
31 |
32 | ```matlab
33 | % Load an image
34 | im1 = imread('data/oxbuild_lite/all_souls_000002.jpg') ;
35 | % Let the second image be a rotated and scaled version of the first
36 | im3 = imresize(imrotate(im1,35,'bilinear'),0.7) ;% Display the images
37 | subplot(1,2,1) ; imagesc(im1) ; axis equal off ; hold on ;
38 | subplot(1,2,2) ; imagesc(im3) ; axis equal off ;
39 | ```
40 |
41 | A SIFT frame is a circle with an orientation and is specified by four parameters: the center $t_x$, $t_y$, the scale $s$, and the rotation $\theta$ (in radians), resulting in a vector of four parameters $(s, \theta, t_x, t_y)$. Now compute and visualise the SIFT feature detections (frames):
42 |
43 | ```matlab
44 | % Compute SIFT features for each
45 | [frames1, descrs1] = getFeatures(im1, 'peakThreshold', 0.01) ;
46 | [frames3, descrs3] = getFeatures(im3, 'peakThreshold', 0.01) ;
47 | subplot(1,2,1) ; imagesc(im1) ; axis equal off ; hold on ;
48 | vl_plotframe(frames1, 'linewidth', 2) ;
49 | subplot(1,2,2) ; imagesc(im3) ; axis equal off ; hold on ;
50 | vl_plotframe(frames3, 'linewidth', 2) ;
51 | ```
52 |
53 | Examine the second image and its rotated and scaled version and convince yourself that the detections overlap the same scene regions (even though the circles have moved their image position and changed radius). It is helpful to zoom into a smaller image area using the MATLAB magnifying glass tool. This demonstrates that the detection process transforms (is co-variant) with translations, rotations and isotropic scalings. This class of transformations is known as a similarity or equiform.
54 |
55 | > **Task:** The number of detected features can be controlled by changing the `peakThreshold` option. A larger value will select features that correspond to higher contrast structures in the image. Try this now: run again the same code, but increase `peakThreshold` two or three times.
56 |
57 | Now repeat the exercise with a pair of natural images. Start by loading the second one:
58 |
59 | ```matlab
60 | % Load a second image
61 | im2 = imread('data/oxbuild_lite/all_souls_000015.jpg') ;
62 | ```
63 |
64 | and plot images and feature frames. Again you should see that many of the detections overlap the same scene region. Note that, while repeatability occurs for the pair of natural views, it is much betterfor the synthetically rotated pair.
65 |
66 | > **Question:** Note the change in density of detections across the image. Why does it change? Will it be a problem for matching? How could it be avoided?
67 |
68 | > **Question:** Occasionally, a feature is detected multiple times, with different orientations. This may happen when the orientation assignment is ambiguous. Which kind of image structure would result in ambiguous orientation assignment?
69 |
70 | ### Stage I.B: SIFT features descriptors and matching between images
71 |
72 | Next we will use the descriptor computed over each detection to match the detections between images. We will start with the simplest matching scheme (first nearest neighbour of descriptors) and then add more sophisticated methods to eliminate any mismatches.
73 |
74 | * Visualize the SIFT descriptors for the detected feature frames with the function `vl_plotsiftdescriptor`. Then use `vl_plotframe` to overlay the corresponding frames.
75 |
76 | > **Question:** Note the descriptors are computed over a much larger region (shown in blue) than the detection (shown in green). Why?
77 |
78 | * Compute first nearest neighbours matches - for each SIFT descriptor in the first image, compute its nearest neighbour in the second image with the function `findNeighbours`.
79 |
80 | * Visualize the correspondences using lines joining matched SIFT features with the function `plotMatches`.
81 |
82 | > **Question:** Notice that there are many mismatches. Examine some of the mismatches to understand why the mistakes are being made. For example, is the change in lighting a problem? What additional constraints can be applied to remove the mismatches?
83 |
84 | **Hint:** You can visualize a subset of the matches using:
85 | ```matlab
86 | figure; plotMatches(im1,im2,frames1,frames2,matches(:,3:200:end));
87 | ```
88 |
89 | ### Stage I.C: Improving SIFT matching using Lowe’s second nearest neighbour test
90 |
91 | Lowe introduced a second nearest neighbour (2nd NN) test to identify, and hence remove, ambiguous matches. The idea is to identify distinctive matches by a threshold on the ratio of first to second NN distances. In the MATLAB file, the ratio is `nnThreshold` = 1NN distance / 2NN distance.
92 |
93 | * Vary the ratio `nnThreshold` in a range from 0.1 to 0.9, and examine how the number of matches and number of mismatches changes.
94 | * A value of `nnThreshold = 0.8` is often a good compromise between losing too many matches and rejecting mismatches.
95 |
96 | >**Question:** Examine some of the remaining mismatches to understand why they have occurred. How could they be removed?
97 |
98 | ### Stage I.D: Improving SIFT matching using a geometric transformation
99 |
100 | In addition to the 2nd NN test, we can also require consistency between the matches and a geometric transformation between the images. For the moment we will look for matches that are consistent with a similarity transformation
101 |
102 | $$
103 | \begin{bmatrix}
104 | x' \\ y'
105 | \end{bmatrix}
106 | =
107 | sR(\theta)
108 | \begin{bmatrix}
109 | x \\ y
110 | \end{bmatrix}
111 | +
112 | \begin{bmatrix}
113 | t_x \\ t_y
114 | \end{bmatrix}
115 | $$
116 |
117 | which consists of a rotation by $\theta$, an isotropic scaling (i.e. same in all directions) by s, and a translation by a vector $(t_x, t_y)$. This transformation is specified by four parameters $(s,\theta,t_x,t_y)$ and can be computed from a single correspondence between SIFT detections in each image.
118 |
119 | > **Task:** Work out how to compute this transformation from a single correspondence.
120 |
121 | **Hint:** Recall from Stage I.A that a SIFT feature frame is an oriented circle and map one onto the other.
122 |
123 | The matches consistent with a similarity can then be found using a RANSAC inspired algorithm, implemented by the function `geometricVerification`:
124 |
125 | **RANSAC-like algorithm for geometric verification**
126 |
127 | 1. For each tentative correspondence in turn:
128 | 1. compute the similarity transformation;
129 | 2. map all the SIFT detections in one image to the other using this transformation;
130 | 3. accept matches that are within a threshold distance to the mapped detection (inliers);
131 | 4. count the number of accepted matches;
132 | 5. optionally, fit a more accurate affine transformation or homography to the accepted matches and test re-validate the matches.
133 | 2. Choose the transformation with the highest count of inliers.
134 |
135 | ------
136 |
137 | After this algorithm the inliers are consistent with the transformation and are retained, and most mismatches should now be removed.
138 |
139 | > **Task:** The figure generated by `plotMatches` supports the interactive visualisation of the transformation found by `geometricVerification`. Try hovering with the mouse on the figure and check that corresponding image points are highlighted in the two images.
140 |
141 | **Skip to [Part 2](#part2) on fast track**
142 |
143 | > **Task:** Test this procedure by varying the threshold distance (edit `geometricVerification` and change the `opts.tolerance1`, `opts.tolerance2`, and `opts.tolerance3` parameters, where the last two thresholds are relative to the optional iterative fitting of an affine transformation or homography to the inliers). Note the number of inliers and number of mismatches.
144 |
145 | If more matches are required the geometric transformation can be used alone, without also requiring the 2nd NN test. Indeed, since the 1st NN may not be the correct match, a list of potential (putative) matches can be generated for each SIFT descriptor by including the 1st NN, 2nd NN, 3rd NN etc. Investigate how the number of correct matches (and time for computation) grows as the potential match list is extended, and the geometric transformation is used to select inliers. To this end:
146 |
147 | > **Task:** Change the code to include in the match list the 1st NN, 2nd NN, 3rd NN, … best matches for each feature.
148 | > **Task:** Run geometric verification and check the number of verified matches using this expanded list.
149 |
150 | **Hint:** You can use MATLAB’s tic and toc functions to measure the execution time of a snippet of code. For example
151 | ```matlab
152 | tic ; pause(3) ; toc
153 | ```
154 | will pause MATLAB for three seconds and return an elapsed time approximately equal to 3. See `help tic` for details.
155 |
156 | ## Part II: Affine co-variant detectors {#part2}
157 |
158 | So far the change in viewpoint between images has been a similarity transformation. Now we consider more severe viewpoint changes - for example where an object is fronto-parallel in one view, and turns away from the camera in the other as in the graffiti wall images below:
159 |
160 |
161 |
162 | In this case, there is foreshortening (anisotropic scaling) and perspective distortions between the images (as well as in-plane rotation, translation and scaling). A circle in one image cannot cover the same scene area as a circle in the other, but an ellipse can. Affine co-variant detectors are designed to find such regions.
163 |
164 | In the following we will compare the number of matches using a similarity and affine co-variant detector as the viewpoint becomes progressively more extreme. The detectors are SIFT (for similarity) and SIFT+affine adaptation (for affine), while the descriptor are in both cases SIFT.
165 |
166 | > **Task:** Open and examine the script `exercise2.m` in the MATLAB editor. Run the script.
167 |
168 | Note the behaviour in the number of verified matches as the viewpoint becomes more extreme. Observe that the matches also identify the regions of the images that are in common.
169 |
170 | > **Question:** The transformation between the images induced by the plane is a planar homography. The detections are only affine co-variant (not as general as a planar homography). So how can descriptors computed on these detections possibly match?
171 |
172 | **Note:** There are many other detector variants that could be used for this task. These can be activated by the method option of getFeatures.m (see also help `vl_covdet`).
173 |
174 | ## Part III: Towards large scale retrieval {#part3}
175 |
176 | In large scale retrieval the goal is to match a query image to a large database of images (for example the WWW or Wikipedia). The quality of a match is measured as the number of geometrically verified feature correspondences between the query and a database image. While the techniques discussed in Part I and II are sufficient to do this, in practice they require too much memory to store the SIFT descriptors for all the detections in all the database images. We explore next two key ideas: one to reduce the memory footprint and pre-compute descriptor matches; the other to speed up image retrieval.
177 |
178 | > **Task:** Open and edit the script `exercise3.m` in the MATLAB editor, and cut and paste to work through the following stages.
179 |
180 | ### Stage III.A: Accelerating descriptor matching with visual words
181 |
182 | Instead of matching feature descriptors directly as done in Part I and II, descriptors are usually mapped first to discrete symbols, also called visual words, by means of a clustering technique like K-Means. The descriptors that are assigned to the same visual word are considered matched. Each of the rows in the following figure illustrates image patches that are mapped to the same visual word, and are hence indistinguishable by the representation.
183 |
184 |
185 |
186 | Then, matching two sets of feature descriptors (from two images) reduces to finding the intersection of two sets of symbols.
187 |
188 | > **Tasks:**
189 |
190 | > * Load a visual word dictionary and an associated approximate nearest neighbour (ANN) matcher (the ANN matcher is used to determine the closest visual word to each descriptor and is based on a forest of KD trees).
191 | > * Given SIFT descriptors for two images, quantise them (assign them) into the corresponding visual words.
192 | > * Find corresponding features by looking for the same visual words in the two images and note the computation time.
193 | > * Geometrically verify these initial correspondences and count the number of inlier matches found.
194 | > * Find corresponding features by using the method of Part I and II, i.e. by comparing the descriptors directly, and note the computation time. Geometrically verify these initial correspondences and count the number of inlier matches found.
195 | > * Compare the speed and number of inliers when using visual words vs raw SIFT descriptors by means of the function `matchWords`. Note, you should repeat the timing (by running the matching again) as the first time you run it there may be a delay as certain MATLAB components are loaded into memory.
196 | > * **Optional:** compare the speed and number of matches over another pair of images (from part I and II).
197 |
198 | > **Questions:**
199 |
200 | > * The size of the vocabulary (the number of clusters) is an important parameter in visual word algorithms. How does the size affect the number of inliers and the difficulty of computing the transformation?
201 | > * In the above procedure the time required to convert the descriptors into visual words was not accounted for. Why?
202 | > * What is the speedup in searching a large, fixed database of 10, 100, 1000 images?
203 |
204 | **Skip to [Stage III.B](stage3b) on fast track**
205 |
206 | Often multiple feature occurrences are mapped to the same visual word. In this case `matchWords` generates only one of the possible matches.
207 |
208 | > **Tasks:**
209 |
210 | > * Modify `matchWords` to generate more than one match for cases in which multiple features are mapped to the same visual word.This can be achieved by increasing the value of `maxNumMatches`.
211 | > * Most of these additional matches are incorrect. Filter them out by running `geometricVerification`.
212 | > * Compare the number of inliers obtained before and after this modification.
213 |
214 | ### Stage III.B: Searching with an inverted index {#stage3b}
215 |
216 | While matching with visual words is much faster than doing so by comparing feature descriptors directly, scoring images directly based on the number of geometrically verified matches still entails fitting a geometric model, a relatively slow operation. Rather than scoring all the images in the database in this way, we are going to use an approximation and count the number of visual words shared between two images.
217 |
218 | To this end, one computes a histogram of the visual words in a query image and for each of the database images. Then the number of visual words in common can be computed from the intersection of the two histograms.
219 |
220 | The histogram intersection can be thought as a similarity measure between two histograms. In practice, this measure can be refined in several ways:
221 |
222 | * By reducing the importance of common visual words. This is similar to a stop-words list and can be implemented by weighting each word by the `inverse document frequency' (the inverse of the frequency of occurrence of that visual word over the entire database of images).
223 | * By normalising the weighted histograms to unit vectors and using the cosine between them as similarity. This can be implemented easily as the inner product between normalised histograms.
224 |
225 | Computing histogram similarities can be implemented extremely efficiently using an inverted file index. In this exercise, inner products between normalized histograms are computed quite efficiently using MATLAB's built-in sparse matrix engine.
226 |
227 | We now apply this retrieval method to search using a query image within a 660 image subset of the Oxford 5k building image set.
228 |
229 | > **Task:** How many erroneously matched images do you count in the top results?
230 | > **Question:** Why does the top image have a score of 1?
231 |
232 | ### Stage III.C: Geometric rescoring {#stage3c}
233 |
234 | Histogram-based retrieval results are good but far from perfect. Given a short list of top ranked images from the previous step, we are now going to re-score them based on the number of inlier matches after a geometric verification step.
235 |
236 | > **Question:** Why is the top score much larger than 1 now?
237 | > **Question:** Are the retrieval results improved after geometric verification?
238 |
239 | ### Stage III.D: Full system {#stage3d}
240 |
241 | Now try the full system to retrieve matches to an unseen query image.
242 |
243 |
244 |
245 | ## Part IV: Large scale retrieval
246 |
247 | **Skip and end here on fast track**
248 |
249 | The images below are all details of paintings. The goal of this last part of the practical is to identify the paintings that they came from. For this we selected a set of 1734 images of paintings from Wikipedia.
250 |
251 |
252 |
253 | To identify the details you can either:
254 |
255 | 1. use your knowledge of art
256 | 2. search through the 1734 Wikipedia images until you find matches
257 | 3. build a recognition system and match the details automatically
258 |
259 | We follow route (3) here. Look through and run `exercise4.m`. This uses the techniques described in Part III in order to construct an index for 1734 Wikipedia images so that they may be searched quickly. Use the code to find from which paintings these details come from.
260 |
261 | Note, although the index is stored locally, the matching images are downloaded from Wikipedia and displayed. Click on the image to reach the Wikipedia page for that painting (and hence identify it).
262 |
263 | > **Task:** Use the code to visually search Wikipedia for further paintings from Van Gogh downloaded from the Internet.
264 |
265 | **Note:** the code supports URL in place of filenames.
266 |
267 | Take note of the code output.
268 |
269 | > **Questions:**
270 |
271 | > * How many features are there in the painting database?
272 | > * How much memory does the image database take?
273 | > * What are the stages of the search? And how long does each of the stages take for one of the query images?
274 |
275 | That completes this practical.
276 |
277 | ## Links and further work
278 |
279 | * The code for this practical is written using the software package [VLFeat](http://www.vlfeat.org). This is a software library written in MATLAB and C, and is freely available as source code and binary.
280 | * The images for this practical are taken from the [Affine Covariant Features dataset](http://www.robots.ox.ac.uk/~vgg/research/affine/), and the [Oxford Buildings benchmark](http://www.robots.ox.ac.uk/~vgg/data/oxbuildings/).
281 | * For a tutorial on large scale visual search and references to the literature, see the lectures by Josef Sivic and Cordelia Schmid [here](http://www.di.ens.fr/willow/events/cvml2012/materials/).
282 | * For recent developments in large scale search (compact image descriptors, compression with product quantization), see [these lectures](https://sites.google.com/site/lsvr13/) by Herve Jegou and Florent Perronnin.
283 |
284 | ## Acknowledgements
285 |
286 | * Guidance from Josef Sivic, Ivan Laptev and Cordelia Schmid
287 | * Mircea Cimpoi for scripts for downloading and linking to Wikipedia paintings
288 | * Comments from Relja Arandjelovic, Karen Simonyan, Omkar Parkhi, Meelis Lootus, Hossein Azizpour, Max Jaderberg
289 | * Funding from ERC grant VisRec Grant No. 228180, and a PASCAL Harvest Grant.
290 |
291 |
292 |
293 | ## History
294 |
295 | * Used in the Oxford AIMS CDT, 2014-18
296 | * Used at [ENS/INRIA Visual Recognition and Machine Learning Summer School, 2012](http://www.di.ens.fr/willow/events/cvml2012/).
297 | * Used at [JHU Summer School on Human Language Technology, 2012](http://www.clsp.jhu.edu/workshops/archive/ws-12/summer-school/).
298 |
--------------------------------------------------------------------------------
/exercise1.m:
--------------------------------------------------------------------------------
1 | % PART I: basic features
2 |
3 | % setup MATLAB to use our software
4 | setup ;
5 |
6 | %% -------------------------------------------------------------------
7 | % Stage I.A: SIFT features detection
8 | % --------------------------------------------------------------------
9 |
10 | % Load an image
11 | im1 = imread('data/oxbuild_lite/all_souls_000002.jpg') ;
12 |
13 | % Let the second image be a rotated and scaled version of the first
14 | im3 = imresize(imrotate(im1,35,'bilinear'),0.7) ;
15 |
16 | % Display the images
17 | figure(1) ;
18 | set(gcf,'name', 'Part I.A: Original image and rotated and scaled version') ;
19 | subplot(1,2,1) ; imagesc(im1) ; axis equal off ; hold on ;
20 | subplot(1,2,2) ; imagesc(im3) ; axis equal off ;
21 |
22 | % Compute SIFT features for each
23 | [frames1, descrs1] = getFeatures(im1, 'peakThreshold', 0.01) ;
24 | [frames3, descrs3] = getFeatures(im3, 'peakThreshold', 0.01) ;
25 |
26 | figure(2) ;
27 | set(gcf,'name', 'Part I.A: SIFT features detection - synthetic pair') ;
28 | subplot(1,2,1) ; imagesc(im1) ; axis equal off ; hold on ;
29 | vl_plotframe(frames1, 'linewidth', 2) ;
30 |
31 | subplot(1,2,2) ; imagesc(im3) ; axis equal off ; hold on ;
32 | vl_plotframe(frames3, 'linewidth', 2) ;
33 |
34 | % Load a second image of the same scene
35 | im2 = imread('data/oxbuild_lite/all_souls_000015.jpg') ;
36 |
37 | % Display the images
38 | figure(3) ;
39 | set(gcf,'name', 'Part I.A: Original images - real pair') ;
40 | subplot(1,2,1) ; imagesc(im1) ; axis equal off ; hold on ;
41 | subplot(1,2,2) ; imagesc(im2) ; axis equal off ;
42 |
43 | [frames2, descrs2] = getFeatures(im2, 'peakThreshold', 0.01) ;
44 |
45 | figure(4) ;
46 | set(gcf,'name', 'Part I.A: SIFT features detection - real pair') ;
47 | subplot(1,2,1) ; imagesc(im1) ; axis equal off ; hold on ;
48 | vl_plotframe(frames1, 'linewidth', 2) ;
49 |
50 | subplot(1,2,2) ; imagesc(im2) ; axis equal off ; hold on ;
51 | vl_plotframe(frames2, 'linewidth', 2) ;
52 |
53 | %% -------------------------------------------------------------------
54 | % Stage I.B: SIFT features descriptors and matching between images
55 | % --------------------------------------------------------------------
56 |
57 | % Visualize SIFT descriptors (only a few)
58 | figure(5) ; clf ;
59 | set(gcf,'name', 'Part I.B: SIFT descriptors') ;
60 | imagesc(im1) ; axis equal off ;
61 | vl_plotsiftdescriptor(descrs1(:,1:50:end), ...
62 | frames1(:,1:50:end)) ;
63 | hold on ;
64 | vl_plotframe(frames1(:,1:50:end)) ;
65 |
66 | % Find for each descriptor in im1 the closest descriptor in im2
67 | nn = findNeighbours(descrs1, descrs2) ;
68 |
69 | % Construct a matrix of matches. Each column stores two index of
70 | % matching features in im1 and im2
71 | matches = [1:size(descrs1,2) ; nn(1,:)] ;
72 |
73 | % Display the matches
74 | figure(6) ; clf ;
75 | set(gcf,'name', 'Part I.B: SIFT descriptors - matching') ;
76 | plotMatches(im1,im2,frames1,frames2,matches) ;
77 | title('Nearest neighbour matches') ;
78 |
79 | %% -------------------------------------------------------------------
80 | % Stage I.C: Better matching w/ Lowe's second nearest neighbour test
81 | % --------------------------------------------------------------------
82 |
83 | % Find the top two neighbours as well as their distances
84 | [nn, dist2] = findNeighbours(descrs1, descrs2, 2) ;
85 |
86 | % Accept neighbours if their second best match is sufficiently far off
87 | nnThreshold = 0.8 ;
88 | ratio2 = dist2(1,:) ./ dist2(2,:) ;
89 | ok = ratio2 <= nnThreshold^2 ;
90 |
91 | % Construct a list of filtered matches
92 | matches_2nn = [find(ok) ; nn(1, ok)] ;
93 |
94 | % Alternatively, do not do the second nearest neighbourhood test.
95 | % Instead, match each feature to its two closest neighbours and let
96 | % the geometric verification step figure it out (in stage I.D below).
97 |
98 | % matches_2nn = [1:size(nn,2), 1:size(nn,2) ; nn(1,:), nn(2,:)] ;
99 |
100 | % Display the matches
101 | figure(7) ; clf ;
102 | set(gcf,'name', 'Part I.C: SIFT descriptors - Lowe''s test') ;
103 | plotMatches(im1,im2,frames1,frames2,matches_2nn) ;
104 | title('Matches filtered by the second nearest neighbour test') ;
105 |
106 | %% -------------------------------------------------------------------
107 | % Stage I.D: Better matching w/ geometric transformation
108 | % --------------------------------------------------------------------
109 |
110 | [inliers, H] = geometricVerification(frames1, frames2, matches_2nn, 'numRefinementIterations', 8) ;
111 | matches_geo = matches_2nn(:, inliers) ;
112 |
113 | % Display the matches
114 | figure(8) ; clf ;
115 | set(gcf,'name', 'Part I.D: SIFT descriptors - geometric verification') ;
116 | plotMatches(im1,im2,frames1,frames2,matches_geo, 'homography', H) ;
117 | title('Matches filtered by geometric verification') ;
118 |
--------------------------------------------------------------------------------
/exercise2.m:
--------------------------------------------------------------------------------
1 | % PART II: Affine co-variant detectors
2 |
3 | % setup MATLAB to use our software
4 | setup ;
5 |
6 | % choose which images to use in the evaluation
7 | imgPaths = {'data/graf/img1.png',...
8 | 'data/graf/img2.png',...
9 | 'data/graf/img3.png',...
10 | 'data/graf/img4.png',...
11 | 'data/graf/img5.png',...
12 | 'data/graf/img6.png'} ;
13 |
14 | figure(100) ; clf ; set(gcf, 'name', 'Part II: Affine co-variant detectors - summary') ;
15 |
16 | for o = 1:2
17 | % Get the features for the reference image
18 | im1 = imread(imgPaths{1}) ;
19 | [frames1,descrs1] = getFeatures(im1, 'affineAdaptation',o==2) ;
20 |
21 | for t = 2:numel(imgPaths)
22 | % Get the feature for another image
23 | im2 = imread(imgPaths{t}) ;
24 | [frames2,descrs2] = getFeatures(im2, 'affineAdaptation',o==2) ;
25 |
26 | % Get the feature descriptor neighbours
27 | [nn, dist2] = findNeighbours(descrs1, descrs2, 2) ;
28 |
29 | % Second nearest neighbour pre-filtering
30 | nnThreshold = 0.8 ;
31 | ratio2 = dist2(1,:) ./ dist2(2,:) ;
32 | ok = ratio2 <= nnThreshold^2 ;
33 | matches_2nn = [find(ok) ; nn(1,ok)] ;
34 |
35 | % Geometric verification
36 | [inliers, H] = geometricVerification(frames1,frames2,matches_2nn,...
37 | 'numRefinementIterations', 6) ;
38 | matches_geom = matches_2nn(:, inliers) ;
39 |
40 | % Count the number of inliers
41 | numInliers(t,o) = size(matches_geom,2) ;
42 |
43 | % Visualize
44 | n = (t-2)*2 + o ;
45 | h = subplot(numel(imgPaths)-1, 2, n, 'parent', 100) ;
46 | axes(h) ;
47 | plotMatches(im1,im2,frames1,frames2,matches_geom) ;
48 | switch o
49 | case 1, type = 'similarity' ;
50 | case 2, type = 'affinity' ;
51 | end
52 | title(sprintf('From 1 to %d with %s: num: %d', t, type, numInliers(t,o))) ;
53 |
54 | figure(n) ; clf ;
55 | set(gcf, 'name', sprintf('Part II: Affine co-variant detectors - from 1 to %d with %s', t,type)) ;
56 | plotMatches(im1,im2,frames1,frames2,matches_geom,'homography',H) ;
57 | %c = copyobj(h, gcf) ; set(c, 'position', [0 0 1 1]) ;
58 | drawnow ;
59 | end
60 | end
61 |
62 | % Quantitative evaluation
63 | figure(101) ; clf ;
64 | set(gcf, 'name', sprintf('Part II: Affine co-variant detectors - quantitative comparison')) ;
65 | plot(2:size(numInliers,1),numInliers(2:end,:),'linewidth', 3) ;
66 | axis tight ; grid on ;
67 | legend('similarity co-variant', 'affine co-variant') ;
68 | xlabel('image pair') ;
69 | ylabel('num. verified feature matches') ;
70 |
--------------------------------------------------------------------------------
/exercise3.m:
--------------------------------------------------------------------------------
1 | % PART III: Towards large-scale retrieval
2 |
3 | % setup MATLAB to use our software
4 | setup ;
5 |
6 | %% -------------------------------------------------------------------
7 | % Stage III.A: Accelerating descriptor matching with visual words
8 | % --------------------------------------------------------------------
9 |
10 | % Load a visual word vocabulary
11 | load('data/oxbuild_lite_imdb_100k_disc_dog.mat', 'vocab', 'kdtree') ;
12 |
13 | % Load the two images
14 | im1 = imread('data/oxbuild_lite/ashmolean_000007.jpg') ;
15 | im2 = imread('data/oxbuild_lite/ashmolean_000028.jpg') ;
16 |
17 | % Compute SIFT features for each
18 | [frames1, descrs1] = getFeatures(im1, 'peakThreshold', 0.001, 'orientation', false) ;
19 | [frames2, descrs2] = getFeatures(im2, 'peakThreshold', 0.001, 'orientation', false) ;
20 |
21 | % Get the matches based on the raw descriptors
22 | tic ;
23 | [nn, dist2] = findNeighbours(descrs1, descrs2, 2) ;
24 | nnThreshold = 0.85 ;
25 | ratio2 = dist2(1,:) ./ dist2(2,:) ;
26 | ok = ratio2 <= nnThreshold^2 ;
27 | matches_raw = [find(ok) ; nn(1,ok)] ;
28 | time_raw = toc ;
29 |
30 | % Quantise the descriptors
31 | words1 = vl_kdtreequery(kdtree, vocab, descrs1, 'maxNumComparisons', 1024) ;
32 | words2 = vl_kdtreequery(kdtree, vocab, descrs2, 'maxNumComparisons', 1024) ;
33 |
34 | % Get the matches based on the quantized descriptors
35 | tic ;
36 | matches_word = matchWords(words1,words2) ;
37 | time_word = toc;
38 |
39 | % Count inliers
40 | [inliers_raw, H_raw] = geometricVerification(frames1,frames2,matches_raw,'numRefinementIterations', 3) ;
41 | [inliers_word, H_word] = geometricVerification(frames1,frames2,matches_word,'numRefinementIterations', 3) ;
42 |
43 | figure(1) ; clf ;
44 | set(gcf,'name', 'III.B: Accelerating descriptor matching with visual words') ;
45 |
46 | subplot(2,1,1) ; plotMatches(im1,im2,frames1,frames2,matches_raw(:,inliers_raw), 'homography', H_raw) ;
47 | title(sprintf('Verified matches on raw descriptors (%d in %.3g s)',numel(inliers_raw),time_raw)) ;
48 |
49 | subplot(2,1,2) ; plotMatches(im1,im2,frames1,frames2,matches_word(:,inliers_word), 'homography', H_word) ;
50 | title(sprintf('Verified matches on visual words (%d in %.3g s)',numel(inliers_word),time_word)) ;
51 |
52 | %% -------------------------------------------------------------------
53 | % Stage III.B: Searching with an inverted index
54 | % --------------------------------------------------------------------
55 |
56 | % Load an image DB
57 | imdb = loadIndex('data/oxbuild_lite_imdb_100k_ellipse_dog.mat') ;
58 |
59 | % Compute a histogram for the query image
60 | [h,frames,words] = getHistogramFromImage(imdb, im2) ;
61 |
62 | % Score the other images by similarity to the query
63 | tic ;
64 | scores = h' * imdb.index ;
65 | time_index = toc ;
66 |
67 | % Plot results by decreasing score
68 | figure(2) ; clf ;
69 | plotRetrievedImages(imdb, scores, 'num', 25) ;
70 | set(gcf,'name', 'III.B: Searching with an inverted index') ;
71 | fprintf('Search time per database image: %.3g s\n', time_index / size(imdb.index,2)) ;
72 |
73 | %% -------------------------------------------------------------------
74 | % Stage III.C: Geometric reranking
75 | % --------------------------------------------------------------------
76 |
77 | % Rescore the top 16 images based on the number of
78 | % inlier matches.
79 |
80 | [~, perm] = sort(scores, 'descend') ;
81 | for rank = 1:25
82 | matches = matchWords(words,imdb.images.words{perm(rank)}) ;
83 | inliers = geometricVerification(frames,imdb.images.frames{perm(rank)},...
84 | matches,'numRefinementIterations', 3) ;
85 | newScore = numel(inliers) ;
86 | scores(perm(rank)) = max(scores(perm(rank)), newScore) ;
87 | end
88 |
89 | % Plot results by decreasing score
90 | figure(3) ; clf ;
91 | plotRetrievedImages(imdb, scores, 'num', 25) ;
92 | set(gcf,'name', 'III.B: Searching with an inverted index - verification') ;
93 |
94 | %% -------------------------------------------------------------------
95 | % Stage III.D: Full system
96 | % --------------------------------------------------------------------
97 |
98 | % Load the database if not already in memory or if it is the one
99 | % from exercise4
100 | if ~exist('imdb', 'var') || isfield(imdb.images, 'wikiNames')
101 | imdb = loadIndex('data/oxbuild_lite_imdb_100k_ellipse_dog.mat', ...
102 | 'sqrtHistograms', true) ;
103 | end
104 |
105 | % Search the database for a match to a given image. Note that URL
106 | % can be a path to a file or a URL pointing to an image in the
107 | % Internet.
108 |
109 | url1 = 'data/queries/mistery-building1.jpg' ;
110 | res = search(imdb, url1, 'box', []) ;
111 |
112 | % Display the results
113 | figure(4) ; clf ; set(gcf,'name', 'Part III.D: query image') ;
114 | plotQueryImage(imdb, res) ;
115 |
116 | figure(5) ; clf ; set(gcf,'name', 'Part III.D: search results') ;
117 | plotRetrievedImages(imdb, res) ;
118 |
--------------------------------------------------------------------------------
/exercise4.m:
--------------------------------------------------------------------------------
1 | % PART IV: Large scale image retrieval
2 |
3 | % setup MATLAB to use our software
4 | setup ;
5 |
6 | % Load the database if not already in memory, or if the one
7 | % is from exercise3.
8 | if ~exist('imdb', 'var') || ~isfield(imdb.images,'wikiName')
9 | imdb = loadIndex('data/paintings_imdb_100k_disc_dog.mat', ...
10 | 'sqrtHistograms', true) ;
11 | imdb.dir = '' ; % art images are not shipped with practical
12 | end
13 |
14 | % Search the database for a match to a given image. Note that URL
15 | % can be a path to a file or a URL pointing to an image in the
16 | % Internet.
17 |
18 | url1 = 'data/queries/mistery-painting1.jpg' ;
19 | url2 = 'data/queries/mistery-painting2.jpg' ;
20 | url3 = 'data/queries/mistery-painting3.jpg' ;
21 | res = search(imdb, url1, 'box', []) ;
22 |
23 | % Display the results
24 | figure(1) ; clf ; set(gcf,'name', 'Part IV: query image') ;
25 | plotQueryImage(imdb, res) ;
26 |
27 | figure(2) ; clf ; set(gcf,'name', 'Part IV: search results') ;
28 | plotRetrievedImages(imdb, res, 'num', 9) ;
29 |
--------------------------------------------------------------------------------
/extra/Makefile:
--------------------------------------------------------------------------------
1 | name ?= practical-instance-recognition
2 | ver ?= 2018a
3 |
4 | code=\
5 | exercise1.m \
6 | exercise2.m \
7 | exercise3.m \
8 | exercise4.m \
9 | findNeighbours.m \
10 | geometricVerification.m \
11 | getFeatures.m \
12 | getHistogramFromImage.m \
13 | getHistogram.m \
14 | loadIndex.m \
15 | matchWords.m \
16 | plotMatches.m \
17 | plotQueryImage.m \
18 | plotRetrievedImages.m \
19 | search.m \
20 | setup.m \
21 | README.md \
22 | vlfeat
23 |
24 | doc=\
25 | doc/images \
26 | doc/base.css \
27 | doc/instructions.html
28 |
29 | data=\
30 | data/oxbuild_lite \
31 | data/oxbuild_lite_imdb.mat \
32 | data/oxbuild_lite_imdb_100k_ellipse_dog.mat \
33 | data/oxbuild_lite_imdb_100k_disc_dog.mat \
34 | data/paintings_imdb_100k_disc_dog.mat \
35 | data/queries \
36 | data/graf
37 |
38 | include extra/practical/Makefile
39 |
--------------------------------------------------------------------------------
/extra/annkmeans.m:
--------------------------------------------------------------------------------
1 | function [centers, tree, en] = annkmeans(X, K, varargin)
2 | % ANNKMEANS Approximate Nearest Neighbors K-Means
3 | %
4 | % Example:: To cluster the data X into K parts:
5 | % [CENTERS, EN] = ANNKMEANS(X, K) ;
6 | %
7 | % Options are:
8 | %
9 | % Seed:: 0
10 | % Random number generation seed (for initialization).
11 | %
12 | % NumTrees:: 3
13 | % Number of trees in the kd-tree forest.
14 | %
15 | % MaxNumComparisons:: 500
16 | % Maximum number of comparisons when querying the kd-tree.
17 | %
18 | % MaxNumIterations:: 100
19 | % Maximum number of k-means iterations.
20 |
21 | % Andrea Vedaldi
22 |
23 | opts.seed = 0 ;
24 | opts.numTrees = 3 ;
25 | opts.maxNumComparisons = 500 ;
26 | opts.maxNumIterations = 100 ;
27 | opts.verbose = 0 ;
28 | opts.tolerance = 0.001 ;
29 | opts = vl_argparse(opts, varargin) ;
30 |
31 | % get initial centers
32 | rand('state',opts.seed) ;
33 | centers = vl_colsubset(X, K) ;
34 |
35 | if opts.verbose
36 | fprintf('%s: clustering %d vectors into %d parts\n', ...
37 | mfilename, size(X,2), K) ;
38 | fprintf('%s: random seed = %g\n', mfilename, opts.seed) ;
39 | fprintf('%s: tolerance = %g\n', mfilename, opts.tolerance) ;
40 | fprintf('%s: numTrees = %d\n', mfilename, opts.numTrees) ;
41 | fprintf('%s: maxNumComparisons = %d\n', mfilename, opts.maxNumComparisons) ;
42 | fprintf('%s: maxNumIterations = %d\n', mfilename, opts.maxNumIterations) ;
43 | end
44 |
45 | % chunk the data up
46 | numData = size(X,2) ;
47 | numChunks = max(matlabpool('size'), 1) ;
48 | data = Composite() ;
49 | dist = Composite() ;
50 | assign = Composite() ;
51 |
52 | for i = 1:numChunks
53 | chunk = i:numChunks:numData ;
54 | data{i} = X(:, chunk) ;
55 | dist{i} = inf(1, numel(chunk), class(X)) ;
56 | assign{i} = zeros(1, numel(chunk)) ;
57 | end
58 | %clear X ;
59 |
60 | E = [] ;
61 |
62 | for t = 1:opts.maxNumIterations
63 | % compute kd-tree
64 | tree = vl_kdtreebuild(centers, 'numTrees', opts.numTrees) ;
65 |
66 | % get the updated cluster assignments and partial centers
67 | spmd
68 | [centers_, mass_, en_, assign, dist] = update(opts, ...
69 | data,K,centers,tree,...
70 | assign,dist) ;
71 | end
72 |
73 | % compute the new cluster centers
74 | centers = zeros(size(centers),class(centers)) ;
75 | mass = zeros(1,K);
76 | en = 0 ;
77 | for i = 1:length(centers_)
78 | centers = centers + centers_{i} ;
79 | mass = mass + mass_{i} ;
80 | en = en + en_{i} ;
81 | end
82 | centers = bsxfun(@times, centers, 1./max(mass,eps)) ;
83 | E(t) = en ;
84 |
85 | % re-initialize any center with no mass
86 | rei = find(mass == 0) ;
87 | centers(:, rei) = vl_colsubset(X, length(rei)) ;
88 |
89 | if opts.verbose
90 | figure(1) ; clf ;
91 | plot(E,'linewidth', 2) ;
92 | xlim([1 opts.maxNumIterations]) ;
93 | title(sprintf('%s: iteration %d', mfilename, t)) ;
94 | xlabel('iterations') ;
95 | ylabel('energy') ;
96 | grid on ; drawnow ;
97 | fprintf('%s: %d: energy = %g, reinitialized = %d\n', mfilename,t,E(t),length(rei)) ;
98 | end
99 |
100 | if t > 1 && E(t) > (1 - opts.tolerance) * E(t-1), break ; end
101 | end
102 |
103 | % prepare final resutls
104 | en = E(end) ;
105 |
106 | % --------------------------------------------------------------------
107 | function [centers, mass, en, assign, dist] = ...
108 | update(opts,X,K,centers,tree,assign,dist)
109 | % --------------------------------------------------------------------
110 |
111 | [assign_, dist_] = vl_kdtreequery(tree, centers, X, ...
112 | 'maxComparisons', opts.maxNumComparisons) ;
113 | ok = dist_ < dist ;
114 | assign(ok) = assign_(ok) ;
115 | dist(ok) = dist_(ok) ;
116 |
117 | for b = 1:K
118 | centers(:, b) = sum(X(:, assign == b),2) ;
119 | mass(b) = sum(assign == b) ;
120 | end
121 | en = sum(dist) ;
122 |
--------------------------------------------------------------------------------
/extra/bootstrap-data.sh:
--------------------------------------------------------------------------------
1 | #!/bin/sh
2 |
3 | mkdir -p data/archives
4 |
5 | if test ! -d data/graf
6 | then
7 | wget -c -nc \
8 | http://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/graf.tar.gz \
9 | -O data/archives/graf.tgz
10 | mkdir -p data/graf
11 | (cd data/graf ; tar xzvf ../archives/graf.tgz)
12 | (cd data/graf ; rm -f *.png ; mogrify -format png *.ppm)
13 | (cd data/graf ; rm -f *.ppm)
14 | fi
15 |
16 | cp -vr extra/queries data/
17 |
18 | if test ! -d data/oxbuild_images
19 | then
20 | wget -c -nc \
21 | http://www.robots.ox.ac.uk/~vgg/data/oxbuildings/oxbuild_images.tgz \
22 | -O data/archives/oxbuild_images.tgz
23 | mkdir -p data/oxbuild_images
24 | (cd data/oxbuild_images ; tar xvf ../archives/oxbuild_images.tgz)
25 | fi
26 |
27 | if test ! -d data/oxbuild_gt
28 | then
29 | wget -c -nc \
30 | http://www.robots.ox.ac.uk/~vgg/data/oxbuildings/gt_files_170407.tgz \
31 | -O data/archives/gt_files_170407.tgz
32 | mkdir -p data/oxbuild_gt
33 | (cd data/oxbuild_gt ; tar xvf ../archives/gt_files_170407.tgz)
34 | fi
35 |
36 | if test ! -d data/oxbuild_compute_ap.cpp
37 | then
38 | wget -c -nc \
39 | http://www.robots.ox.ac.uk/~vgg/data/oxbuildings/compute_ap.cpp \
40 | -O data/archives/compute_ap.cpp
41 | (cd data ; cp -vf archives/compute_ap.cpp oxbuild_compute_ap.cpp)
42 | fi
43 |
44 | # Create a lite version
45 | if test ! -d data/oxbuild_lite
46 | then
47 | mkdir -p data/oxbuild_lite
48 | (
49 | ls -1 data/oxbuild_gt/*_{good,ok}.txt | sort | xargs cat
50 | ls -1 data/oxbuild_gt/*_junk.txt | sort | xargs cat | head -n 300
51 | ) | sort | uniq > data/oxbuild_lite.txt
52 | cat data/oxbuild_lite.txt | sed "s/^\(.*\)$/data\/oxbuild_images\/\1.jpg/" | xargs -I % cp -v % data/oxbuild_lite
53 | fi
54 |
--------------------------------------------------------------------------------
/extra/evaluate.m:
--------------------------------------------------------------------------------
1 | function evaluate()
2 | % EVALUATE
3 |
4 | % prepare index
5 | switch 13
6 | case 1, imdbPath = 'data/oxbuild_imdb_100k_disc.mat' ;
7 | case 2, imdbPath = 'data/oxbuild_imdb_100k_odisc.mat' ;
8 | case 3, imdbPath = 'data/oxbuild_imdb_100k_ellipse.mat' ;
9 | case 4, imdbPath = 'data/oxbuild_imdb_100k_oellipse.mat' ;
10 | case 11, imdbPath = 'data/oxbuild_imdb_100k_disc_dog2.mat' ;
11 | case 12, imdbPath = 'data/oxbuild_imdb_100k_odisc_dog.mat' ;
12 | case 13, imdbPath = 'data/oxbuild_imdb_100k_ellipse_dog2.mat' ;
13 | case 14, imdbPath = 'data/oxbuild_imdb_100k_oellipse_dog.mat' ;
14 | end
15 | imdb = loadIndex(imdbPath, 'sqrtHistograms', false, 'shortListSize', 200) ;
16 |
17 | % run evaluation
18 | load('data/oxbuild_query.mat', 'query') ;
19 | diary([imdbPath(1:end-4) '.txt']) ;
20 | diary on ;
21 | fprintf('imdb:\n') ;
22 | disp(imdb) ;
23 | results = doEvaluation(imdb, query) ;
24 | diary off ;
25 |
26 | % --------------------------------------------------------------------
27 | function results = doEvaluation(imdb, query)
28 | % --------------------------------------------------------------------
29 |
30 | results = cell(1,numel(query)) ;
31 | for i = 1:numel(query)
32 | k = find(imdb.images.id == query(i).imageId) ;
33 | assert(~isempty(k)) ;
34 |
35 | % database labels for evaluation in retrieval (make sure we
36 | % ignore the query image too)
37 | y = - ones(1, numel(imdb.images.id)) ;
38 | y(query(i).good) = 1 ;
39 | y(query(i).ok) = 1 ;
40 | y(query(i).junk) = 0 ;
41 | y(k) = 0 ; % ooops ?
42 |
43 | results{i} = search(imdb, imdb.images.id(k), ...
44 | 'box', query(i).box, ...
45 | 'verbose', false) ;
46 |
47 | [rc,pr,info] = vl_pr(y, results{i}.index_scores) ;
48 | results{i}.index_rc = rc ;
49 | results{i}.index_pr = pr ;
50 | results{i}.index_ap = info.ap ;
51 |
52 | [rc,pr,info] = vl_pr(y, results{i}.geom_scores) ;
53 | results{i}.geom_rc = rc ;
54 | results{i}.geom_pr = pr ;
55 | results{i}.geom_ap = info.ap ;
56 |
57 | fprintf('query %03d: %-20s mAP:%5.2f mAP+geom:%5.2f\n', i, ...
58 | query(i).name, results{i}.index_ap, results{i}.geom_ap) ;
59 |
60 | if 0
61 | figure(10) ; clf ; hold on ;
62 | plot(results(i).index_rc, results(i).index_pr, 'color', 'b', 'linewidth', 3) ;
63 | plot(results(i).geom_rc, results(i).geom_pr, 'color', 'g', 'linewidth', 2) ;
64 | grid on ; axis equal ;
65 | title(sprintf('%s', query(i).name), 'interpreter', 'none') ;
66 | legend(sprintf('index: %.2f',results(i).index_ap*100), ...
67 | sprintf('index+geom: %.2f',results(i).geom_ap*100)) ;
68 | xlim([0 1]) ;ylim([0 1]);
69 | drawnow ;
70 | end
71 | end
72 |
73 | results = cat(2, results{:}) ;
74 |
75 | fprintf('features: time %.2g\n', ...
76 | mean([results.features_time])) ;
77 | fprintf('index: mAP: %g, time: %.2f\n', ...
78 | mean([results.index_ap])*100, ...
79 | mean([results.index_time])) ;
80 | fprintf('index+geom: mAP: %g, time: %.2f\n', ...
81 | mean([results.geom_ap])*100, ...
82 | mean([results.geom_time])) ;
83 |
--------------------------------------------------------------------------------
/extra/experimental/crawler/crawl_all_paintings.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/
2 |
3 | import os
4 | from selenium import webdriver
5 | import time
6 | import urllib
7 | import sys
8 | import string
9 | import traceback
10 |
11 | from selenium.common.exceptions import ElementNotVisibleException
12 |
13 |
14 | def get_next_page_link(browser):
15 | pager_elements = browser.find_element_by_class_name("pager-items")
16 |
17 | pagelinks = pager_elements.find_elements_by_tag_name("a")
18 |
19 | for pagelink in pagelinks:
20 | if pagelink.text == "Next":
21 | return pagelink
22 | return None
23 |
24 | def get_painters(browser):
25 | all_painter_divs = browser.find_elements_by_class_name("search-item")
26 |
27 | painter_names = [];
28 | painter_urls = [];
29 |
30 | print browser.current_url
31 | for painterdiv in all_painter_divs:
32 | ahref = painterdiv.find_element_by_tag_name("a");
33 | painter_name = ahref.text
34 | idx = painter_name.find('\n')
35 | painter_names.append(painter_name[:idx])
36 | painter_url = ahref.get_attribute("href");
37 | painter_urls.append(painter_url)
38 |
39 | return {'painter_names' : painter_names, 'painter_links' : painter_urls }
40 |
41 |
42 | startAt = 0
43 |
44 | DataDir = '/data/datasets/paintings/'
45 |
46 | do_download = True
47 | do_overwrite = False
48 |
49 | if not os.path.exists(DataDir):
50 | os.mkdir(DataDir)
51 |
52 |
53 | browser = webdriver.Firefox()
54 | browser.implicitly_wait(60)
55 |
56 | painter_list = []
57 | painter_urls = []
58 | for letter in string.uppercase[:26]:
59 | browser.get('http://www.wikipaintings.org/en/alphabet/' + letter)
60 | painter_elements = get_painters(browser)
61 | painter_list = painter_list + painter_elements['painter_names']
62 | #painter_list = painter_list + painter_elemens
63 | painter_urls = painter_urls + painter_elements['painter_links']
64 |
65 | paintings_artpage_list = []
66 | paintings_name_list = []
67 | paintings_src_list = []
68 |
69 |
70 | # write painters
71 | with open(os.path.join(DataDir, 'all_painter_list.txt'), 'wb') as fp:
72 | for ii in range(0, len(painter_list)):
73 | fp.write((painter_urls[ii] + "\t" + painter_list[ii] + "\n").encode("UTF-8"))
74 | fp.closed
75 |
76 | for ii in range(startAt, len(painter_urls)):
77 | idx_slash = painter_urls[ii].rfind('/') + 1
78 | painter_name = painter_urls[ii][idx_slash:]
79 |
80 | if os.path.exists(os.path.join(DataDir, 'lists', 'lst_' + painter_name + '_detailed.txt')):
81 | continue
82 |
83 | try:
84 | with open(os.path.join(DataDir, 'lists', 'lst_' + painter_name + '_detailed.txt'), 'wb') as fdet:
85 | with open(os.path.join(DataDir, 'lists', 'lst_' + painter_name + '_download.txt'), 'wb') as fdown:
86 |
87 | print str(ii) + " " + painter_list[ii] + " " + painter_urls[ii]
88 | #hack to fit on the screen if painter portrait is larger
89 | browser.get(painter_urls[ii] + '/mode/all-paintings/')
90 | crt_painting_urls = []
91 | crt_painting_names = []
92 | crt_paintings_src = []
93 | # click on the current / style link to show the jcarousel
94 | try:
95 | paintings_div = browser.find_element_by_id("paintings")
96 |
97 | time.sleep(1)
98 | # click on the first painting
99 |
100 | container_a = paintings_div.find_element_by_class_name("mr20")
101 | image_element = container_a.find_element_by_tag_name("img")
102 | image_element.click()
103 | time.sleep(1)
104 |
105 | current_ul = browser.find_element_by_id("artistPaintings")
106 | container_a = current_ul.find_element_by_class_name("rimage")
107 | image_element = container_a.find_element_by_tag_name("img")
108 | image_element.click()
109 | print browser.current_url
110 | except ElementNotVisibleException:
111 | # click on the first painting
112 | paintings_div = browser.find_element_by_id("paintings")
113 |
114 | time.sleep(1)
115 | # click on the first painting
116 |
117 | container_a = paintings_div.find_element_by_class_name("mr20")
118 | image_element = container_a.find_element_by_tag_name("img")
119 | image_element.click()
120 | time.sleep(1)
121 |
122 | current_ul = browser.find_element_by_id("artistPaintings")
123 | container_a = current_ul.find_element_by_class_name("rimage")
124 | image_element = container_a.find_element_by_tag_name("img")
125 | image_element.click()
126 |
127 | time.sleep(1)
128 |
129 | span_total = browser.find_element_by_class_name("totalslides")
130 | total_paintings_str = span_total.get_attribute("innerHTML")
131 | try:
132 | total_paintings = int(total_paintings_str)
133 | except:
134 | total_paintings_str = span_total.get_attribute("innerHTML")
135 | total_paintings = int(total_paintings_str)
136 |
137 | for pp in range(0, total_paintings):
138 | time.sleep(1)
139 | active_slide = browser.find_element_by_class_name("activeslide")
140 | nextslide_link = browser.find_element_by_id("nextslide")
141 |
142 | crt_url = browser.current_url
143 | idx_dash = crt_url.rfind('-') + 1
144 |
145 | img_id = crt_url[idx_dash :]
146 |
147 | galleryData = browser.find_element_by_id("galleryData")
148 | pelems = galleryData.find_elements_by_tag_name("p")
149 | pelem = pelems[0];
150 | ahref = pelem.find_element_by_tag_name("a")
151 | href_text = ahref.text
152 | artwork_link = pelem.find_element_by_tag_name("a").get_attribute("href")
153 |
154 | genre = "N/A"
155 | style = "N/A"
156 | year = "N/A"
157 |
158 | for pelem in pelems:
159 | if (pelem.text.lower().startswith("genre")):
160 | genre = pelem.find_element_by_tag_name("span").text
161 | if (pelem.text.lower().startswith("style")):
162 | style = pelem.find_element_by_tag_name("span").text
163 | if (pelem.text.lower().startswith("completion")):
164 | year = pelem.find_element_by_tag_name("span").text
165 |
166 |
167 | img_src = active_slide.find_element_by_tag_name("img").get_attribute("src")
168 | exclamation = img_src.find('!');
169 | if exclamation > -1:
170 | img_src = img_src[:exclamation]
171 | # Avoid copyright protected images
172 | if (img_src.lower() ==
173 | "http://cdnc.wikipaintings.org/zur/Global/Images/Global/FRAME-600x480.jpg".lower()):
174 | continue
175 | #there should be a smarter way to do this
176 | fdet.write((img_id + u".jpg\t" + img_src + '\t' +
177 | artwork_link + '\t' + href_text + '\t' +
178 | painter_list[ii] + '\t' + style + '\t' + genre +
179 | '\t' + year + '\n').encode('UTF-8'))
180 | fdown.write(img_src + u'\n')
181 |
182 | if (do_download):
183 | dst_dir = os.path.join(DataDir, 'images', img_id[0:4])
184 | dst_file = os.path.join(dst_dir, img_id + ".jpg")
185 | if (not os.path.exists(dst_file) or do_overwrite):
186 | if (not os.path.exists(dst_dir)):
187 | os.mkdir(dst_dir)
188 | urllib.urlretrieve(img_src, dst_file)
189 |
190 | nextslide_link.click()
191 |
192 | fdown.closed
193 | fdet.closed
194 | except:
195 | print "Failed download for : " + painter_list[ii]
196 | traceback.print_exc(file=sys.stderr)
197 |
198 | browser.close()
199 |
200 |
--------------------------------------------------------------------------------
/extra/experimental/crawler/crawl_wikipaintings.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/
2 |
3 | import os
4 | from selenium import webdriver
5 | import time
6 | import urllib
7 | import sys
8 | import traceback
9 | from selenium.common.exceptions import ElementNotVisibleException
10 |
11 |
12 | def get_next_page_link(browser):
13 | pager_elements = browser.find_element_by_class_name("pager-items")
14 |
15 | pagelinks = pager_elements.find_elements_by_tag_name("a")
16 |
17 | for pagelink in pagelinks:
18 | if pagelink.text == "Next":
19 | return pagelink
20 | return None
21 |
22 | def get_painters(browser):
23 | list_container = browser.find_element_by_id("listContainer")
24 | all_divs = list_container.find_elements_by_tag_name("div");
25 | painter_names = [];
26 | painter_urls = [];
27 | first_painting = [];
28 | print browser.current_url
29 | for painterdiv in all_divs:
30 | div_id = painterdiv.get_attribute("id");
31 | if div_id.startswith("a-") and not div_id.endswith("-slider"):
32 | print painterdiv.get_attribute("id")
33 | h2_elem = painterdiv.find_element_by_class_name("mr20")
34 | painter_name = h2_elem.text
35 | painter_names.append(painter_name)
36 | painter_url = h2_elem.find_element_by_tag_name("a").get_attribute("href");
37 | painter_urls.append(painter_url)
38 |
39 | return {'painter_names' : painter_names, 'painter_links' : painter_urls }
40 |
41 |
42 | def parse_args():
43 | if len(sys.argv) == 1:
44 | return
45 | try:
46 | for arg in sys.argv:
47 | if (arg.startswith('-')):
48 | tmp = arg[1:].lower()
49 | parts = tmp.split('=')
50 | if (parts[0] == 'datadir'):
51 | _datadir = parts[1]
52 | elif (parts[0] == 'style'):
53 | _painting_style = parts[1]
54 | return {'datadir' : _datadir, 'style': _painting_style}
55 | except:
56 | print "Usage: -datadir=