10 |
11 | In order to display the results of our work with OpenCV, we call opencv.getOutput(). This function returns a PImage with the current state of the image we're processing with OpenCV. Processing's image() function can display this result just like any other PImage.
12 |
13 | Take note: OpenCV for Processing automatically converted the image to grayscale. It does this by default because a lot of OpenCV operations only work with grayscale images.
--------------------------------------------------------------------------------
/book/filters/blur.md:
--------------------------------------------------------------------------------
1 | # Blur
2 |
3 | 
4 |
5 | Blurring is a surprisingly useful operation in computer vision. It might seem like blurring an image would obscure exactly the information that you're seeking to extract. To the contrary, blurring is similar to taking an average: it combines the value of each pixel with its neighbors. This is really useful for eliminating small variations or noise that might hurt the results of other operations like edge detection, contour finding, etc.
6 |
7 | ## Blur Strength and Direction - Convolutions and Kernels
8 |
9 | OpenCV's blur() function uses a type of blur known as a "box blur". In order to understand its effects -- and the various options available when applying a blur -- you'll need to know a little bit about how OpenCV applies blurs and other image filters.
10 |
11 | OpenCV blurs an image by applying what's called [a Kernel](http://en.wikipedia.org/wiki/Kernel_(image_processing)). A Kernel tells you how to change the value of any given pixel by combining it with different amounts of the neighboring pixels. The kernel is applied to every pixel in the image one-by-one to produce the final image (this operation known as a [convolution](http://en.wikipedia.org/wiki/Convolution)).
12 |
13 | Kernels are frequently represented with diagrams like this:
14 |
15 | 
16 |
17 | The middle box represents the current pixel. The values in each box indicate what portion of each pixel should be used to produce the output. The case shown here is the most boring possible kernel. It uses all of the current pixel and none of the neighbors. Applying this kernel would leave the image unchanged.
18 |
19 | Here's the most basic kernel for applying a box blur:
20 |
21 | 
22 |
23 | This kernel makes the pixel into an average of itself and each of the eight neighboring pixels. In other words, each of the nine pixels in the kernel contribute 1/9th of the final value.
24 |
25 | We describe this as a kernel size of one since it includes one pixel in each direction away from the current pixel. If we wanted to blur the image more, we could expand the size of our kernel, adding in more adjacent pixels.
26 |
27 | Here's a kernel for a 2-pixel box blur:
28 |
29 | 
30 |
31 | Expanding to a 2-pixel radius results in a kernel that covers 25 pixels. Hence, each pixel contributes 1/25th of our final value.
32 |
33 | The GIF at the top of this page showed a blur with a 10-pixel kernel. Here's what a blur with a 50-pixel radius kernel looks like:
34 |
35 | 
36 |
37 | In addition to kernels that mix all the pixels equally, we can use other kernels that have different effects. For example, we could construct a kernel that only mixed-in the pixels on the same row as the current pixel like so:
38 |
39 | 
40 |
41 | The result would be a horizontal blur like this:
42 |
43 | 
44 |
45 | And likewise, we could do the same thing in the vertical direction with this kernel:
46 |
47 | 
48 |
49 | producing something like this:
50 |
51 | 
52 |
53 | ### Color Images
54 |
55 | It's also worth noting that, unlike some other filters, blur() can be applied to color images:
56 |
57 | 
58 |
59 | ## Parameters
60 |
61 | OpenCV for Processing's blur() function has two different forms. Passing a single argument applies a simple box blur to the image with a kernel size determined by the argument you pass in.
62 |
63 | opencv.blur(10);
64 |
65 | To blur an image vertically or horizontally, you need to create a non-square kernel. The second form of blur() takes two arguments: one for the size of the kernel in the x- and y-dimensions.
66 |
67 | opencv.blur(5, 1);
68 |
69 | would apply a horizontal blur with the kernel we showed above: five pixels wide, all on the same row.
70 |
71 | opencv.blur(1,5);
72 |
73 | would apply a vertical blur: one pixel wide and five tall, just like we saw in the vertical kernel.
74 |
75 | Neither argument to this function can be zero, or an error will be raised. How can you construct a kernel that takes up zero rows or zero columns?
76 |
77 | ## Related Functions
78 |
79 | * [dilate()](https://github.com/atduskgreg/opencv-processing-book/blob/master/book/filters/dilate_erode.md)
80 | * [erode()](https://github.com/atduskgreg/opencv-processing-book/blob/master/book/filters/dilate_erode.md)
81 |
--------------------------------------------------------------------------------
/book/filters/brightness_contrast.md:
--------------------------------------------------------------------------------
1 | # Brightness and Contrast
--------------------------------------------------------------------------------
/book/filters/dilate_erode.md:
--------------------------------------------------------------------------------
1 | # Dilate and Erode
2 |
3 | 
4 |
5 | Dilation and erosion are both examples of "morphological" filters, functions that affect the shape of areas of pixels in the image. This can be a hard idea to get your head around on first exposure. How can an image filter change the shape of something in the image?
6 |
7 | The key to understanding it is to keep in mind that these filters don't change the shape of the objects pictured in the image; they affect areas in the image where the pixels are similar. Each morphological filter changes the shape of these regions in different ways.
8 |
9 | Dilation (shown above on the left) expands the white areas of the image. Examine the edge of the man's face and the stripes on his pajamas. When the filter is on, there's more white in the areas. The edge of his face spreads and smooths and the stripes on the pajamas get fat.
10 |
11 | (_NOTE: All of these illustrations are applying erosion and dilation to an image that has already been thresholded both because this makes their effects more visible and because those filters are particularly useful in combination. You can run erosion and dilation on non-binary grayscale images and even on color images. See the [threshold page](https://github.com/atduskgreg/opencv-processing-book/blob/master/book/filters/threshold.md) for more on that filter._)
12 |
13 | Erosion does the opposite. It contracts or thins these white areas. Compare the stripes in the image on the right to those on the left. When the erode filter is on, the stripes get thinner, more black is visible between them. And look at the man's left eye. The small spot of white visible within it disappears and the black shadow expands.
14 |
15 | Now, how is this useful? Dilation and erosion are frequently used together in order to close holes in an image. Take a look at this example of them both being applied to a scan in sequence:
16 |
17 | 
18 |
19 | On first glance, this appears to have done a lot less than each filter in isolation. But look closely at those stripes on the pajamas. When the filters are applied, the black holes in them get filled up, like they did under dilate, but the stripes don't spread out. In other areas of the image, a bunch of small isolated bits of black and white disappeared. The image is less noisy and the large areas of black or white are smoother and more continuous.
--------------------------------------------------------------------------------
/book/filters/edge_detection.md:
--------------------------------------------------------------------------------
1 | # Edge Detection
2 |
3 | OpenCV includes a number of filters meant to reveal the edges of objects including the Canny filter, the Scharr filter, and the Sobel filter. Each of these has different properties and is useful in different situations.
4 |
5 | ## Canny Edge Detection
6 |
7 | Unlike the other two edge detection filters, the Canny filter returns a binary image: all-white pixels where it found an edge, and all-black pixels where it did not find an edge. Also, uniquely, the Canny filter has no directional options
8 |
9 | ## Sobel Edge Detection
10 |
11 | 
12 |
13 | Notice the differences between the two modes, how each one affects the sides of the film frame and sprocket holes, the vertical stripes in the actor's shirt, and the lines in the soundtrack on the left of the frame.
14 |
15 | ### Parameters
16 |
17 | The parameters for findSobelEdges() can be a little bit counterintuitive.
18 |
19 | The Sobel edge-detection algorithm works by amplifying changes in the values of neighboring pixels. It can look at horizontal neighbors, vertical neighbors, or both. Counterintuitively, looking for changes between horizontal neighbors ends up detecting vertical edges and vice versa. (Imagine an image with a strong vertical edge in it. The pixels that make up that edge differ strongly not from the ones immediately above and below them (their vertical neighbors) but from those to their left and right (their horizontal neighbors).)
20 |
21 | The findSobelEdges() function takes two arguments, each an integer in the range 0-2. The first argument tells the Sobel algorithm how many horizontal neighbors to look at and the second how many vertical neighbors. The fewer neighbors you look at, the thicker and more intense the edges will be.
22 |
23 | 
24 |
25 | So, putting this into action, we can find horizontal edges like this:
26 |
27 | opencv.findSobelEdges(0,1);
28 |
29 | vertical edges:
30 |
31 | opencv.findSobelEdges(1,0);
32 |
33 | thinner vertical edges:
34 |
35 | opencv.findSobelEdges(2,0);
36 |
37 | and edges in both directions equally:
38 |
39 | opencv.findSobelEdges(1,1);
40 |
41 | ## Scharr Edge Detection
42 |
--------------------------------------------------------------------------------
/book/filters/equalize_histogram.md:
--------------------------------------------------------------------------------
1 | # Histogram Equalization
2 |
3 | You've probably seen histograms used to represent information about images in photo editing applications like Photoshop. Histograms show the amount of pixels at each different level of brightness in a complete image or in a particular color channel.
4 |
5 | The following illustration shows four histograms for a picture of me: one for brightness (shown in gray) and one each for the red, green, and blue channels.
6 |
7 | 
8 |
9 | Taller regions of the histogram mean that there are more pixels in the image at that level of brightness. Looking at the histograms for this image we can see that it is pretty dark; there are no lines on the rightmost side of any of the histograms where the brightest pixels are represented. You can also see that the red histogram is taller than the green and blue ones. This reflects how much of the image is taken up with my pink-ish face and the red and purple hues of the print on the wall behind me.
10 |
11 | Manipulating this distribution of pixels is a powerful way to change to brightness and contrast of an image. Specifically, OpenCV provides us with a function to "equalize" the histogram. Equalizing a histogram takes its values and spreads them out over the full possible range. Here's the result when we run it on our scanned film frame:
12 |
13 | 
14 |
15 | Kind of disappointing, huh? The image is certainly brighter. And, if we look closely, we can see that certain areas have significantly more detail visible (especially the lower partial frame).
16 |
17 | However, if we look at the actual histogram for this image, we can see the effect of equalizing it:
18 |
19 | 
20 |
21 | The values of the graph are shifted significantly towards the right, or brighter, side. The equalization had relatively little visible effect because this image was pretty evenly exposed to begin with. Between the dark black of the frame separators, the bright white of the sprocket holes, and the smooth grays of the frame itself, the original scan covered the full range of the histogram.
22 |
23 | Let's look at what histogram equalization does to that badly underexposed image of me from before.
24 |
25 | 
26 |
27 | Now, here you can see real results. The image is much brighter and much higher contrast. If we select the right input, histogram equalization can make things much more visible, especially in woefully under or overexposed images.
--------------------------------------------------------------------------------
/book/filters/filters-part.html:
--------------------------------------------------------------------------------
1 | Filters are the fundamental building block of any computer vision system.
5 |
4 |
5 | OpenCV's inRange() function is very similar to [threshold()](https://github.com/atduskgreg/opencv-processing-book/blob/master/book/filters/threshold.md). It also takes a grayscale image and converts it into a "binary" image: one where every pixel is either all-white or all-black based on whether or not the pixel is within the given range of of values.
6 |
7 | Where threshold() selects for pixels that are above a given value, inRange() lets you specify a maximum as well as a minimum value.
8 |
9 | To see the difference, compare the same image filtered with inRange(35, 50) and with a simple threshold(35):
10 |
11 | 
12 |
13 | Where threshold(35) catches the bright near-white of the wall as well as the light gray of my face, inRange(35,50), leaves the wall as black, i.e. un-selected, along with the highlight on the upper-right side of my face.
14 |
15 | ## Use with Color Images and the Hue Channel
16 |
17 | 
18 |
19 | Like most of the other OpenCV filter functions, inRange() can be used on any single-channel image, not just grayscale versions of color images. In particular, inRange() is especially useful when applied to the Hue channel. For more about this technique see [Color Tracking in HSV Color Space](https://github.com/atduskgreg/opencv-processing-book/blob/master/book/tracking/hsv_color.md).
20 |
21 | ## Parameters
22 |
23 | inRange(int min, int max) takes two arguments: the first represents the lower bound of the range, the second represents the upper bound. It will result in a binary OpenCV output image with white where the pixels were in-range and black where they were out-of-range.
24 |
25 | ## Related Material
26 |
27 | * [threshold()](https://github.com/atduskgreg/opencv-processing-book/blob/master/book/filters/threshold.md)
28 | * [adaptiveThreshold()](https://github.com/atduskgreg/opencv-processing-book/blob/master/book/filters/adaptive_threshold.md)
29 | * useColor() - sets the color mode. Options are HSB for Hue/Saturation/Brightness and RGB for Red/Green/Blue.
30 | * [Color Tracking in HSV Color Space](https://github.com/atduskgreg/opencv-processing-book/blob/master/book/tracking/hsv_color.md)
--------------------------------------------------------------------------------
/book/filters/region_of_interest.md:
--------------------------------------------------------------------------------
1 | # Region of Interest
2 |
3 | 
4 |
5 | When filtering images, sometimes we don't want to apply our filters to the entire image. Sometimes we'd rather select a particular area of the image just filter it. Maybe we only care about a feature if it appears in a particular spot. Or maybe we're running filtering operations that will have different results depending on their input (like we just saw above with histogram equalization).
6 |
7 | OpenCV lets us select a Region of Interest (ROI) in order to do just that. After we set our ROI, subsequent filters will only apply to that area. In the GIF above you can see our threshold filter applied just to the area around the actors face.
8 |
9 | ## Combined with Histogram Equalization
10 |
11 | 
12 |
13 | As we saw, the effects of histogram equalization depend on the range of grays in the image being processed. Our original frame scans have a wide range of gray values and therefore the filter had little effect. But, if we select the right Region of Interest, histogram equalization can really help us bring out the sprocket holes.
14 |
15 | In the GIF above, I've selected a small square on the right as an ROI. This area captures the edge of the frame and one of the sprocket holes. Unlike our earlier attempt on the whole image, running histogram equalization on just this area has a dramatic effect. It significantly darkens the edges of the sprocket hole making it much easier to detect.
16 |
17 | Using an ROI like this, makes our image filters work a little bit more like our eyes, which are constantly refocusing and re-adjusting to different objects and light conditions in front of us.
18 |
19 | ## Parameters
20 |
21 | A Region of Interest is a rectangular area of the image we select to exclusively receive OpenCV operations. Hence, the arguments to setROI() mirror those to rect(), the Processing function for drawing rectangles: x, y, width, height.
22 |
23 | opencv.setROI(400,250,150,150);
24 | opencv.threshold(50);
25 |
26 | _Note: there's also a corresponding opencv.releaseROI() function that ends the use of the Region of Interest. After calling it, further OpenCV functions will affect the entire image._
--------------------------------------------------------------------------------
/book/filters/threshold.md:
--------------------------------------------------------------------------------
1 | # Threshold
2 |
3 | 
4 |
5 | Thresholding is maybe the single most common image filter used with OpenCV. It takes a grayscale image and converts it into a "binary" image: one where every pixel is either all-white or all-black based on whether it's lighter or darker than the given threshold.
6 |
7 | Many super-useful OpenCV functions require a binary image, like [contour finding](https://github.com/atduskgreg/opencv-processing-book/blob/master/book/tracking/contours_and_lines.md).
8 |
9 | In the animated GIF above, we're applying a threshold at 50 to a scanned frame of film from [the Kinograph project](https://github.com/atduskgreg/opencv-processing-book/blob/master/book/projects/kinograph_1.md). When we threshold our scan, the subtle grays that make up the area with the sprocket holes on the right of the frame disappear completely into white. And the black bars of the frame separators become even more prominent.
10 |
11 | Thresholding is not limited to grayscale images. You can apply it to color images and individual color channels as well (see the code samples below).
12 |
13 | ## Related Functions
14 |
15 | * [adaptiveThreshold()](https://github.com/atduskgreg/opencv-processing-book/blob/master/book/filters/adaptive_threshold.md)
16 | * [inRange()](https://github.com/atduskgreg/opencv-processing-book/blob/master/book/filters/in_range.md)
17 |
18 | ## Parameters
19 |
20 | The threshold() function takes a single argument: an integer value from 0-255:
21 |
22 | opencv.threshold(50);
23 |
24 | This value represents a value of gray (0 being black and 255 white). After threshold() is applied, pixels below this value will be black and those above it will be white.
25 |
26 | ## Code
27 |
--------------------------------------------------------------------------------
/book/projects/kinograph_1.md:
--------------------------------------------------------------------------------
1 | # Test Case 1: Film Scanner Frame Extractor
2 |
3 | ## Part 1: Understanding the Problem
4 |
5 | ### Kinograph: DIY Film Scanner
6 |
7 | [Kinograph](http://mepler.com/Kinograph) is an open source DIY film scanner currently under development by Matt Epler, an archivist and hacker. Film scanners digitize analogue movies by photographing each frame of film one at a time. Existing models cost upwards of $100,000. Much of the cost arises from the need for precision hardware to place each frame of film in exactly the same location every time.
8 |
9 | Using computer vision, we can reduce the need for this precision. Instead of having to get each film frame into exactly the same spot, we can write a program that finds each frame within an image and extracts it.
10 |
11 | Kinograph's software uses exactly these techniques. The result is that Kinograph's hardware costs less than $1,000 and will be available to many more archivists enabling the preservation of many films that would otherwise be lost.
12 |
13 | In this test case, we'll approach the problem of detecting and extracting frames from images of a film captured with Kinograph. We'll see how OpenCV's image filtering, edge detection, contour detection, and polygon approximation functions can be used to align and extract individual frames from the scanned film.
14 |
15 | ### The Problem: Rotation and Overscan
16 |
17 | 
18 |
19 | Raw Kinograph scans (as seen above) present two main challenges for frame extraction: rotation and overscan.
20 |
21 | As the film moves through the machine it does not always stay perfectly parallel to the camera that captures the scans. In order to correct this problem, we'll have to figure out the orientation of the frame of film within each scan so we can rotate the image to make the frame parallel with its edges.
22 |
23 | Each Kinograph scan also captures more than just the current frame. As you can see in the image above, half of the previous and next frames are also visible. This excess area is called "overscan" in the digitization biz. It's necessary for other processes like stitching the audio track together, but it's exactly what we have to eliminate to produce a video file that looks like the original movie. In order to extract each frame, we'll have to find its borders within the larger scan.
24 |
25 | ### How to Think about the Problem
26 |
27 | So, how can we go about detecting the rotation and frame boundary of each scanned frame? Let's look critically at our example scan above. What parts of this image might help us extract the information we need?
28 |
29 | Well, first of all, we know that we'll be processing multiple frames with different content within each frame. This is a moving picture, after all. So, we need to look for parts of the image that will stay consistent even as the content of the frame changes.
30 |
31 | Secondly, we need parts of the image that stick out sharply from their surroundings. Unlike the human vision system, computer vision techniques are highly sensitive to subtle variations in the image. Uneven exposure, image noise, subtle gradations of brightness -- they can all make image features that seem obvious to the eye dissolve into a mixed-up jumble of pixels.
32 |
33 | And, finally, we need to pick parts of the image that will actually help us find the orientation and location of the image. That means straight lines and other geometric forms that correspond to the edges of the frame and the direction of the strip of film within the scan.
34 |
35 | #### Quiz
36 |
37 | Click on two parts of the image that seem promising for detecting the orientation and location of the frame. Remember, you're looking for parts of the image that are:
38 |
39 | * Consistent even as the content of the frame changes
40 | * Stark enough to be resistant to variations in image quality
41 | * Useful in detecting the orientation and location of the film frame.
42 |
43 | _ANSWER: frame separators, sprocket holes, frame edges_
44 |
45 | #### Answer
46 |
47 | The three parts of the image that best fit our criteria are:
48 |
49 | * The horizontal frame separators
50 | * The frame's vertical edges
51 | * The sprocket holes
52 |
53 | _CLOSE-UP IMAGE OF FRAME SEPARATOR_
54 |
55 | As big black horizontal bars, the frame separators are maybe the starkest portion of the image. Their edges make-up straight horizontal lines, so finding them would tell us the orientation of the frame with the scan. And they border the frame itself on top and bottom, giving us half of what we need to find the frame's location. However, the frame separators have a downside too: what happens when the frame goes completely black? Fade outs, credit sequences, and title cards are all common situations in which black areas of the frame would bleed into the frame borders, screwing up our detection.
56 |
57 | _CLOSE-UP IMAGE OF FRAME EDGE_
58 |
59 | The frame's vertical edges aren't as stark as the separators, but they do contrast nicely with the neighboring translucent area of the film. As straight lines, they also provide solid information about the orientation of the scan. And, as edges of the frame, they give us its left and right bounds. In fact, in combination with the vertical information from the frame separators (or any other source), the frame separators would give us enough to find the complete frame position.
60 |
61 | _CLOSE-UP IMAGE OF SPROCKET HOLE_
62 |
63 | The last part of the image that seems promising is the sprocket holes. They are geometric in shape and certain of them correspond to the top and bottom of the frame. Compared to the frame separators and edges, they're much lower contrast. On the other hand, they're located on a part of the film that won't be affected by changes within the frame. Also, the sprocket holes were designed specifically to register each frame in the projector so we can rely on them being positioned correctly. One major downside of the sprocket holes is that there are more of them than we need. In order to use them, we'd need a way to distinguish the holes at the top and bottom of the frame from all the others that appear in each scan.
--------------------------------------------------------------------------------
/book/projects/kinograph_2.md:
--------------------------------------------------------------------------------
1 | # Test Case 1: Film Scanner Frame Extractor
2 |
3 | ## Part 2: Filtering the Image
4 |
5 | ### Why Filter?
6 |
7 | Now that we know what parts of the image we'd like to work with, how do we go about detecting them?
8 |
9 | In the end, what we're looking for is geometric data: we want the position, orientation, and scale of the film frame so we can extract it from the scan. This will take the form of angles, x-y coordinates, and scaling factors.
10 |
11 | There are lots of techniques for extracting data like these from images -- both simple ones that we can implement ourselves (like checking the brightness of particular pixels) and more complex ones provided by OpenCV (eg. contour finding).
12 |
13 | But before we're ready to use any of those, we need to prepare our image so that those techniques will actually provide us information about the parts of the image we're curious about. To do that, we'll apply a series of filters to our source image meant to bring out the frame separators, frame edges, and sprocket holes.
14 |
15 | ### Image Filters to Find Frame Features
16 |
17 | OpenCV provides a large selection of image filtering functions. In this section, I'll introduce you to the ones that'll be most useful for working with the Kinograph scans. We won't be writing code quite yet -- I just want you to get a feel for each of these filtering functions, how they effect our scans, and how they might be useful.
18 |
19 | These image filters are some of the most commonly used tools in the OpenCV toolbelt. The goal is for you to develop an intuition about when to reach for which tool. We'll be using [thresholding](https://github.com/atduskgreg/opencv-processing-book/blob/master/book/filters/threshold.md), [Sobel edge detection](https://github.com/atduskgreg/opencv-processing-book/blob/master/book/filters/edge_detection.md), [dilation and erosion](https://github.com/atduskgreg/opencv-processing-book/blob/master/book/filters/dilate_erode.md), [histogram equalization](https://github.com/atduskgreg/opencv-processing-book/blob/master/book/filters/equalize_histogram.md), and [region of interest](https://github.com/atduskgreg/opencv-processing-book/blob/master/book/filters/region_of_interest.md). If you haven't used these filters before, review those pages about them before proceeding.
--------------------------------------------------------------------------------
/book/projects/kinograph_3.md:
--------------------------------------------------------------------------------
1 | # Test Case 1: Film Scanner Frame Extractor
2 |
3 | ## Part 3: Applying Filters
4 |
5 | We've been talking for awhile here without looking at any code. In this section, we're going to remedy that. We'll look at the basics of using OpenCV in Processing. We'll see how to load up an image, apply filters to it, and see the output.
6 |
7 | Once you've got those basics down, we'll take another look at the filters introduced in the last section. This time, I'll show you the parameters for each one and explain what they do.
8 |
9 | While you're reading keep this question in mind: how could use these filters (alone or in combination) to accentuate the frame borders, sprocket holes, and frame edges in our scan? There'll be a quiz at the end that asks you to do exactly that.
10 |
11 | ### Quiz: Filter for Frame Features
12 |
13 | As we saw in the last section, the purpose of all of these filters is to bring out the parts of an image from which we can derive useful data. In the case of Kinograph, those parts are sprocket holes, the frame borders, and the frame edges.
14 |
15 | In this quiz, your job is to apply what you just learned about these filter functions and their parameters to a Kinograph scan. You can use them individually or in combination. After you've applied your filters, we'll evaluate the results to see how much you've revealed each image feature. Keep trying until you get all three (either with a single combination of filters or in multiple tries).
16 |
17 | _Interactive THINGY!_
--------------------------------------------------------------------------------
/book/projects/kinograph_4.md:
--------------------------------------------------------------------------------
1 | # Test Case 1: Film Scanner Frame Extractor
2 |
3 | ## Part 4: Finding Contours
--------------------------------------------------------------------------------
/book/projects/projects-part.html:
--------------------------------------------------------------------------------
1 | In this section we explore a number of projects that apply computer vision concepts to real applications.
5 |opencv.startBackgroundSubtraction() - Setup the background subtraction process.
48 | * opencv.updateBackground() - Update the background based on the current image.
49 | * opencv.dilate() - Thicken the shapes in the current image.
50 | * opencv.erode() - Thin the shapes in the current image. When used in combination with opencv.dilate() this will close holes.
51 | * opencv.findContours() - Find contours based on the current gray image.
52 |
53 | ### Browse the Code
--------------------------------------------------------------------------------
/book/tracking/brightest_point.md:
--------------------------------------------------------------------------------
1 | # Track the Brightest Point
2 |
3 | _Probably the single simplest way to track an object with OpenCV is based on brightness. Brightness tracking is highly sensitive to lighting conditions and other changes in the scene, but it can work effectively in situations where you have a lot of control over the environment. In this section, we'll use OpenCV's max() function to track the brightest point in an image. We'll also see how a variation of brightness tracking, applied to the individual color channels of an image, can be used for a crude form of color tracking._
4 |
5 |
6 |
7 | ## Video Summary
8 |
9 | * Brightness tracking is one of the simplest ways to track an object.
10 | * It's very fragile (affected by changing lighting conditions) so if you want to use it you'll need to control the environment
11 | * By default, our OpenCV object works off of a grayscale version of the input image.
12 | * Grayscale pixel values range from 0 (for black) to 255 (for white).
13 | * The pixel with the maximum value in this image will be the brightest point. (See the note about Luma as a measure of brightness below.)
14 | * If multiple pixels share the same maximum value (usually all white or all black if the image itself is all black), then the top-leftmost pixel will be selected.
15 | * We can make this dynamic by processing each new frame of a Capture object.
16 | * This technique works for any grayscale image.
17 | * OpenCV gives us the red, blue, and green channels of our image as individual grayscale images.
18 | * Finding the max of these individual channels will find the reddest, greenest, and bluest channel respectively.
19 | * This form of color tracking is very crude because of the limitations of RGB color space
20 | * See the Color Tracking in HSV Color Space for a more robust technique for tracking color.
21 |
22 | ### Luma: A More Accurate Measure of Brightness
23 |
24 | [Luma](http://en.wikipedia.org/wiki/Luma_%28video%29) is a representation of the brightness of a pixel that matches human perception more accurately than its gray value, which is a simple average of the reg, green, and blue components. Luma can be calculated as an unequal combination of the red, green and blue components of the pixel according to the following formula: 0.2126\*R + 0.7152\*G + 0.0722\*B. To use Luma for brightness tracking with our OpenCV image, you could iterate through the image and calculate the Luma value for each pixel, which would be slow. Or, you could use OpenCV's [Imgproc.cvtColor()](http://docs.opencv.org/java/org/opencv/imgproc/Imgproc.html#cvtColor) function in concert with opencv.getBufferColor() to convert the image to the [LAB color space](https://en.wikipedia.org/wiki/Lab_color_space), which includes Luma as one of its three channels.
25 |
26 | This image demonstrates the comparison. The difference can be subtle. Click through to the larger size to make sure you see it. Look carefully at the right side of the image around the flashlight.
27 |
28 | 
29 |
30 | The code for this example (showing how to access the Luma channel) is here: [LumaVsGray.pde](https://github.com/atduskgreg/OpenCVPro/blob/library_rename/examples/LumaVsGray/LumaVsGray.pde).
31 |
32 | In practice, the grayscale average of RGB is usually used due to its convenience.
33 |
34 | ## Quiz
35 |
36 | Q: What qualities of our input image could cause problems with brightness tracking: A) The presence of many glowing objects. B) Moving shadows cast by passersby. C) The auto-exposure on our camera triggering. D) All of the above.
37 |
38 | Answer: D, all of the above.
39 |
40 | Q: Which are easier to track with this version of color tracking: bright red or dark red objects?
41 |
42 | Answer: Light red. Dark red objects will converge with the shadows in the scene where R, G, and B components are all near 0.
43 |
44 | Q: What are some techniques we could use to prevent the brightest point from jumping around so much when tracking an object with this method?
45 |
46 | Answer: 1) Lerp the x- and y-components of the brightest point between sequential frames. 2) Blur the image before finding the max to smooth over small differences between values. 3) Filter out large jumps in the position of the point, as they're probably due to glitches rather than the continuous motion of the tracked object.
47 |
48 | ## Code
49 |
50 | ### Important Functions
51 |
52 | * opencv.max() - Find the brightest point in the current gray image.
53 | * opencv.setBufferGray() - Set the current gray image to another channel.
54 | * opencv.getBufferG() - Get the green channel of the current image. Useful for passing to opencv.setBufferGray().
55 |
56 | ### Browse the Code
57 |
58 | 1. capture image and load it into opencv
59 | 2. find and draw the max point
60 | 3. Switch to Capture instead of a still image
61 | 4. Use the green channel to track a green object
62 |
63 |
--------------------------------------------------------------------------------
/book/tracking/contours_and_lines.md:
--------------------------------------------------------------------------------
1 | # Finding Contours and Lines
2 |
3 | ## Video Script
4 |
5 | * Figure out the shapes of things
6 | * Enables shape analysis like AR marker detection, shape analysis, image segmentation, perspective analysis, stylization like drawing (Kyle coffee bot)
7 | *
8 |
9 | ## Video Summary
10 |
11 | ## Quiz
12 |
13 | ## Code
14 |
15 | ### Important Functions
16 |
17 | * opencv.findContours()
18 | * contour.getPolygonApproximation()
19 | * contour.getPoints()
20 | * opencv.findLines()
21 | * line.angle
22 | * line.start
23 | * line.end
24 |
25 | ### Browse the Code
26 |
--------------------------------------------------------------------------------
/book/tracking/face_detection.md:
--------------------------------------------------------------------------------
1 | # Face Detection
2 |
3 | _Detecting faces in images is one of the most ubiquitous applications of computer vision. We see it in our digital cameras for autofocus, on our social networks to identify our friends, in public surveillance systems, and many other places. In this project, we'll learn how to detect faces using OpenCV. We'll see a little bit about how the face detection process works, how it can fail, and how we can apply it to detect other things besides just faces._
4 |
5 |
6 |
7 | _Video and images from Adam Harvey's [CV Dazzle](http://cvdazzle.com) used with permission. Clock photo by [jaqian](http://www.flickr.com/photos/jaqian/7292320/) and pedestrian photo by [gandalphcunningham](http://www.flickr.com/photos/gandalfcunningham/2527870434/)._
8 |
9 | ## Video Summary
10 |
11 | * Face detection in OpenCV is easy to use.
12 | * It is necessary to learn about how it is implemented in order to use it effectively and understand its limitations.
13 | * OpenCV's face detector uses the Viola-Jones algorithm, originally introduced in the 2001 paper [Rapid Object Detection Using a Boosted
14 | Cascade of Simple Features](http://www.merl.com/papers/docs/TR2004-043.pdf).
15 | * Viola-Jones works by matching a defined series of patterns of light and dark rectangles to different areas of the image at different scales.
16 | * Faces have a consistent arrangement of highlights and shadows based on how light falls on the facial features.
17 | * Viola-Jones can be trained to recognize these patterns.
18 | * The training process requires a large set of input images and takes a long time and a lot of computing resources.
19 | * The training process produces a "cascade" file (usually stored as XML), which can be loaded into OpenCV for realtime detection.
20 | * The most commonly used cascade is for frontal faces.
21 | * Artist Adam Harvey's [CV Dazzle](http://cvdazzle.com) project consists of makeup designed to camouflage the wearer from Viola-Jones face detection.
22 | * CV Dazzle works by introducing dark and light patches onto the face where the Viola-Jones frontal face cascade does not expect them to be.
23 | * Harvey produced [visualizations](https://vimeo.com/34545827) that illustrate how the Viola-Jones detection process works.
24 | * For more about CV Dazzle and how the Viola-Jones algorithm works, see [this Makematics interview with Adam Harvey](http://makematics.com/research/viola-jones/).
25 | * Viola-Jones can detect multiple faces.
26 | * Find more faces will slow down the processing of each image.
27 | * Viola-Jones can also be used to detect other objects than faces.
28 | * Researchers have created cascades for noses, ears, pedestrians, clock faces, and many other objects.
29 |
30 | ## The Politics of Face Detection
31 |
32 | Face detection brings out strong, sometimes contradictory, reactions in people. It triggers our fear of being observed, of surveillance by governments, corporations, and others figures of authority. In response to these fears, there's something of a tradition of creative projects that produce inventive ways of avoiding face detection.
33 |
34 | Beyond Adam Harvey's CV Dazzle, demonstrated above, artists [Kyle McDonald](http://kylemcdonald.net) and [Aram Bartholl](http://datenform.de/) of the [F.A.T Lab collective](http://fffff.at/) created a video on "[How To Avoid Facial Recognition](http://fffff.at/how-to-avoid-facial-recognition/)"
35 |
36 |
37 |
38 | Very different in style and tone from CV Dazzle, the combination of humor and punk rock attitude in this video simultaneously expresses and mocks the idea of face detection as a form of technological oppression.
39 |
40 | At the end of the video McDonald and Bartholl demonstrate that most face detection systems fail if the face is tilted by more than 15 degrees or so. This is a practical concern when using OpenCV's face detection functions. It can be overcome by rotating the image before processing, but that technique is computationally expensive and rarely done.
41 |
42 | In addition to the fear of being observed by face detection systems, there's a different kind of fear in being invisible to them.
43 |
44 | In 2009, two employees of a computer store uploaded a video called [HP Computers are Racist](http://www.youtube.com/watch?v=t4DT3tQqgRM) in which they demonstrated an HP MediaSmart computer failing to track the face of a black man, but succeeding on a white woman.
45 |
46 |
47 |
48 | [HP responded](http://www.pcmag.com/article2/0,2817,2357429,00.asp) to the video by explaining the technical roots of the problem and insisting it wasn't intentional:
49 |
50 | >"The technology we use is built on standard algorithms that measure the difference in intensity of contrast between the eyes and the upper cheek and nose. We believe that the camera might have difficulty 'seeing' contrast in conditions where there is insufficient foreground lighting."
51 |
52 | ## Quiz
53 |
54 | Q: Which of the following conditions will not make OpenCV face detection run faster: A) A smaller input image. B) Fewer faces present in the image. C) Good lighting.
55 | Reveal the Answer
57 | 58 | Q: Is it easier to detect objects with a cascade or to train a new cascade? 59 |Reveal the Answer
61 | 62 | Q: What is the minimum angle of orientation that will cause the frontal face cascade to fail to detect the face? 63 |Reveal the Answer
65 | 66 | ## Code 67 | 68 | ### Important Functions 69 | 70 | *opencv.loadCascade() - Setup for face tracking (or other cascade-based tracking).
71 | * OpenCV.CASCADE_FRONTALFACE_ALT - A constant referring to the standard cascade for detecting faces. Pass this to opencv.loadCascade() to setup for face detection.
72 | * opencv.detect() - Detect objects in the current image based on the configured cascade. Returns an array of Rectangle objects.
73 |
74 | ### Browse the Code
75 |
76 | * Detect faces in video
77 | * Scaling trick to do it faster
78 | * Detect clocks with a different cascade
79 |
--------------------------------------------------------------------------------
/book/tracking/hsv_color.md:
--------------------------------------------------------------------------------
1 | # Color Tracking in HSV Color Space
2 |
3 | _One of the simplest ways to track an object with OpenCV is based on its color. However, the color space we use to represent the colors in an image can have a big effect on how easy it is to implement color tracking. In this section, we'll prepare an image for color tracking by converting it to HSV color space and filtering it based on a range of Hues. After filtering, we'll be able to easily track our object's position by finding contours._
4 |
5 |
6 |
7 | ## Video Summary
8 |
9 | * Color tracking is useful when you can control the color of the tracked-object or it already sticks out from the background.
10 | * A "color space" describes how we represent a single color as a mix of different elements.
11 | * The most common color space is RGB.
12 | * RGB is the default color space in Processing.
13 | * RGB is flawed for color tracking because changes in color that look small to the eye effect all three channels significantly.
14 | * HSV is a more useful color space for tracking.
15 | * HSV stands for Hue, Saturation, and Value.
16 | * Hue is color. Saturation is intensity. Value is brightness.
17 | * For color tracking we can just use the Hue channel.
18 | * A Histogram is a graph that shows frequency of each color in an image.
19 | * OpenCV provides that lets us filter an image based on a range of values.
20 | * Filtering a colorful image by a range of hues reveals which hue corresponds to which color.
21 | * Hue 0 is red and then it ranges up through orange, yellow, green, white (around 90), and up to blue.
22 | * In OpenCV, hue ranges from 0-180.
23 | * To track an object in live video we apply a range filter to the hue, selecting the color range of our object.
24 | * Then we find contours in that filtered image.
25 | * The contour with the largest area will be the object we want to track
26 | * We can draw the bounding box of that contour.
27 | * The center of the contour's bounding box will be the object's location.
28 |
29 | ## Quiz
30 |
31 | Q: What is the default color space in Processing?
32 |
33 | Answer: RGB
34 |
35 | Q: Are there colors that can be represented in HSV color space that are impossible to represent in RGB color space?
36 |
37 | Answer: No
38 |
39 | Q: Using OpenCV's scale, what color is represented by the following HSV values: 215, 50, 30?
40 |
41 | Answer: None. Hue ranges from 0-180.
42 |
43 | ## Code
44 |
45 | The code sample below will walk you step-by-step through implementing Hue-based color tracking.
46 |
47 | ### Important Functions
48 |
49 | * opencv.useColor() - Tell OpenCV which color space to use.
50 | * opencv.getBufferH() - Access the Hue channel of the image.
51 | * opencv.setBufferGray() - Set the gray channel of the image to the values of another channel.
52 | * opencv.inRange() - Filter the image based on a range of values.
53 | * opencv.findContours() - Find contours in the current image.
54 | * contour.getBoundingBox() - Get the rectangular bounding box around a particular contour.
55 |
56 | ### Browse the Code
57 |
58 |
59 |
60 |
--------------------------------------------------------------------------------
/book/tracking/tracking-part.html:
--------------------------------------------------------------------------------
1 | Tracking allows computer vision systems to follow objects from frame to frame.
5 |This project is a practical introduction to computer vision using OpenCV and Processing.
43 |44 | Read the early draft 45 |
32 | {{ title }}
33 |