├── .editorconfig
├── .gitignore
├── README.md
├── addons.make
├── bin
└── .gitignore
├── media
├── general.gif
├── sound-debug.gif
└── video-debug.gif
└── src
├── main.cpp
├── ofApp.cpp
└── ofApp.h
/.editorconfig:
--------------------------------------------------------------------------------
1 | root = true
2 |
3 | [*]
4 | indent_style = tab
5 | end_of_line = lf
6 | charset = utf-8
7 | trim_trailing_whitespace = true
8 | insert_final_newline = true
9 |
10 | [*.coffee]
11 | indent_style = space
12 |
13 | [{package.json,*.yml}]
14 | indent_style = space
15 | indent_size = 2
16 |
17 | [*.md]
18 | trim_trailing_whitespace = false
19 |
--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
1 | Makefile
2 | *.xcconfig
3 | *.make
4 | !addons.make
5 | *.plist
6 | *.xcodeproj/
7 | /obj/
8 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | 
2 |
3 | ---
4 |
5 | # Demo for Dot-Dot-Dash
6 |
7 | Inspired by the old Apple iPod silhouette (by Susan Alinsangan) campaign.
8 |
9 | 
10 | Apple iPod Silhouette Campaign
11 |
12 | There are a lot of ways this could be achieved, and this definitely isn't
13 | intended to look *exactly* like the campaign.
14 |
15 | ---
16 |
17 | This demo shows a silhouette of one or more persons in front of a camera.
18 | It is intended to be played with music, which is analyzed to detect "bumps"
19 | in the music - at which points the background changes to another color.
20 |
21 | The path algorithm uses contour detection via OpenCV, feeds the vertices into
22 | a simplification algorithm (which reduces the number of points while keeping
23 | the best shape possible), and then re-connecting them using curves before
24 | drawing them on the screen.
25 |
26 | The sound component uses Fast Fourier Transform (FFT) to deduce 27 bands of
27 | audio, and allows the configurator to specify one or more of the first 18 bands
28 | to be averaged and turned into a net peak. This peak is compared against a
29 | threshold, which is then processed by a debounce algorithm, in order to detect
30 | "bumps". These bumps trigger the background to change to another "friendly"
31 | color.
32 |
33 | The colors used are simply different hue shifts with a slightly desaturated
34 | base color.
35 |
36 | ---
37 |
38 | There are many ways this demo could have been achieved. My initial goal was to
39 | train a Haar detector to detect ears (as I couldn't find any existing cascades)
40 | and then use an adjacent upper body detector to add an overlay of the Apple
41 | earbuds and an iPod attached to the "belt" area - similar to the original
42 | campaign. Unfortunately, training a haar cascade is tedious and takes lots and
43 | lots of data - something I couldn't do in a single day, as originally planned.
44 |
45 | The other improvement to this demo would be to change how the analysis works for
46 | bump detection. Ideally, tempo detection would produce the most consistent
47 | results, coupled with peak detection would get song phrasing to look nice
48 | (instead of "bumping" even in quiet parts of the song).
49 |
50 | Ultimately I envisioned this to be used on a sidewalk-type area where
51 | pedestrians would be able to passively interact with it. This means, using
52 | four haar detectors (ear/upper body, frontal and profile), passerbys would
53 | see themselves with the earbuds - without having to do much more than just 'be
54 | there'.
55 |
56 | Lastly, in the event the earbuds would be possible (and a Haar cascade was
57 | trained) then the cable would simply be a Box2D (or if we had enough budget
58 | to figure out body sizes in 3D, bullet physics) enabled cord. People could dance
59 | and the cord would fly around as if it were really on them. Subtle but
60 | effective.
61 |
62 | ---
63 |
64 | # Building / Installing
65 | Clone into `/path/to/open-framework/apps` and then run the `projectGenerator` of
66 | your choice inside the cloned repository. Build, and run.
67 |
68 | This shouldn't be any different than other apps/examples.
69 |
70 | # Controls
71 | There are quite a few controls and two different debug views.
72 |
73 | ### General Controls
74 | Along with the keyboard shortcuts below, you may also drag/drop an audio file
75 | (music) directly onto the demo; it will start immediately, allowing you to
76 | play with the sound settings.
77 |
78 | - `f` - Toggle fullscreen
79 | - `c`/`v` - Decrease/Increase the "frame delay" effect (default off)
80 |
81 | #### Video Debug
82 | The video portion of this demo uses contour detection of video streamed from
83 | the default camera on the system.
84 |
85 | > Mac cameras have built in light compensation which is handled at the hardware
86 | > level, making any sort of OpenCV stuff incredibly annoying. If you're running
87 | > into this problem with the background diffing, then you'll probably need more
88 | > light, or need to step back further.
89 |
90 | The detected blobs are simplified, and then curved. Use the debugging controls
91 | to help fine-tune the settings (`d`).
92 |
93 | - `d` - Toggle the video debug view
94 | - `[space]` - Learn background (should be used with nothing in the shot)
95 | - `h` - Show [holes](http://openframeworks.cc/documentation/ofxOpenCv/ofxCvContourFinder.html#!show_findContours) (off by default)
96 | - `,`/`.` - Decrease/Increase the simplification factor on paths
97 | - `[`/`]` - Decrease/Increase the background subtraction factor
98 | - `n`/`m` - Decrease/Increase the minimum blob size (in pixels)
99 | - `N`/`M` - Decrease/increase the maximum blob size (in pixels)
100 |
101 | > The blob size settings don't make too much of a difference.
102 |
103 | 
104 |
105 | #### Sound Debug
106 | The audio portion of this demo allows the user to drag/drop a music file onto
107 | the application and perform FFT/peak average analysis on the playing music
108 | to change the color of the background.
109 |
110 | - `s` - Toggle the sound debug view
111 | - `z`/`x` - Decrease/Increase the "bump" threshold. Hold Shift to fine-tune.
112 | - `q`/`w` - Decrease/Increase the dither (interpolation) of the sampled peaks.
113 |
114 | > This setting will *drastically* improve the ability to accurately refine
115 | > the "bumps".
116 |
117 | - `-`/`=` (`+`) - Decrease/Increase the "cooldown count". This is the number of
118 | sample updates before allowing another bump to occur after the previous bump.
119 | For instance, if this is at `5`, then every bump will cause a forced timeout
120 | of five more samples before another bump can occur.
121 | - [`Shift`] `0`-`9` - Toggles a band to be included in the "bump" average.
122 | The first 18 bands can be toggled, the first nine with `0` - `9` and the
123 | second nine with `Shift + 0` - `Shift + 9`.
124 |
125 | > I recommend against using the very first band, as I don't think the underlying
126 | FFT algorithms perform a 20hz cut, causing that band to clip almost all of the
127 | time.
128 |
129 |
130 | 
131 |
--------------------------------------------------------------------------------
/addons.make:
--------------------------------------------------------------------------------
1 | ofxOpenCv
2 |
--------------------------------------------------------------------------------
/bin/.gitignore:
--------------------------------------------------------------------------------
1 | *.app/
2 |
--------------------------------------------------------------------------------
/media/general.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Qix-/dot-dot-dash-demo/4b9e2748b30f38ddebfaa429ce8974276ca6bcd4/media/general.gif
--------------------------------------------------------------------------------
/media/sound-debug.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Qix-/dot-dot-dash-demo/4b9e2748b30f38ddebfaa429ce8974276ca6bcd4/media/sound-debug.gif
--------------------------------------------------------------------------------
/media/video-debug.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Qix-/dot-dot-dash-demo/4b9e2748b30f38ddebfaa429ce8974276ca6bcd4/media/video-debug.gif
--------------------------------------------------------------------------------
/src/main.cpp:
--------------------------------------------------------------------------------
1 | #include "ofMain.h"
2 | #include "ofApp.h"
3 |
4 | int main() {
5 | ofSetLogLevel(OF_LOG_VERBOSE);
6 | ofSetupOpenGL(1024, 768, OF_WINDOW);
7 | ofRunApp(new ofApp());
8 | return 0;
9 | }
10 |
--------------------------------------------------------------------------------
/src/ofApp.cpp:
--------------------------------------------------------------------------------
1 | #include
2 | #include
3 | #include "ofApp.h"
4 |
5 | using namespace std;
6 |
7 | static const int IMG_SIZE_W = 800;
8 | static const int IMG_SIZE_H = 600;
9 | static const int THRESHOLD_INC = 1;
10 | static const int MAX_BLOBS = 4; // Keep it around (number of people * 2)
11 | static const float SIMPLIFICATION_INC = 0.1f;
12 | static const float BASS_INC = 0.05f;
13 | static const float BASS_INC_FINE = 0.01f;
14 | static const int BUMP_DEBUG_W = 500;
15 | static const int BUMP_DEBUG_H = 300;
16 | static const float BUMP_DITHER_INC = 0.01f;
17 |
18 | void ofApp::setup() {
19 | this->grabber.setVerbose(true);
20 | this->grabber.initGrabber(IMG_SIZE_W, IMG_SIZE_H);
21 |
22 | cout << "asked for grabber size of "
23 | << IMG_SIZE_W << "x" << IMG_SIZE_H
24 | << ", got " << this->grabber.getWidth() << "x"
25 | << this->grabber.getHeight()
26 | << endl;
27 |
28 | this->image.allocate(IMG_SIZE_W, IMG_SIZE_H);
29 | this->imageGray.allocate(IMG_SIZE_W, IMG_SIZE_H);
30 | this->imageBg.allocate(IMG_SIZE_W, IMG_SIZE_H);
31 | this->imageDiff.allocate(IMG_SIZE_W, IMG_SIZE_H);
32 |
33 | this->findMin = 20;
34 | this->findMax = (IMG_SIZE_W * IMG_SIZE_H) / 3;
35 |
36 | this->learn = true;
37 | this->threshold = 75;
38 | this->simplification = 1.8f;
39 |
40 | this->silhouettes.setFillColor(ofColor::fromHex(0));
41 | this->silhouettes.setFilled(true);
42 | this->holes.setFilled(true);
43 |
44 | this->debug = false;
45 | this->debugSound = false;
46 | this->showCursor = true;
47 |
48 | this->player.setLoop(true);
49 | this->bumpThreshold = 0.4;
50 | memset(&this->bands[0], 0, sizeof(this->bands));
51 | this->bump = 0.0f;
52 | this->dither = 0.14f;
53 |
54 | this->cooldownCount = 5;
55 | this->cooldown = 0;
56 |
57 | this->bg.setHsb(0, 161, 255);
58 | this->onBump(); // Initialize the background color.
59 |
60 | this->showHoles = false;
61 |
62 | this->frameDelay = 0;
63 | this->frame = 0;
64 | }
65 |
66 | void ofApp::update() {
67 | ofBackground(this->bg);
68 |
69 | this->grabber.update();
70 | bool newFrame = this->grabber.isFrameNew();
71 |
72 | if (newFrame && this->frame-- == 0) {
73 | this->frame = this->frameDelay;
74 | this->image.setFromPixels(this->grabber.getPixels());
75 | this->image.mirror(false, true);
76 | this->imageGray = this->image;
77 |
78 | if (this->learn) {
79 | cout << "re-learning background" << endl;
80 | this->learn = false;
81 | this->imageBg = this->imageGray;
82 | }
83 |
84 | this->imageDiff.absDiff(this->imageBg, this->imageGray);
85 | this->imageDiff.threshold(this->threshold);
86 |
87 | this->contourFinder.findContours(this->imageDiff, this->findMin,
88 | this->findMax, MAX_BLOBS, true);
89 |
90 | this->updateContours();
91 | }
92 |
93 | this->updateBump();
94 | }
95 |
96 | void ofApp::onBump() {
97 | this->bg.setHue(rand() % 360);
98 | }
99 |
100 | // make sure 0 <= x <= 1 or you'll get weird results.
101 | float interpolate(float x, float y) {
102 | return (1.0f - pow(x, 3)) * y;
103 | }
104 |
105 | void ofApp::influenceBands(float *levels) {
106 | for (int i = 0; i < BUMP_BANDS; i++) {
107 | if (levels[i] > this->bands[i].level) {
108 | this->bands[i].level = levels[i];
109 | this->bands[i].rawLevel = levels[i];
110 | this->bands[i].cooldown = 0.0f;
111 | } else {
112 | if (this->bands[i].cooldown < 1.0f) {
113 | this->bands[i].cooldown += this->dither;
114 | this->bands[i].cooldown = min(1.0f, this->bands[i].cooldown);
115 | this->bands[i].level = interpolate(
116 | this->bands[i].cooldown,
117 | this->bands[i].rawLevel);
118 | }
119 | }
120 | }
121 | }
122 |
123 | void ofApp::updateBump() {
124 | float *levels = ofSoundGetSpectrum(BUMP_BANDS);
125 | if (!levels) {
126 | // OpenFrameworks has already warned that it's not implemented
127 | // at this point.
128 | // see: ofSoundPlayer.cpp
129 | return;
130 | }
131 |
132 | this->influenceBands(levels);
133 |
134 | int enabled = 0;
135 | this->bump = 0.0f;
136 | for (int i = 0; i < BUMP_BANDS; i++) {
137 | if (this->bands[i].enabled) {
138 | this->bump += this->bands[i].level;
139 | ++enabled;
140 | }
141 | }
142 |
143 | if (!enabled) {
144 | return;
145 | }
146 |
147 | this->bump /= (float) enabled;
148 |
149 | if (this->bump >= this->bumpThreshold) {
150 | if (this->cooldown <= 0) {
151 | this->onBump();
152 | this->cooldown = this->cooldownCount;
153 | } else {
154 | --this->cooldown;
155 | }
156 | }
157 | }
158 |
159 | void ofApp::processPath(std::vector &blob, ofPath &path) {
160 | // first, we use a PolyLine to simplify the blob points
161 | this->simplifier.clear();
162 | this->simplifier.addVertices(blob);
163 | this->simplifier.close();
164 | this->simplifier.simplify(this->simplification);
165 |
166 | // then, we re-curve them so they don't look like origami
167 | std::vector &vertices = this->simplifier.getVertices();
168 | std::vector::iterator itr = vertices.begin();
169 | bool first = true;
170 | while (itr != vertices.end()) {
171 | ofPoint p = *itr;
172 | if (first) {
173 | first = false;
174 | path.moveTo(p);
175 | } else {
176 | path.curveTo(p);
177 | }
178 | ++itr;
179 | }
180 | }
181 |
182 | void ofApp::updateContours() {
183 | this->holes.setFillColor(this->bg);
184 |
185 | this->silhouettes.clear();
186 | this->holes.clear();
187 |
188 | vector::iterator bit = this->contourFinder.blobs.begin();
189 | while (bit != this->contourFinder.blobs.end()) {
190 | ofxCvBlob blob = *(bit++);
191 |
192 | if (blob.hole && !this->showHoles) {
193 | continue;
194 | }
195 |
196 | ofPath &path = blob.hole
197 | ? this->holes
198 | : this->silhouettes;
199 |
200 | this->processPath(blob.pts, path);
201 | path.close();
202 | }
203 |
204 | float scaleFactorX = (float)ofGetWidth() / (float)IMG_SIZE_W;
205 | float scaleFactorY = (float)ofGetHeight() / (float)IMG_SIZE_H;
206 |
207 | this->silhouettes.scale(scaleFactorX, scaleFactorY);
208 | this->holes.scale(scaleFactorX, scaleFactorY);
209 | }
210 |
211 | void ofApp::drawBumpDebug() {
212 | int height = ofGetHeight();
213 | int barWidth = BUMP_DEBUG_W / BUMP_BANDS;
214 |
215 | if (this->bump >= this->bumpThreshold) {
216 | ofSetHexColor(0x00FFFF);
217 | } else {
218 | ofSetHexColor(0xFF0000);
219 | }
220 |
221 | ofDrawRectangle(
222 | 0,
223 | height - (BUMP_DEBUG_H * this->bump),
224 | BUMP_DEBUG_W,
225 | BUMP_DEBUG_H);
226 |
227 | for (int i = 0; i < BUMP_BANDS; i++) {
228 | if (this->bands[i].enabled) {
229 | ofSetHexColor(0xFFFFFF);
230 | } else {
231 | ofSetHexColor(0);
232 | }
233 |
234 | ofDrawRectangle(
235 | i * barWidth,
236 | height - (BUMP_DEBUG_H * this->bands[i].level),
237 | barWidth,
238 | BUMP_DEBUG_H);
239 | }
240 |
241 | ofSetHexColor(0xFF00FF);
242 | ofDrawRectangle(
243 | 0,
244 | height - (BUMP_DEBUG_H * this->bumpThreshold),
245 | BUMP_DEBUG_W,
246 | 1);
247 | }
248 |
249 | void ofApp::draw() {
250 | this->silhouettes.draw(0, 0);
251 | if (this->showHoles) {
252 | this->holes.draw(0, 0);
253 | }
254 |
255 | if (this->debug) {
256 | ofSetHexColor(0xFFFFFF);
257 | this->image.draw(0, 0);
258 | ofSetColor(0xFF, 0xFF, 0xFF, 100);
259 | this->imageDiff.draw(0, 0);
260 | this->contourFinder.draw(0, 0);
261 | }
262 |
263 | if (this->debugSound) {
264 | this->drawBumpDebug();
265 | }
266 | }
267 |
268 | void ofApp::keyPressed(int key) {
269 | // yes, there is definitely a better way to do this.
270 | switch (key) {
271 | case ' ':
272 | this->learn = true;
273 | break;
274 | case 'h':
275 | this->showHoles = !this->showHoles;
276 | cout << "holes: " << this->showHoles << endl;
277 | break;
278 | case '.':
279 | this->simplification += SIMPLIFICATION_INC;
280 | cout << "simplification: " << this->simplification << endl;
281 | break;
282 | case ',':
283 | this->simplification -= SIMPLIFICATION_INC;
284 | cout << "simplification: " << this->simplification << endl;
285 | break;
286 | case ']':
287 | this->threshold += THRESHOLD_INC;
288 | cout << "threshold: " << this->threshold << endl;
289 | break;
290 | case '[':
291 | this->threshold -= THRESHOLD_INC;
292 | cout << "threshold: " << this->threshold << endl;
293 | break;
294 | case 'd':
295 | this->debug = !this->debug;
296 | cout << "debug: " << this->debug << endl;
297 | break;
298 | case 's':
299 | this->debugSound = !this->debugSound;
300 | cout << "debug sound: " << this->debugSound << endl;
301 | break;
302 | case 'f':
303 | if (this->showCursor = !this->showCursor) {
304 | ofShowCursor();
305 | } else {
306 | ofHideCursor();
307 | }
308 | ofToggleFullscreen();
309 | break;
310 | case 'x':
311 | this->bumpThreshold += BASS_INC;
312 | cout << "bass threshold: " << this->bumpThreshold << endl;
313 | break;
314 | case 'z':
315 | this->bumpThreshold -= BASS_INC;
316 | cout << "bass threshold: " << this->bumpThreshold << endl;
317 | break;
318 | case 'X':
319 | this->bumpThreshold += BASS_INC_FINE;
320 | cout << "bass threshold: " << this->bumpThreshold << endl;
321 | break;
322 | case 'Z':
323 | this->bumpThreshold -= BASS_INC_FINE;
324 | cout << "bass threshold: " << this->bumpThreshold << endl;
325 | break;
326 | case 'w':
327 | this->dither += BUMP_DITHER_INC;
328 | cout << "band dither: " << this->dither << endl;
329 | break;
330 | case 'q':
331 | this->dither -= BUMP_DITHER_INC;
332 | cout << "band dither: " << this->dither << endl;
333 | break;
334 | case '=': // +
335 | ++this->cooldownCount;
336 | cout << "cooldown count: " << this->cooldownCount << endl;
337 | break;
338 | case '-':
339 | --this->cooldownCount;
340 | cout << "cooldown count: " << this->cooldownCount << endl;
341 | break;
342 | case 'v':
343 | ++this->frameDelay;
344 | cout << "frame delay: " << this->frameDelay << endl;
345 | break;
346 | case 'c':
347 | --this->frameDelay;
348 | this->frameDelay = max(this->frameDelay, 0);
349 | cout << "frame delay: " << this->frameDelay << endl;
350 | break;
351 | case 'm':
352 | ++this->findMin;
353 | cout << "this min contour size: " << this->findMin << endl;
354 | break;
355 | case 'n':
356 | --this->findMin;
357 | cout << "this min contour size: " << this->findMin << endl;
358 | break;
359 | case 'M':
360 | ++this->findMax;
361 | cout << "this max contour size: " << this->findMax << endl;
362 | break;
363 | case 'N':
364 | --this->findMax;
365 | cout << "this max contour size: " << this->findMax << endl;
366 | break;
367 | case '1':
368 | case '2': // and I'm free ...
369 | case '3':
370 | case '4':
371 | case '5':
372 | case '6':
373 | case '7': // ... free fallin'...
374 | case '8': // (yes, I understand this could be two conditionals.)
375 | case '9':
376 | this->bands[key - '1'].enabled = !this->bands[key - '1'].enabled;
377 | break;
378 | case '!':
379 | this->bands[9].enabled = !this->bands[9].enabled;
380 | break;
381 | case '@':
382 | this->bands[10].enabled = !this->bands[10].enabled;
383 | break;
384 | case '#':
385 | this->bands[11].enabled = !this->bands[11].enabled;
386 | break;
387 | case '$':
388 | this->bands[12].enabled = !this->bands[12].enabled;
389 | break;
390 | case '%':
391 | this->bands[13].enabled = !this->bands[13].enabled;
392 | break;
393 | case '^':
394 | this->bands[14].enabled = !this->bands[14].enabled;
395 | break;
396 | case '&':
397 | this->bands[15].enabled = !this->bands[15].enabled;
398 | break;
399 | case '*':
400 | this->bands[16].enabled = !this->bands[16].enabled;
401 | break;
402 | case '(':
403 | this->bands[17].enabled = !this->bands[17].enabled;
404 | break;
405 | }
406 | }
407 |
408 | void ofApp::dragEvent(ofDragInfo info) {
409 | // currently only support one file at a time.
410 | if (info.files.size() != 1) {
411 | cerr << "only one file is allowed at a time!" << endl;
412 | return;
413 | }
414 |
415 | string filename = info.files[0];
416 | cout << "starting to play: " << filename << endl;
417 |
418 | this->player.stop();
419 | this->player.loadSound(filename, true);
420 | this->player.play();
421 | }
422 |
--------------------------------------------------------------------------------
/src/ofApp.h:
--------------------------------------------------------------------------------
1 | #ifndef OF_APPLE_DDD_H__
2 | #define OF_APPLE_DDD_H__
3 | #pragma once
4 |
5 | #include
6 |
7 | #include "ofMain.h"
8 | #include "ofxOpenCv.h"
9 |
10 | #define BUMP_BANDS 27
11 |
12 | struct Band {
13 | float level;
14 | float rawLevel;
15 | float cooldown;
16 | bool enabled;
17 | };
18 |
19 | class ofApp : public ofBaseApp {
20 | public:
21 | void setup();
22 | void update();
23 | void draw();
24 |
25 | void dragEvent(ofDragInfo info);
26 |
27 | void keyPressed(int key);
28 | private:
29 | void updateContours();
30 | void updateBump();
31 | void influenceBands(float *bands);
32 | void onBump();
33 |
34 | void drawBumpDebug();
35 |
36 | void processPath(std::vector &blob, ofPath &path);
37 |
38 | ofVideoGrabber grabber;
39 |
40 | ofxCvColorImage image;
41 | ofxCvGrayscaleImage imageGray;
42 | ofxCvGrayscaleImage imageBg;
43 | ofxCvGrayscaleImage imageDiff;
44 |
45 | ofxCvContourFinder contourFinder;
46 | int findMax;
47 | int findMin;
48 |
49 | int frame;
50 | int frameDelay;
51 |
52 | ofSoundPlayer player;
53 | float bumpThreshold;
54 | Band bands[BUMP_BANDS];
55 | float bump;
56 | float dither;
57 | bool debugSound;
58 | int cooldownCount;
59 | int cooldown;
60 |
61 | int threshold;
62 | float simplification;
63 | bool learn;
64 |
65 | ofPath silhouettes;
66 | ofPath holes;
67 | bool showHoles;
68 |
69 | ofPolyline simplifier;
70 |
71 | ofColor bg;
72 |
73 | bool debug;
74 | bool showCursor;
75 | };
76 |
77 | #endif
78 |
--------------------------------------------------------------------------------