11 |
12 |
13 |
14 |
15 |
16 |
17 |
23 |
24 |
25 |
26 |
27 |
BRFv4 - API Reference
28 |
Overview
29 |
30 |
31 |
32 | The BRFv4 API is the same on all platforms. Everything described on this page can be applied to Javascript, Actionscript 3 or C++.
33 | Before you read this API, please take a look at what you can do with BRFv4, especially what results to expect from
34 | face detection and face tracking (link: What can I do with it?).
35 | There are two main classes: BRFManager and BRFFace.
36 | Going forward "roi" means "region of interest", the rectangular area of an image an algorithm works in.
37 |
38 |
39 |
40 |
41 |
42 |
BRFManager
43 |
44 |
(control flow)
45 |
46 |
54 |
55 |
(face detection)
56 |
57 |
64 |
65 |
(face tracking)
66 |
67 |
74 |
75 |
(point tracking - optical flow)
76 |
77 |
86 |
87 |
88 |
89 |
BRFFace
90 |
91 |
92 |
93 |
111 |
112 |
113 |
114 |
BRFMode
115 |
116 |
117 |
118 |
123 |
124 |
125 |
126 |
BRFState
127 |
128 |
129 |
130 |
136 |
137 |
138 |
139 |
140 |
↑ top
141 |
142 |
143 |
144 |
145 |
146 |
147 |
148 |
BRFManager
149 |
150 |
151 |
152 |
153 |
154 | BRFManager controls everything. First you need to call:
155 |
156 |
157 | init( ... )
158 |
159 |
160 | Based on the image size the SDK sets reasonable default parameters to start tracking a single face,
161 | but you can change any parameter once BRFv4 is ready. Keep in mind that most parameters are pixel values
162 | and you need to adjust parameters based on the image data size that you put in, eg.
163 |
164 |
165 | setFaceDetectionParams( int minWidth, int maxWidth, int stepSize, int minMergeNeighbors ) : void
166 |
167 |
168 | If you use a 640x480 camera resolution the trackable face size is 480 pixel at maximum.
A reasonable minimum would be 25% of
169 | that max size, which is 120 pixel.
170 |
171 |
172 | For a 1920x1080 resolution the trackable face size is 1080 pixel and 25% of that is 270 pixel.
173 |
174 |
175 | That's why the examples use percentages for all areas and sizes. This makes sure that the camera resolution can easily be changed.
176 |
177 |
178 | Once everything is set up you need to update the image data before calling:
179 |
180 |
181 | update( ... )
182 |
183 |
184 | BRFv4 tries to detect a face. If a face was detected BRFv4 immediately starts the face tracking based on it.
185 |
186 |
187 | Face Detection operates globally. That's why BRFManager and not BRFFace got the result functions:
188 |
189 |
190 | getAllDetectedFaces() : Vector<Rectangle>
191 | getMergedDetectedFaces() : Vector<Rectangle>
192 |
193 |
194 | The detailed face tracking data can be retrieved by calling:
195 |
196 |
197 | getFaces() : Vector<BRFFace>
198 |
199 | which gives you a list of BRFFaces with a length based on your setting of:
200 |
201 |
202 | setNumFacesToTrack( int numFaces )
203 |
204 |
205 | And that's all about it. Init, set parameters, and continuously update.
206 | For more information about the implementation details for your specific platform, see the sample packages.
207 |
208 |
209 |
210 |
(control flow)
211 |
212 |
213 |
214 |
init( Rectangle imageDataSize, Rectangle imageRoi, String appId ) : void
215 |
216 | This function is the starting point.
217 |
218 |
219 | It tells BRFv4 what image data size it should expect, on which area to operate and what your appId is.
220 |
221 | If you want to switch the image data and thus need to change the image data size, just call init() again
222 | to reinit BRFv4.
Don't create a new BRFManager! Reuse a single instance.
223 |
224 | When you first call init() a license check is performed. It's a single server call. If it fails in the trial version
225 | BRFv4 will stop working after about 1 to 2 minutes. The commercial version is fail safe,
226 | which means that even if no connection to the license server can be established, BRFv4 will still work.
227 | Apart from that BRFv4 works fully on the client side.
228 |
229 |
230 | - imageDataSize (Rectangle)
231 | - The size of the image data you put in.
For a 640x480 camera resolution you need an imageDataSize of
232 | Rectangle(0, 0, 640, 480) or for a mobile portrait camera resolutions you will need eg. Rectangle(0, 0, 480, 640).
233 |
234 |
235 | - imageRoi (Rectangle)
236 | - The area within imageDataSize that you want to analyse. Usually you want to analyse the whole image, but
237 | sometimes you only need a part of it and restrict the area.
For a 640x480 camera resolution you will
238 | want to set the roi at the same size as the imageDataSize: Rectangle(0, 0, 640, 480). But maybe you want to
239 | restrict it to the center, eg. Rectangle(80, 0, 480, 480), this would restrict the analysed area from
240 | xStart = 80 to xEnd = 80 + 480 = 560.
241 |
242 | - appId (String)
243 | - Choose your own appId (minimum length is 8 characters). In our examples we usually put something in like
244 | "com.tastenkunst.brfv4.as3.examples". Create your own appId using your reversed company domain.
245 |
246 |
↑ top
247 |
248 |
249 |
250 |
251 |
252 |
253 |
update( Bitmap imageData ) : void
254 |
255 | This function does the tracking job. It operates in either of the three BRFModes.
256 |
257 |
258 | In C++ implementations, eg. OpenCV (camera handling) or iOS, you need to update the imageData on your own before calling update.
259 |
260 |
261 | All other platforms have their known image containers and BRFv4 is already set up to deal with those.
262 |
263 |
264 | The results can be retrieved after calling update() as arrays of Rectangle or BRFFace:
265 |
266 |
267 | getAllDetectedFaces() : Vector<Rectangle>
268 | getMergedDetectedFaces() : Vector<Rectangle>
269 | getFaces() : Vector<BRFFace>
270 |
271 |
272 | - imageData (a platform dependent image data)
273 | - This is platform dependent. In C++ implementations, eg. OpenCV (camera handling) or iOS, you need to update
274 | the imageData on your own before calling update. All other platforms have their known image containers and
275 | BRFv4 is already set up to deal with those, eg. Android(Java): Bitmap, AS3: BitmapData etc.
276 |
277 |
↑ top
278 |
279 |
280 |
281 |
282 |
283 |
284 |
reset() : void
285 |
286 | All face trackers reset and start over by performing face detection. Also all point tracking points are removed.
287 |
288 |
↑ top
289 |
290 |
291 |
292 |
293 |
294 |
295 |
getMode() : String
296 |
297 | Returns the BRFMode that is currently set.
298 |
299 |
↑ top
300 |
301 |
302 |
303 |
304 |
305 |
306 |
setMode(String mode) : void
307 |
308 | Sets one of the three modes from BRFMode:
309 |
310 |
311 |
312 | - BRFMode.FACE_DETECTION
313 | - This mode will only perform the face detection part and skip the detailed analysis.
314 |
315 | - BRFMode.FACE_TRACKING
316 | - This is the usual mode for face tracking. It performs face detection to find a face and then tracks
317 | the facial features in detail.
318 |
319 | - BRFMode.POINT_TRACKING
320 | - This mode will skip on face detection and tracking and only perform point tracking. You can perform
321 | face tracking and point tracking simultaneously by choosing FACE_TRACKING as mode.
322 |
323 |
324 |
325 | - mode (String)
326 | - either BRFMode.FACE_DETECTION, BRFMode.FACE_TRACKING or BRFMode.POINT_TRACKING
327 |
328 |
↑ top
329 |
330 |
331 |
332 |
333 |
(face detection)
334 |
335 |
336 |
337 |
setFaceDetectionParams( int minWidth, int maxWidth, int stepSize, int minMergeNeighbors ) : void
338 |
339 | Internally BRFv4 uses a DYNx480 (landscape) or 480xDYN (portrait) image for it's analysis. So 480px is the
340 | base size that every other input size compares to (eg. 1280x720 -> 854x480).
341 |
342 |
343 | The minimum detectable face size for the following resolutions are:
344 |
345 |
346 | - 640 x 480: 24px ( 480 / 480 = 1.00 * 24 = 24) (base)
347 | - 1280 x 720: 36px ( 720 / 480 = 1.50 * 24 = 36)
348 | - 1920 x 1080: 54px (1080 / 480 = 2.25 * 24 = 54)
349 |
350 |
351 | Also: faces (blue) are only detected at step sizes multiple of 12.
352 | So the actual face detection layers (sizes) are:
353 |
354 |
355 | - 640 x 480: 24, 36, 48, 60, 72, ..., 456, 468, 480
356 | - 1280 x 720: 36, 54, 72, 90, 108, ..., 684, 702, 720
357 | - 1920 x 1080: 54, 81, 108, 135, 162, ..., 1026, 1053, 1080
358 |
359 |
360 |
361 | Detected faces (blue) get merged (yellow) if they are
362 | + roughly placed in the same location,
363 | + roughly the same size and
364 | + have at least minMergeNeighbors of other rectangle in the same spot.
365 |
366 |
367 | - minWidth (int)
368 | - The starting/minimum face size (in pixel) that the algorithm is looking for.
369 |
370 | - maxWidth (int)
371 | - The end/maximum face size (in pixel) that the algorithm is looking for.
372 |
373 | - stepSize (int)
374 | - Minimum step size is 12, must be a multiple of 12 (12, 24, 36 etc.). By increasing the stepSize you skip detection layers and make the face detection less accurate, but perform faster.
375 |
376 | - minMergeNeighbors (int)
377 | - Number of candidates (blue) necessary in one spot to merge all of them into a "detected face" (yellow).
378 |
379 |
↑ top
380 |
381 |
382 |
383 |
384 |
385 |
386 |
setFaceDetectionRoi( Rectangle roi ) : void
387 |
388 | Sets the region of interest, the area within the image data where the face detection shall do its work.
389 |
390 |
391 | - roi (Rectangle)
392 | - A rectangular area within the image.
393 |
394 |
↑ top
395 |
396 |
397 |
398 |
399 |
400 |
401 |
getAllDetectedFaces( ) : Vector<Rectangle>
402 |
403 | Returns all face candidates (blue) that were detected during the last update.
404 |
405 |
↑ top
406 |
407 |
408 |
409 |
410 |
411 |
412 |
getMergedDetectedFaces( ) : Vector<Rectangle>
413 |
414 | Returns all actual merged/detected faces (yellow) of the last update call.
415 |
416 |
↑ top
417 |
418 |
419 |
420 |
421 |
(face tracking)
422 |
423 |
424 |
425 |
setNumFacesToTrack( int numFaces ) : void
426 |
427 | By default BRFv4 sets the number of faces to track to 1 (single face tracking). To track more than one face
428 | increase this value. Be aware that increasing the number of faces to track will hurt the performance with
429 | every face you add.
430 |
431 |
432 | - numFaces (int)
433 | - The number of faces you want to track, usually 1.
434 |
435 |
↑ top
436 |
437 |
438 |
439 |
440 |
441 |
442 |
setFaceTrackingStartParams( double minWidth, double maxWidth, double rx, double ry, double rz ) : void
443 |
444 | By setting these parameters you can control the conditions for picking up a face. Restrict the distance/range to
445 | the camera by setting minWidth and maxWidth. Smaller values mean more distance to the camera. Restricting
446 | the starting angles may help to create a consistent user experience.
447 | Keep the minWidth and maxWidth values about the same as for the params set in setFaceDetectionParams();
448 |
449 |
450 | - minWidth (double)
451 | - A pixel based value. The minimum size a face must have to be picked up.
452 | - maxWidth (double)
453 | - A pixel based value. The maximum size a face can have to be picked up.
454 | - rx (double)
455 | - A degree angle. Rotation around the X axis (pitch, turning head up/down)
456 | - ry (double)
457 | - A degree angle. Rotation around the Y axis (yaw, turning head to the left/right)
458 | - rz (double)
459 | - A degree angle. Rotation around the Z axis (roll, tilt head to the left/right)
460 |
461 |
↑ top
462 |
463 |
464 |
465 |
466 |
467 |
468 |
setFaceTrackingResetParams( double minWidth, double maxWidth, double rx, double ry, double rz ) : void
469 |
470 | By setting these parameters you can control the conditions for resetting the tracking of a face. If a face gets
471 | smaller than minWidth or larger than maxWidth or if the head turns more than rx, ry, or rz, the tracking will reset.
472 |
473 |
474 | - minWidth (double)
475 | - A pixel based value. Faces smaller than this value will trigger a reset of the tracking.
476 | - maxWidth (double)
477 | - A pixel based value. Faces larger than this value will trigger a reset of the tracking.
478 | - rx (double)
479 | - A degree angle. Rotation around the X axis (pitch, turning head up/down).
480 | - ry (double)
481 | - A degree angle. Rotation around the Y axis (yaw, turning head to the left/right)
482 | - rz (double)
483 | - A degree angle. Rotation around the Z axis (roll, tilt head to the left/right)
484 |
485 |
↑ top
486 |
487 |
488 |
489 |
490 |
491 |
492 |
getFaces( ) : Vector<BRFFace>
493 |
494 | Returns the list of currently tracked faces. Please see BRFFace for details.
495 |
496 |
↑ top
497 |
498 |
499 |
500 |
501 |
(point tracking)
502 |
503 |
504 |
505 |
setOpticalFlowParams( int patchSize, int numLevels, int numIterations, double error ) : void
506 |
507 | Optical Flow is a point tracking algorithm. It is used in BRFv4 to stabilize fast movements. You can perform
508 | this point tracking either standalone by setting BRFMode.POINT_TRACKING or in parallel by setting either one of
509 | the other two modes.
510 |
511 |
512 | - patchSize (int)
513 | -
514 | This is the area around a point that is used to compare to the new image. Has to be an odd number.
515 | Larger values will decrease performance. Lower values will decrease accuracy. 21 (10 pixels on in each direction)
516 | is a good default value.
517 |
518 | - numLevels (int)
519 | - The number of pyramid levels to use. Either 1, 2, 3 or 4 (default). Lower values will decrease accuracy.
520 | - numIterations (int)
521 | - The maximum number of iterations to perform the search.
522 | - error (double)
523 | - Usually smaller than 0.0001. Higher values will decrease accuracy.
524 |
525 |
↑ top
526 |
527 |
528 |
529 |
530 |
531 |
532 |
addOpticalFlowPoints( Vector<Point> pointArray ) : void
533 |
534 | Adds new points to track. This should be done synchronously right before calling update(). If you call this method in the middle
535 | of an update call, eg. by doing so in a click/touch handler, the tracking might get confused.
536 |
537 |
538 | - pointArray (Vector<Point>)
539 | -
540 | A list of points to add to the optical flow tracking.
541 |
542 |
543 |
↑ top
544 |
545 |
546 |
547 |
548 |
549 |
550 |
getOpticalFlowCheckPointsValidBeforeTracking() : Boolean
551 |
552 | Returns whether BRF will check the validity of points.
553 |
554 |
↑ top
555 |
556 |
557 |
558 |
559 |
560 |
561 |
setOpticalFlowCheckPointsValidBeforeTracking( Boolean value ) : void
562 |
563 | Sets whether BRF will check the validity of points. In continuous point tracking there will be
564 | points that fail to be tracked. Their state will be false after an update call. As a developer you can react
565 | on failed points by checking the state. If you wish that BRF removes failed points right before the next
566 | tracking, set this value to true. If you wish to handle the removal or resetting of points yourself, set
567 | this value to false.
568 |
569 |
570 | - value (Boolean)
571 | -
572 | true or false
573 |
574 |
575 |
↑ top
576 |
577 |
578 |
579 |
580 |
581 |
582 |
getOpticalFlowPoints() : Vector<Point>
583 |
584 | Returns all currently tracked points.
585 |
586 |
↑ top
587 |
588 |
589 |
590 |
591 |
592 |
593 |
getOpticalFlowPointStates() : Vector<Boolean>
594 |
595 | Returns the states of all currently tracked points.
596 |
597 |
↑ top
598 |
599 |
600 |
601 |
602 |
603 |
604 |
605 |
606 |
BRFFace
607 |
608 |
609 |
610 |
611 |
612 | BRFFace holds all information about a tracked face. BRFv4 is now able to track more than one face, so every
613 | BRFFace has it's individual state. These states are being set automatically by BRFv4 depending on what it
614 | currently does.
615 |
616 | Compared to BRFv3 there was a change on how the vertices are laid out.
617 | 
618 | The origin for the 3D transformation (scale, translationX, translationY, rotationX, rotationY, rotationZ)
619 | is right behind vertex index 27 (top of the nose). So if you want to put a 3D object or PNG file on top of a
620 | tracked face you know exactly where to position it.
621 |
622 |
623 |
624 |
625 |
626 |
lastState (String)
627 |
628 | The last state this face was in before calling update. See BRFState.
629 |
630 |
↑ top
631 |
632 |
633 |
634 |
635 |
636 |
637 |
state (String)
638 |
639 | The state this face is currently in. Either one of the following BRFStates:
640 |
641 |
642 | - BRFState.RESET
643 | - This face was reset because of a call to reset or because it exceeded the reset parameters.
644 |
645 | - BRFState.FACE_DETECTION
646 | - BRFv4 is looking for a face, still nothing found.
647 |
648 | - BRFState.FACE_TRACKING_START
649 | -
650 | BRFv4 found a face and tries to align the face tracking to the identified face. In this state the candide3
651 | model and 3D position might not be correct yet.
652 |
653 |
654 | - BRFState.FACE_TRACKING
655 | - BRFv4 aligned the face and is now tracking it.
656 |
657 |
↑ top
658 |
659 |
660 |
661 |
662 |
663 |
664 |
nextState (String)
665 |
666 | The next thing BRF does with this faces. See BRFState.
667 |
668 |
↑ top
669 |
670 |
671 |
672 |
673 |
674 |
675 |
vertices (float[])
676 |
677 | A list of vertex positions in the form of [x, y, x, y, ..., x, y], length is therefore: 68 * 2
678 |
679 | Getting the position of a certain vertex would be like:
680 |
681 | var vx = vertices[index * 2]; // where index is eg. 27 for the top of the nose.
682 | var vy = vertices[index * 2 + 1];
683 |
684 |
↑ top
685 |
686 |
687 |
688 |
689 |
690 |
691 |
triangles (int[])
692 |
693 | The vertices can be connected and triangles can be formed.
This is the vertex index list for the triangles
694 | int the following form: [i0, i1, i2, i0, i1, i2, ... , i0, i1, i2] where 3 indices span a triangle.
695 |
696 |
↑ top
697 |
698 |
699 |
700 |
701 |
702 |
703 |
points (Vector<Point>)
704 |
705 | Same as vertices, but a list of points in the form of [{x, y}, {x, y}, ..., {x, y}], length is therefore: 68
706 |
707 | Getting the position of a certain point would be like:
708 |
709 | var px = points[index].x; // where index is eg. 27 for the top of the nose.
710 | var py = points[index].y;
711 |
712 |
↑ top
713 |
714 |
715 |
716 |
717 |
718 |
719 |
bounds (Rectangle)
720 |
721 | This is the outline rectangle of all vertices.
722 |
723 |
↑ top
724 |
725 |
726 |
727 |
728 |
729 |
730 |
refRect (Rectangle)
731 |
732 | This is the reference rectangle that the face tracking algorithm uses to perform the tracking.
733 |
734 |
↑ top
735 |
736 |
737 |
738 |
739 |
740 |
741 |
candideVertices (float[])
742 |
743 | The candide 3 model vertices. Can be used to visualize the 3D position, scaling and rotation.
744 |
745 |
↑ top
746 |
747 |
748 |
749 |
750 |
751 |
752 |
candideTriangles (int[])
753 |
754 | Triangle list to be able to draw the candide 3 model.
755 |
756 |
↑ top
757 |
758 |
759 |
760 |
761 |
762 |
763 |
scale (float)
764 |
765 | The scale of the whole face.
766 |
767 |
↑ top
768 |
769 |
770 |
771 |
772 |
773 |
774 |
translationX (float)
775 |
776 | The x position of the origin of face on the image. The origin is right behind vertex index 27 (top of the nose).
777 |
778 |
↑ top
779 |
780 |
781 |
782 |
783 |
784 |
785 |
translationY (float)
786 |
787 | The y position of the origin of face on the image. The origin is right behind vertex index 27 (top of the nose).
788 |
789 |
↑ top
790 |
791 |
792 |
793 |
794 |
795 |
796 |
rotationX (float)
797 |
798 | A radian angle. Rotation around the X axis (pitch, turning head up/down)
799 |
800 |
↑ top
801 |
802 |
803 |
804 |
805 |
806 |
807 |
rotationY (float)
808 |
809 | A radian angle. Rotation around the Y axis (yaw, turning head to the left/right)
810 |
811 |
↑ top
812 |
813 |
814 |
815 |
816 |
817 |
818 |
rotationZ (float)
819 |
820 | A radian angle. Rotation around the Z axis (roll, tilt head to the left/right)
821 |
822 |
↑ top
823 |
824 |
825 |
826 |
827 |
828 |
829 |
BRFMode
830 |
831 |
832 |
833 |
834 |
835 | BRFv4 can operate in either of the following modes.
836 |
837 |
838 |
839 |
840 |
841 |
FACE_DETECTION (String)
842 |
843 | Set this mode if you want to skip the face tracking part and only detect face rectangles. Point tracking can
844 | also be done synchronously.
845 |
846 |
↑ top
847 |
848 |
849 |
850 |
851 |
852 |
853 |
FACE_TRACKING (String)
854 |
855 | Set this mode if you want the full face detection and face tracking. Point tracking can
856 | also be done synchronously.
857 |
858 |
↑ top
859 |
860 |
861 |
862 |
863 |
864 |
865 |
POINT_TRACKING (String)
866 |
867 | Set this mode if you want to skip face detection and face tracking and only track points.
868 |
869 |
↑ top
870 |
871 |
872 |
873 |
874 |
875 |
876 |
BRFState
877 |
878 |
879 |
880 |
881 |
882 | A BRFFace can be in one of the following states.
883 |
884 |
885 |
886 |
887 |
888 |
FACE_DETECTION (String)
889 |
890 | The BRFFace tried to initially detect a face in the image data.
891 |
892 |
↑ top
893 |
894 |
895 |
896 |
897 |
898 |
899 |
FACE_TRACKING_START (String)
900 |
901 | The BRFFace found a face in the image data and tries to align the face tracking.
902 |
903 |
↑ top
904 |
905 |
906 |
907 |
908 |
909 |
910 |
FACE_TRACKING (String)
911 |
912 | The BRFFace is tracking the face in the image data.
913 |
914 |
↑ top
915 |
916 |
917 |
918 |
919 |
920 |
921 |
RESET (String)
922 |
923 | The BRFFace triggered a reset, either because of a call to BRFManager.reset() or because
924 | it exceeds the reset parameters set in BRFManager.setFaceTrackingResetParams();
925 |
926 |
↑ top
927 |
928 |
929 |
930 |
931 |
932 |
933 |
53 | 
57 | 
61 | 
65 | 
90 | 
94 | 
98 | 
102 | 
117 | 
121 | 
125 | 
129 | 
144 | 
148 | 
152 | 
156 | 
174 | 
178 |