├── .eslintignore ├── .eslintrc.js ├── .gitignore ├── .node-version ├── .vscode └── settings.json ├── LICENSE ├── README.md ├── dist-plain ├── style.css ├── vue-camera-gestures.es.js ├── vue-camera-gestures.iife.js └── vue-camera-gestures.umd.js ├── dist ├── style.css ├── vue-camera-gestures.es.js ├── vue-camera-gestures.iife.js └── vue-camera-gestures.umd.js ├── docs ├── .vitepress │ └── config.js ├── api-reference │ └── index.md ├── components │ ├── demo-01.vue │ └── load-mobile-net.vue ├── guide │ └── index.md ├── index.md └── public │ ├── Logo.png │ ├── android-chrome-192x192.png │ ├── android-chrome-512x512.png │ ├── apple-touch-icon.png │ ├── favicon-16x16.png │ ├── favicon-32x32.png │ └── site.webmanifest ├── index.html ├── lib ├── cameraGestures.vue ├── index.js └── loadMobilenet.js ├── logos ├── 1024x1024.png ├── 512x512.png ├── Preview.png └── RoundedCorners.png ├── package.json ├── src ├── Test.vue └── test-entry.js ├── vite-no-dependencies.config.js ├── vite.config.js └── yarn.lock /.eslintignore: -------------------------------------------------------------------------------- 1 | dist 2 | dist-plain -------------------------------------------------------------------------------- /.eslintrc.js: -------------------------------------------------------------------------------- 1 | module.exports = { 2 | extends: [ 3 | "eslint:recommended", 4 | "plugin:vue/vue3-recommended", 5 | "prettier", 6 | "prettier/vue", 7 | ], 8 | rules: { 9 | 'semi': ['error', 'never'] 10 | }, 11 | env: { 12 | browser: true, 13 | node: true 14 | } 15 | } -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | node_modules 2 | .DS_Store 3 | dist-ssr 4 | *.local 5 | docs/.vitepress/dist -------------------------------------------------------------------------------- /.node-version: -------------------------------------------------------------------------------- 1 | 14.15.1 -------------------------------------------------------------------------------- /.vscode/settings.json: -------------------------------------------------------------------------------- 1 | { 2 | "vetur.validation.template": false, 3 | "eslint.validate": ["javascript", "javascriptreact", "vue"] 4 | } -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Daniel Elkington 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # [Vue Camera Gestures](https://vue.cameragestures.com) 2 | 3 | Let users control your Vue app using AI, their camera, and gestures of their choice in just 1 line of HTML! 4 | 5 | [Demo and full documentation](https://vue.cameragestures.com) 6 | 7 | ## Installation 8 | ```bash 9 | npm i vue-camera-gestures --save 10 | ``` 11 | Register the component globally 12 | ```js 13 | import CameraGestures from 'vue-camera-gestures' 14 | import 'vue-camera-gestures/dist/style.css' 15 | 16 | app.component('camera-gestures', CameraGestures) 17 | ``` 18 | 19 | ## Getting Started 20 | ```html 21 | 22 | ``` 23 | This will prompt the user to train and verify a 'Fancy Gesture'. When they perform this gesture the `doSomething()` method will be called. 24 | 25 | The name and number of the events is completely configurable - subscribe to as many as you need. 26 | 27 | To find out how to customize the component further, check out the [docs](https://vue.cameragestures.com). 28 | 29 | ## Vue Support 30 | vue-camera-gestures 1.x supports Vue 2 and the source code is in the `v1-vue2` branch. Find the docs [here](vue2.cameragestures.com) 31 | vue-camera-gestures 2.x supports Vue 3 and the source code is in the `master` branch. Find the docs [here](vue3.cameragestures.com) 32 | -------------------------------------------------------------------------------- /dist-plain/style.css: -------------------------------------------------------------------------------- 1 | .camera-gestures-container[data-v-34778a66]{width:227px}video.camera-gestures-camera-feed[data-v-34778a66]{transform:rotateY(180deg);-webkit-transform:rotateY(180deg);-moz-transform:rotateY(180deg);width:227px;max-width:100%}.camera-gestures-progress-bar[data-v-34778a66]{height:5px;background:#41b883;border-radius:5px 0 0 5px}.camera-gestures-progress-bar.invisible[data-v-34778a66]{background:0 0}.camera-gestures-instructions[data-v-34778a66]{text-align:center}.camera-gestures-loader-container[data-v-34778a66]{width:227px;height:100px}.camera-gestures-lds-ring[data-v-34778a66]{display:block;position:relative;left:calc(50% - 32px);top:calc(50% - 32px);width:64px;height:64px}.camera-gestures-lds-ring div[data-v-34778a66]{box-sizing:border-box;display:block;position:absolute;width:51px;height:51px;margin:6px;border:6px solid #41b883;border-radius:50%;animation:camera-gestures-lds-ring-34778a66 1.2s cubic-bezier(.5,0,.5,1) infinite;border-color:#41b883 transparent transparent transparent}.camera-gestures-lds-ring div[data-v-34778a66]:nth-child(1){animation-delay:-.45s}.camera-gestures-lds-ring div[data-v-34778a66]:nth-child(2){animation-delay:-.3s}.camera-gestures-lds-ring div[data-v-34778a66]:nth-child(3){animation-delay:-.15s}@keyframes camera-gestures-lds-ring-34778a66{0%{transform:rotate(0)}100%{transform:rotate(360deg)}} -------------------------------------------------------------------------------- /dist-plain/vue-camera-gestures.es.js: -------------------------------------------------------------------------------- 1 | import{load as e}from"@tensorflow-models/mobilenet";import{browser as t,tensor as i}from"@tensorflow/tfjs";import{create as r}from"@tensorflow-models/knn-classifier";import{pushScopeId as s,popScopeId as n,openBlock as a,createBlock as o,renderSlot as u,createCommentVNode as c,withDirectives as d,createVNode as h,vShow as l,toDisplayString as m,withScopeId as g}from"vue";var p=async()=>window.vueCameraGestures_loadMobilenetPromise?(await window.vueCameraGestures_loadMobilenetPromise,window.vueCameraGestures_mobilenet):(window.vueCameraGestures_mobilenet||(window.vueCameraGestures_loadMobilenetPromise=e().then((e=>{window.vueCameraGestures_mobilenet=e})).catch((e=>{throw window.vueCameraGestures_loadMobilenetPromise=void 0,e})),await window.vueCameraGestures_loadMobilenetPromise),window.vueCameraGestures_mobilenet);const f={name:"CameraGestures",props:{doVerification:{type:Boolean,default:!0},fireOnce:{type:Boolean,default:!0},gestures:{type:Array,default:void 0},model:{type:String,default:void 0},neutralTrainingPrompt:{type:String,default:"Maintain a neutral position"},neutralVerificationPrompt:{type:String,default:"Verify neutral position"},requiredAccuracy:{type:Number,default:90},showCameraFeedAfterTrainingCycle:{type:Boolean,default:!0},showCameraFeedDuringTraining:{type:Boolean,default:!0},showCameraFeedDuringVerification:{type:Boolean,default:!0},throttleEvents:{type:Number,default:0},trainingDelay:{type:Number,default:1e3},trainingPromptPrefix:{type:String,default:"Perform a gesture: "},trainingTime:{type:Number,default:3e3},trainNeutralLast:{type:Boolean,default:!1},verificationDelay:{type:Number,default:1e3},verificationPromptPrefix:{type:String,default:"Verify gesture: "},verificationTime:{type:Number,default:1e3}},emits:["done-training","done-verification","neutral","verification-failed"],data:function(){return{videoPlaying:!1,busyLoadingMobilenet:!0,state:"training",preparing:!1,currentGestureIndex:-1,timeStartedWaiting:null,timeToFinishWaiting:null,progress:0,gestureVerifyingCorrectSamples:0,gestureVerifyingIncorrectSamples:0,verifyingRetried:!1,lastGestureIndexDetected:-1,lastGestureDetectedTime:null}},computed:{computedGestures:function(){if(void 0===this.gestures){return Object.keys(this.$attrs).filter((e=>e.startsWith("on"))).map((e=>e.substring(2))).map((e=>e.toLowerCase())).map((e=>{let t=e.replace(/([A-Z])/g," $1");return t=t.charAt(0).toUpperCase()+t.slice(1),{event:e,fireOnce:this.fireOnce,name:t,requiredAccuracy:this.requiredAccuracy,throttleEvent:this.throttleEvents,trainingDelay:this.trainingDelay,trainingPrompt:this.trainingPromptPrefix+t,trainingTime:this.trainingTime,verificationDelay:this.verificationDelay,verificationPrompt:this.verificationPromptPrefix+t,verificationTime:this.verificationTime,isNeutral:!1}}))}return this.gestures.map((e=>{let t;return e.name?t=e.name:(t=e.event.replace(/([A-Z])/g," $1"),t=t.charAt(0).toUpperCase()+t.slice(1)),{event:e.event,fireOnce:void 0===e.fireOnce?this.fireOnce:e.fireOnce,name:t,requiredAccuracy:void 0===e.requiredAccuracy?this.requiredAccuracy:e.requiredAccuracy,throttleEvent:void 0===e.throttleEvent?this.throttleEvents:e.throttleEvent,trainingDelay:void 0===e.trainingDelay?this.trainingDelay:e.trainingDelay,trainingPrompt:void 0===e.trainingPrompt?this.trainingPromptPrefix+t:e.trainingPrompt,trainingTime:void 0===e.trainingTime?this.trainingTime:e.trainingTime,verificationDelay:void 0===e.verificationDelay?this.verificationDelay:e.verificationDelay,verificationPrompt:void 0===e.verificationPrompt?this.verificationPromptPrefix+t:e.verificationPrompt,verificationTime:void 0===e.verificationTime?this.verificationTime:e.verificationTime}}))},currentGesture:function(){return this.currentGestureIndex>-1?this.computedGestures[this.currentGestureIndex]:void 0},currentEvent:function(){switch(this.currentGestureIndex){case-2:return"neutral";case-1:return;default:return this.currentGesture.event}},currentEventName:function(){return void 0===this.currentGesture?void 0:this.currentGesture.name},currentInstruction:function(){if("training"===this.state)switch(this.currentGestureIndex){case-2:return this.neutralTrainingPrompt;case-1:return;default:return this.currentGesture.trainingPrompt}else if("testing"===this.state)switch(this.currentGestureIndex){case-2:return this.neutralVerificationPrompt;case-1:return;default:return this.currentGesture.verificationPrompt}},showCameraFeed:function(){switch(this.state){case"training":return this.showCameraFeedDuringTraining;case"testing":return this.showCameraFeedDuringVerification;default:return this.showCameraFeedAfterTrainingCycle}},showProgressBar:function(){return-1!==this.currentGestureIndex&&!this.preparing}},mounted:async function(){this.knn=r(),this.mobilenet=await p(),this.busyLoadingMobilenet=!1,this.mediaStream=await navigator.mediaDevices.getUserMedia({video:!0,audio:!1}),this.$refs.video.srcObject=this.mediaStream,this.$refs.video.play(),this.animationFrameId=requestAnimationFrame(this.animate),this.updateState(),this.silenceWarnings()},unmounted:function(){this.knn&&this.knn.dispose(),this.mediaStream&&this.mediaStream.getTracks().forEach((e=>e.stop())),this.unsilenceWarnings()},methods:{async animate(){if(this.videoPlaying){const e=t.fromPixels(this.$refs.video);switch(this.state){case"training":this.trainFrame(e);break;case"testing":this.testFrame(e);break;case"predicting":this.predictFrame(e)}e.dispose()}this.animationFrameId=requestAnimationFrame(this.animate)},trainFrame(e){if(-1!==this.currentGestureIndex&&!this.preparing){const t=this.classIndexFromGestureIndex(this.currentGestureIndex),i=this.mobilenet.infer(e,"conv_preds");this.knn.addExample(i,t),i.dispose()}},async testFrame(e){if(-1!==this.currentGestureIndex&&!this.preparing){const t=this.mobilenet.infer(e,"conv_preds"),i=await this.knn.predictClass(t,10);this.gestureIndexFromClassIndex(i.classIndex)===this.currentGestureIndex?this.gestureVerifyingCorrectSamples++:this.gestureVerifyingIncorrectSamples++,t.dispose()}},async predictFrame(e){const t=this.mobilenet.infer(e,"conv_preds"),i=await this.knn.predictClass(t,10),r=parseInt(i.label),s=this.gestureIndexFromClassIndex(r),n=-2===s,a=n?void 0:this.computedGestures[s],o=n?"neutral":this.computedGestures[s].event,u=n?this.fireOnce:a.fireOnce,c=n?this.throttleEvents:a.throttleEvent,d=this.lastGestureIndexDetected;if(this.lastGestureIndexDetected=s,s!==d)this.silenceWarnings(),this.$emit(o),this.unsilenceWarnings(),this.lastGestureDetectedTime=(new Date).getTime();else if(!u)if(c>0){const e=(new Date).getTime();e-this.lastGestureDetectedTime>=c&&(this.silenceWarnings(),this.$emit(o),this.unsilenceWarnings(),this.lastGestureDetectedTime=e)}else this.silenceWarnings(),this.$emit(o),this.unsilenceWarnings();t.dispose()},updateState(){if(this.model)return this.loadModelFromJson(this.model),this.state="predicting",void(this.currentGestureIndex=-1);if(this.preparing)return this.preparing=!1,this.scheduleUpdateState(),void requestAnimationFrame(this.updateProgress);if("testing"===this.state){if(100*(this.gestureVerifyingCorrectSamples+0)/(this.gestureVerifyingCorrectSamples+this.gestureVerifyingIncorrectSamples)<(-2===this.currentGestureIndex?this.requiredAccuracy:this.currentGesture.requiredAccuracy))return this.verifyingRetried?(this.getModelJson().then((e=>this.$emit("verification-failed",e))),void this.reset()):(this.verifyingRetried=!0,this.preparing=!0,this.gestureVerifyingIncorrectSamples=this.gestureVerifyingCorrectSamples=0,void this.scheduleUpdateState());this.verifyingRetried=!1,this.gestureVerifyingIncorrectSamples=this.gestureVerifyingCorrectSamples=0}const e=this.currentGestureIndex===this.computedGestures.length-1;return-1===this.currentGestureIndex&&!this.trainNeutralLast||e&&this.trainNeutralLast?(this.currentGestureIndex=-2,this.preparing=!0,void this.scheduleUpdateState()):-2===this.currentGestureIndex&&this.trainNeutralLast||e?("training"===this.state&&this.doVerification?(this.getModelJson().then((e=>this.$emit("done-training",e))),this.state="testing",this.currentGestureIndex=this.trainNeutralLast?0:-2,this.preparing=!0):("testing"===this.state&&this.getModelJson().then((e=>this.$emit("done-verification",e))),this.state="predicting",this.currentGestureIndex=-1),void this.scheduleUpdateState()):(this.currentGestureIndex=-2===this.currentGestureIndex?0:this.currentGestureIndex+1,this.preparing=!0,void this.scheduleUpdateState())},scheduleUpdateState(){let e;if("training"===this.state)if(-2===this.currentGestureIndex)e=this.preparing?this.trainingDelay:this.trainingTime;else{if(!(this.currentGestureIndex>-1))return;e=this.preparing?this.currentGesture.trainingDelay:this.currentGesture.trainingTime}else{if("testing"!==this.state)return;if(-2===this.currentGestureIndex)e=this.preparing?this.verificationDelay:this.verificationTime;else{if(!(this.currentGestureIndex>-1))return;e=this.preparing?this.currentGesture.verificationDelay:this.currentGesture.verificationTime}}this.timeStartedWaiting=(new Date).getTime(),this.timeToFinishWaiting=this.timeStartedWaiting+e,this.updateStateTimeoutId=setTimeout(this.updateState,e)},updateProgress(){const e=this.timeToFinishWaiting-this.timeStartedWaiting,t=(new Date).getTime()-this.timeStartedWaiting;this.progress=t>e?1:t/e,this.showProgressBar&&requestAnimationFrame(this.updateProgress)},reset(){this.knn.clearAllClasses(),this.state="training",this.preparing=!1,this.currentGestureIndex=-1,this.verifyingRetried=!1,clearTimeout(this.updateStateTimeoutId),this.updateState()},classIndexFromGestureIndex(e){return this.trainNeutralLast?-2===e?this.computedGestures.length:e:-2===e?0:e+1},gestureIndexFromClassIndex(e){return this.trainNeutralLast?e===this.computedGestures.length?-2:e:0===e?-2:e-1},async getModelJson(){const e=this.knn.getClassifierDataset(),t=[];for(const i in e)t.push({label:i,values:Array.from(await e[i].data()),shape:e[i].shape});return JSON.stringify(t)},loadModelFromJson(e){const t=JSON.parse(e),r={};t.forEach((e=>{r[e.label]=i(e.values,e.shape)})),this.knn.setClassifierDataset(r)},silenceWarnings(){this.consoleWarnSave=console.warn,console.warn=(e,...t)=>{e.includes("[Vue warn]: Component emitted event")||this.consoleWarnSave(e,...t)}},unsilenceWarnings(){console.warn=this.consoleWarnSave}}},v=g("data-v-34778a66");s("data-v-34778a66");const y={class:"camera-gestures-container"},G={key:0,class:"camera-gestures-loader-container"},w=h("div",{class:"camera-gestures-lds-ring"},[h("div"),h("div"),h("div"),h("div")],-1);n();const I=v(((e,t,i,r,s,n)=>(a(),o("div",y,[u(e.$slots,"loading",{loading:e.busyLoadingMobilenet},(()=>[e.busyLoadingMobilenet?(a(),o("div",G,[w])):c("",!0)])),d(h("video",{ref:"video",autoplay:"",playsinline:"",class:"camera-gestures-camera-feed",onPlaying:t[1]||(t[1]=t=>e.videoPlaying=!0),onPause:t[2]||(t[2]=t=>e.videoPlaying=!1)},null,544),[[l,!e.busyLoadingMobilenet&&n.showCameraFeed]]),u(e.$slots,"progress",{inProgress:n.showProgressBar,progress:e.progress},(()=>[h("div",{style:{width:227*e.progress+"px"},class:[{invisible:!n.showProgressBar},"camera-gestures-progress-bar"]},null,6)])),u(e.$slots,"instructions",{training:"training"===e.state,verifying:"testing"===e.state,event:n.currentEvent,eventName:n.currentEventName},(()=>[d(h("p",{class:"camera-gestures-instructions"},m(n.currentInstruction),513),[[l,n.currentInstruction]])]))]))));f.render=I,f.__scopeId="data-v-34778a66","undefined"!=typeof window?window.CameraGesturesComponent=f:"undefined"!=typeof global&&(global.CameraGesturesComponent=f);export default f;export{p as loadMobilenet}; 2 | -------------------------------------------------------------------------------- /dist-plain/vue-camera-gestures.iife.js: -------------------------------------------------------------------------------- 1 | var CameraGestures=function(e,t,i,r,s){"use strict";var n=async()=>window.vueCameraGestures_loadMobilenetPromise?(await window.vueCameraGestures_loadMobilenetPromise,window.vueCameraGestures_mobilenet):(window.vueCameraGestures_mobilenet||(window.vueCameraGestures_loadMobilenetPromise=t.load().then((e=>{window.vueCameraGestures_mobilenet=e})).catch((e=>{throw window.vueCameraGestures_loadMobilenetPromise=void 0,e})),await window.vueCameraGestures_loadMobilenetPromise),window.vueCameraGestures_mobilenet);const a={name:"CameraGestures",props:{doVerification:{type:Boolean,default:!0},fireOnce:{type:Boolean,default:!0},gestures:{type:Array,default:void 0},model:{type:String,default:void 0},neutralTrainingPrompt:{type:String,default:"Maintain a neutral position"},neutralVerificationPrompt:{type:String,default:"Verify neutral position"},requiredAccuracy:{type:Number,default:90},showCameraFeedAfterTrainingCycle:{type:Boolean,default:!0},showCameraFeedDuringTraining:{type:Boolean,default:!0},showCameraFeedDuringVerification:{type:Boolean,default:!0},throttleEvents:{type:Number,default:0},trainingDelay:{type:Number,default:1e3},trainingPromptPrefix:{type:String,default:"Perform a gesture: "},trainingTime:{type:Number,default:3e3},trainNeutralLast:{type:Boolean,default:!1},verificationDelay:{type:Number,default:1e3},verificationPromptPrefix:{type:String,default:"Verify gesture: "},verificationTime:{type:Number,default:1e3}},emits:["done-training","done-verification","neutral","verification-failed"],data:function(){return{videoPlaying:!1,busyLoadingMobilenet:!0,state:"training",preparing:!1,currentGestureIndex:-1,timeStartedWaiting:null,timeToFinishWaiting:null,progress:0,gestureVerifyingCorrectSamples:0,gestureVerifyingIncorrectSamples:0,verifyingRetried:!1,lastGestureIndexDetected:-1,lastGestureDetectedTime:null}},computed:{computedGestures:function(){if(void 0===this.gestures){return Object.keys(this.$attrs).filter((e=>e.startsWith("on"))).map((e=>e.substring(2))).map((e=>e.toLowerCase())).map((e=>{let t=e.replace(/([A-Z])/g," $1");return t=t.charAt(0).toUpperCase()+t.slice(1),{event:e,fireOnce:this.fireOnce,name:t,requiredAccuracy:this.requiredAccuracy,throttleEvent:this.throttleEvents,trainingDelay:this.trainingDelay,trainingPrompt:this.trainingPromptPrefix+t,trainingTime:this.trainingTime,verificationDelay:this.verificationDelay,verificationPrompt:this.verificationPromptPrefix+t,verificationTime:this.verificationTime,isNeutral:!1}}))}return this.gestures.map((e=>{let t;return e.name?t=e.name:(t=e.event.replace(/([A-Z])/g," $1"),t=t.charAt(0).toUpperCase()+t.slice(1)),{event:e.event,fireOnce:void 0===e.fireOnce?this.fireOnce:e.fireOnce,name:t,requiredAccuracy:void 0===e.requiredAccuracy?this.requiredAccuracy:e.requiredAccuracy,throttleEvent:void 0===e.throttleEvent?this.throttleEvents:e.throttleEvent,trainingDelay:void 0===e.trainingDelay?this.trainingDelay:e.trainingDelay,trainingPrompt:void 0===e.trainingPrompt?this.trainingPromptPrefix+t:e.trainingPrompt,trainingTime:void 0===e.trainingTime?this.trainingTime:e.trainingTime,verificationDelay:void 0===e.verificationDelay?this.verificationDelay:e.verificationDelay,verificationPrompt:void 0===e.verificationPrompt?this.verificationPromptPrefix+t:e.verificationPrompt,verificationTime:void 0===e.verificationTime?this.verificationTime:e.verificationTime}}))},currentGesture:function(){return this.currentGestureIndex>-1?this.computedGestures[this.currentGestureIndex]:void 0},currentEvent:function(){switch(this.currentGestureIndex){case-2:return"neutral";case-1:return;default:return this.currentGesture.event}},currentEventName:function(){return void 0===this.currentGesture?void 0:this.currentGesture.name},currentInstruction:function(){if("training"===this.state)switch(this.currentGestureIndex){case-2:return this.neutralTrainingPrompt;case-1:return;default:return this.currentGesture.trainingPrompt}else if("testing"===this.state)switch(this.currentGestureIndex){case-2:return this.neutralVerificationPrompt;case-1:return;default:return this.currentGesture.verificationPrompt}},showCameraFeed:function(){switch(this.state){case"training":return this.showCameraFeedDuringTraining;case"testing":return this.showCameraFeedDuringVerification;default:return this.showCameraFeedAfterTrainingCycle}},showProgressBar:function(){return-1!==this.currentGestureIndex&&!this.preparing}},mounted:async function(){this.knn=r.create(),this.mobilenet=await n(),this.busyLoadingMobilenet=!1,this.mediaStream=await navigator.mediaDevices.getUserMedia({video:!0,audio:!1}),this.$refs.video.srcObject=this.mediaStream,this.$refs.video.play(),this.animationFrameId=requestAnimationFrame(this.animate),this.updateState(),this.silenceWarnings()},unmounted:function(){this.knn&&this.knn.dispose(),this.mediaStream&&this.mediaStream.getTracks().forEach((e=>e.stop())),this.unsilenceWarnings()},methods:{async animate(){if(this.videoPlaying){const e=i.browser.fromPixels(this.$refs.video);switch(this.state){case"training":this.trainFrame(e);break;case"testing":this.testFrame(e);break;case"predicting":this.predictFrame(e)}e.dispose()}this.animationFrameId=requestAnimationFrame(this.animate)},trainFrame(e){if(-1!==this.currentGestureIndex&&!this.preparing){const t=this.classIndexFromGestureIndex(this.currentGestureIndex),i=this.mobilenet.infer(e,"conv_preds");this.knn.addExample(i,t),i.dispose()}},async testFrame(e){if(-1!==this.currentGestureIndex&&!this.preparing){const t=this.mobilenet.infer(e,"conv_preds"),i=await this.knn.predictClass(t,10);this.gestureIndexFromClassIndex(i.classIndex)===this.currentGestureIndex?this.gestureVerifyingCorrectSamples++:this.gestureVerifyingIncorrectSamples++,t.dispose()}},async predictFrame(e){const t=this.mobilenet.infer(e,"conv_preds"),i=await this.knn.predictClass(t,10),r=parseInt(i.label),s=this.gestureIndexFromClassIndex(r),n=-2===s,a=n?void 0:this.computedGestures[s],o=n?"neutral":this.computedGestures[s].event,u=n?this.fireOnce:a.fireOnce,c=n?this.throttleEvents:a.throttleEvent,d=this.lastGestureIndexDetected;if(this.lastGestureIndexDetected=s,s!==d)this.silenceWarnings(),this.$emit(o),this.unsilenceWarnings(),this.lastGestureDetectedTime=(new Date).getTime();else if(!u)if(c>0){const e=(new Date).getTime();e-this.lastGestureDetectedTime>=c&&(this.silenceWarnings(),this.$emit(o),this.unsilenceWarnings(),this.lastGestureDetectedTime=e)}else this.silenceWarnings(),this.$emit(o),this.unsilenceWarnings();t.dispose()},updateState(){if(this.model)return this.loadModelFromJson(this.model),this.state="predicting",void(this.currentGestureIndex=-1);if(this.preparing)return this.preparing=!1,this.scheduleUpdateState(),void requestAnimationFrame(this.updateProgress);if("testing"===this.state){if(100*(this.gestureVerifyingCorrectSamples+0)/(this.gestureVerifyingCorrectSamples+this.gestureVerifyingIncorrectSamples)<(-2===this.currentGestureIndex?this.requiredAccuracy:this.currentGesture.requiredAccuracy))return this.verifyingRetried?(this.getModelJson().then((e=>this.$emit("verification-failed",e))),void this.reset()):(this.verifyingRetried=!0,this.preparing=!0,this.gestureVerifyingIncorrectSamples=this.gestureVerifyingCorrectSamples=0,void this.scheduleUpdateState());this.verifyingRetried=!1,this.gestureVerifyingIncorrectSamples=this.gestureVerifyingCorrectSamples=0}const e=this.currentGestureIndex===this.computedGestures.length-1;return-1===this.currentGestureIndex&&!this.trainNeutralLast||e&&this.trainNeutralLast?(this.currentGestureIndex=-2,this.preparing=!0,void this.scheduleUpdateState()):-2===this.currentGestureIndex&&this.trainNeutralLast||e?("training"===this.state&&this.doVerification?(this.getModelJson().then((e=>this.$emit("done-training",e))),this.state="testing",this.currentGestureIndex=this.trainNeutralLast?0:-2,this.preparing=!0):("testing"===this.state&&this.getModelJson().then((e=>this.$emit("done-verification",e))),this.state="predicting",this.currentGestureIndex=-1),void this.scheduleUpdateState()):(this.currentGestureIndex=-2===this.currentGestureIndex?0:this.currentGestureIndex+1,this.preparing=!0,void this.scheduleUpdateState())},scheduleUpdateState(){let e;if("training"===this.state)if(-2===this.currentGestureIndex)e=this.preparing?this.trainingDelay:this.trainingTime;else{if(!(this.currentGestureIndex>-1))return;e=this.preparing?this.currentGesture.trainingDelay:this.currentGesture.trainingTime}else{if("testing"!==this.state)return;if(-2===this.currentGestureIndex)e=this.preparing?this.verificationDelay:this.verificationTime;else{if(!(this.currentGestureIndex>-1))return;e=this.preparing?this.currentGesture.verificationDelay:this.currentGesture.verificationTime}}this.timeStartedWaiting=(new Date).getTime(),this.timeToFinishWaiting=this.timeStartedWaiting+e,this.updateStateTimeoutId=setTimeout(this.updateState,e)},updateProgress(){const e=this.timeToFinishWaiting-this.timeStartedWaiting,t=(new Date).getTime()-this.timeStartedWaiting;this.progress=t>e?1:t/e,this.showProgressBar&&requestAnimationFrame(this.updateProgress)},reset(){this.knn.clearAllClasses(),this.state="training",this.preparing=!1,this.currentGestureIndex=-1,this.verifyingRetried=!1,clearTimeout(this.updateStateTimeoutId),this.updateState()},classIndexFromGestureIndex(e){return this.trainNeutralLast?-2===e?this.computedGestures.length:e:-2===e?0:e+1},gestureIndexFromClassIndex(e){return this.trainNeutralLast?e===this.computedGestures.length?-2:e:0===e?-2:e-1},async getModelJson(){const e=this.knn.getClassifierDataset(),t=[];for(const i in e)t.push({label:i,values:Array.from(await e[i].data()),shape:e[i].shape});return JSON.stringify(t)},loadModelFromJson(e){const t=JSON.parse(e),r={};t.forEach((e=>{r[e.label]=i.tensor(e.values,e.shape)})),this.knn.setClassifierDataset(r)},silenceWarnings(){this.consoleWarnSave=console.warn,console.warn=(e,...t)=>{e.includes("[Vue warn]: Component emitted event")||this.consoleWarnSave(e,...t)}},unsilenceWarnings(){console.warn=this.consoleWarnSave}}},o=s.withScopeId("data-v-34778a66");s.pushScopeId("data-v-34778a66");const u={class:"camera-gestures-container"},c={key:0,class:"camera-gestures-loader-container"},d=s.createVNode("div",{class:"camera-gestures-lds-ring"},[s.createVNode("div"),s.createVNode("div"),s.createVNode("div"),s.createVNode("div")],-1);s.popScopeId();const h=o(((e,t,i,r,n,a)=>(s.openBlock(),s.createBlock("div",u,[s.renderSlot(e.$slots,"loading",{loading:e.busyLoadingMobilenet},(()=>[e.busyLoadingMobilenet?(s.openBlock(),s.createBlock("div",c,[d])):s.createCommentVNode("",!0)])),s.withDirectives(s.createVNode("video",{ref:"video",autoplay:"",playsinline:"",class:"camera-gestures-camera-feed",onPlaying:t[1]||(t[1]=t=>e.videoPlaying=!0),onPause:t[2]||(t[2]=t=>e.videoPlaying=!1)},null,544),[[s.vShow,!e.busyLoadingMobilenet&&a.showCameraFeed]]),s.renderSlot(e.$slots,"progress",{inProgress:a.showProgressBar,progress:e.progress},(()=>[s.createVNode("div",{style:{width:227*e.progress+"px"},class:[{invisible:!a.showProgressBar},"camera-gestures-progress-bar"]},null,6)])),s.renderSlot(e.$slots,"instructions",{training:"training"===e.state,verifying:"testing"===e.state,event:a.currentEvent,eventName:a.currentEventName},(()=>[s.withDirectives(s.createVNode("p",{class:"camera-gestures-instructions"},s.toDisplayString(a.currentInstruction),513),[[s.vShow,a.currentInstruction]])]))]))));return a.render=h,a.__scopeId="data-v-34778a66","undefined"!=typeof window?window.CameraGesturesComponent=a:"undefined"!=typeof global&&(global.CameraGesturesComponent=a),e.default=a,e.loadMobilenet=n,Object.defineProperty(e,"__esModule",{value:!0}),e[Symbol.toStringTag]="Module",e}({},mobilenet,tf,knnClassifier,Vue); 2 | -------------------------------------------------------------------------------- /dist-plain/vue-camera-gestures.umd.js: -------------------------------------------------------------------------------- 1 | !function(e,t){"object"==typeof exports&&"undefined"!=typeof module?t(exports,require("@tensorflow-models/mobilenet"),require("@tensorflow/tfjs"),require("@tensorflow-models/knn-classifier"),require("vue")):"function"==typeof define&&define.amd?define(["exports","@tensorflow-models/mobilenet","@tensorflow/tfjs","@tensorflow-models/knn-classifier","vue"],t):t((e="undefined"!=typeof globalThis?globalThis:e||self).CameraGestures={},e.mobilenet,e.tf,e.knnClassifier,e.Vue)}(this,(function(e,t,i,r,s){"use strict";var n=async()=>window.vueCameraGestures_loadMobilenetPromise?(await window.vueCameraGestures_loadMobilenetPromise,window.vueCameraGestures_mobilenet):(window.vueCameraGestures_mobilenet||(window.vueCameraGestures_loadMobilenetPromise=t.load().then((e=>{window.vueCameraGestures_mobilenet=e})).catch((e=>{throw window.vueCameraGestures_loadMobilenetPromise=void 0,e})),await window.vueCameraGestures_loadMobilenetPromise),window.vueCameraGestures_mobilenet);const a={name:"CameraGestures",props:{doVerification:{type:Boolean,default:!0},fireOnce:{type:Boolean,default:!0},gestures:{type:Array,default:void 0},model:{type:String,default:void 0},neutralTrainingPrompt:{type:String,default:"Maintain a neutral position"},neutralVerificationPrompt:{type:String,default:"Verify neutral position"},requiredAccuracy:{type:Number,default:90},showCameraFeedAfterTrainingCycle:{type:Boolean,default:!0},showCameraFeedDuringTraining:{type:Boolean,default:!0},showCameraFeedDuringVerification:{type:Boolean,default:!0},throttleEvents:{type:Number,default:0},trainingDelay:{type:Number,default:1e3},trainingPromptPrefix:{type:String,default:"Perform a gesture: "},trainingTime:{type:Number,default:3e3},trainNeutralLast:{type:Boolean,default:!1},verificationDelay:{type:Number,default:1e3},verificationPromptPrefix:{type:String,default:"Verify gesture: "},verificationTime:{type:Number,default:1e3}},emits:["done-training","done-verification","neutral","verification-failed"],data:function(){return{videoPlaying:!1,busyLoadingMobilenet:!0,state:"training",preparing:!1,currentGestureIndex:-1,timeStartedWaiting:null,timeToFinishWaiting:null,progress:0,gestureVerifyingCorrectSamples:0,gestureVerifyingIncorrectSamples:0,verifyingRetried:!1,lastGestureIndexDetected:-1,lastGestureDetectedTime:null}},computed:{computedGestures:function(){if(void 0===this.gestures){return Object.keys(this.$attrs).filter((e=>e.startsWith("on"))).map((e=>e.substring(2))).map((e=>e.toLowerCase())).map((e=>{let t=e.replace(/([A-Z])/g," $1");return t=t.charAt(0).toUpperCase()+t.slice(1),{event:e,fireOnce:this.fireOnce,name:t,requiredAccuracy:this.requiredAccuracy,throttleEvent:this.throttleEvents,trainingDelay:this.trainingDelay,trainingPrompt:this.trainingPromptPrefix+t,trainingTime:this.trainingTime,verificationDelay:this.verificationDelay,verificationPrompt:this.verificationPromptPrefix+t,verificationTime:this.verificationTime,isNeutral:!1}}))}return this.gestures.map((e=>{let t;return e.name?t=e.name:(t=e.event.replace(/([A-Z])/g," $1"),t=t.charAt(0).toUpperCase()+t.slice(1)),{event:e.event,fireOnce:void 0===e.fireOnce?this.fireOnce:e.fireOnce,name:t,requiredAccuracy:void 0===e.requiredAccuracy?this.requiredAccuracy:e.requiredAccuracy,throttleEvent:void 0===e.throttleEvent?this.throttleEvents:e.throttleEvent,trainingDelay:void 0===e.trainingDelay?this.trainingDelay:e.trainingDelay,trainingPrompt:void 0===e.trainingPrompt?this.trainingPromptPrefix+t:e.trainingPrompt,trainingTime:void 0===e.trainingTime?this.trainingTime:e.trainingTime,verificationDelay:void 0===e.verificationDelay?this.verificationDelay:e.verificationDelay,verificationPrompt:void 0===e.verificationPrompt?this.verificationPromptPrefix+t:e.verificationPrompt,verificationTime:void 0===e.verificationTime?this.verificationTime:e.verificationTime}}))},currentGesture:function(){return this.currentGestureIndex>-1?this.computedGestures[this.currentGestureIndex]:void 0},currentEvent:function(){switch(this.currentGestureIndex){case-2:return"neutral";case-1:return;default:return this.currentGesture.event}},currentEventName:function(){return void 0===this.currentGesture?void 0:this.currentGesture.name},currentInstruction:function(){if("training"===this.state)switch(this.currentGestureIndex){case-2:return this.neutralTrainingPrompt;case-1:return;default:return this.currentGesture.trainingPrompt}else if("testing"===this.state)switch(this.currentGestureIndex){case-2:return this.neutralVerificationPrompt;case-1:return;default:return this.currentGesture.verificationPrompt}},showCameraFeed:function(){switch(this.state){case"training":return this.showCameraFeedDuringTraining;case"testing":return this.showCameraFeedDuringVerification;default:return this.showCameraFeedAfterTrainingCycle}},showProgressBar:function(){return-1!==this.currentGestureIndex&&!this.preparing}},mounted:async function(){this.knn=r.create(),this.mobilenet=await n(),this.busyLoadingMobilenet=!1,this.mediaStream=await navigator.mediaDevices.getUserMedia({video:!0,audio:!1}),this.$refs.video.srcObject=this.mediaStream,this.$refs.video.play(),this.animationFrameId=requestAnimationFrame(this.animate),this.updateState(),this.silenceWarnings()},unmounted:function(){this.knn&&this.knn.dispose(),this.mediaStream&&this.mediaStream.getTracks().forEach((e=>e.stop())),this.unsilenceWarnings()},methods:{async animate(){if(this.videoPlaying){const e=i.browser.fromPixels(this.$refs.video);switch(this.state){case"training":this.trainFrame(e);break;case"testing":this.testFrame(e);break;case"predicting":this.predictFrame(e)}e.dispose()}this.animationFrameId=requestAnimationFrame(this.animate)},trainFrame(e){if(-1!==this.currentGestureIndex&&!this.preparing){const t=this.classIndexFromGestureIndex(this.currentGestureIndex),i=this.mobilenet.infer(e,"conv_preds");this.knn.addExample(i,t),i.dispose()}},async testFrame(e){if(-1!==this.currentGestureIndex&&!this.preparing){const t=this.mobilenet.infer(e,"conv_preds"),i=await this.knn.predictClass(t,10);this.gestureIndexFromClassIndex(i.classIndex)===this.currentGestureIndex?this.gestureVerifyingCorrectSamples++:this.gestureVerifyingIncorrectSamples++,t.dispose()}},async predictFrame(e){const t=this.mobilenet.infer(e,"conv_preds"),i=await this.knn.predictClass(t,10),r=parseInt(i.label),s=this.gestureIndexFromClassIndex(r),n=-2===s,a=n?void 0:this.computedGestures[s],o=n?"neutral":this.computedGestures[s].event,u=n?this.fireOnce:a.fireOnce,c=n?this.throttleEvents:a.throttleEvent,d=this.lastGestureIndexDetected;if(this.lastGestureIndexDetected=s,s!==d)this.silenceWarnings(),this.$emit(o),this.unsilenceWarnings(),this.lastGestureDetectedTime=(new Date).getTime();else if(!u)if(c>0){const e=(new Date).getTime();e-this.lastGestureDetectedTime>=c&&(this.silenceWarnings(),this.$emit(o),this.unsilenceWarnings(),this.lastGestureDetectedTime=e)}else this.silenceWarnings(),this.$emit(o),this.unsilenceWarnings();t.dispose()},updateState(){if(this.model)return this.loadModelFromJson(this.model),this.state="predicting",void(this.currentGestureIndex=-1);if(this.preparing)return this.preparing=!1,this.scheduleUpdateState(),void requestAnimationFrame(this.updateProgress);if("testing"===this.state){if(100*(this.gestureVerifyingCorrectSamples+0)/(this.gestureVerifyingCorrectSamples+this.gestureVerifyingIncorrectSamples)<(-2===this.currentGestureIndex?this.requiredAccuracy:this.currentGesture.requiredAccuracy))return this.verifyingRetried?(this.getModelJson().then((e=>this.$emit("verification-failed",e))),void this.reset()):(this.verifyingRetried=!0,this.preparing=!0,this.gestureVerifyingIncorrectSamples=this.gestureVerifyingCorrectSamples=0,void this.scheduleUpdateState());this.verifyingRetried=!1,this.gestureVerifyingIncorrectSamples=this.gestureVerifyingCorrectSamples=0}const e=this.currentGestureIndex===this.computedGestures.length-1;return-1===this.currentGestureIndex&&!this.trainNeutralLast||e&&this.trainNeutralLast?(this.currentGestureIndex=-2,this.preparing=!0,void this.scheduleUpdateState()):-2===this.currentGestureIndex&&this.trainNeutralLast||e?("training"===this.state&&this.doVerification?(this.getModelJson().then((e=>this.$emit("done-training",e))),this.state="testing",this.currentGestureIndex=this.trainNeutralLast?0:-2,this.preparing=!0):("testing"===this.state&&this.getModelJson().then((e=>this.$emit("done-verification",e))),this.state="predicting",this.currentGestureIndex=-1),void this.scheduleUpdateState()):(this.currentGestureIndex=-2===this.currentGestureIndex?0:this.currentGestureIndex+1,this.preparing=!0,void this.scheduleUpdateState())},scheduleUpdateState(){let e;if("training"===this.state)if(-2===this.currentGestureIndex)e=this.preparing?this.trainingDelay:this.trainingTime;else{if(!(this.currentGestureIndex>-1))return;e=this.preparing?this.currentGesture.trainingDelay:this.currentGesture.trainingTime}else{if("testing"!==this.state)return;if(-2===this.currentGestureIndex)e=this.preparing?this.verificationDelay:this.verificationTime;else{if(!(this.currentGestureIndex>-1))return;e=this.preparing?this.currentGesture.verificationDelay:this.currentGesture.verificationTime}}this.timeStartedWaiting=(new Date).getTime(),this.timeToFinishWaiting=this.timeStartedWaiting+e,this.updateStateTimeoutId=setTimeout(this.updateState,e)},updateProgress(){const e=this.timeToFinishWaiting-this.timeStartedWaiting,t=(new Date).getTime()-this.timeStartedWaiting;this.progress=t>e?1:t/e,this.showProgressBar&&requestAnimationFrame(this.updateProgress)},reset(){this.knn.clearAllClasses(),this.state="training",this.preparing=!1,this.currentGestureIndex=-1,this.verifyingRetried=!1,clearTimeout(this.updateStateTimeoutId),this.updateState()},classIndexFromGestureIndex(e){return this.trainNeutralLast?-2===e?this.computedGestures.length:e:-2===e?0:e+1},gestureIndexFromClassIndex(e){return this.trainNeutralLast?e===this.computedGestures.length?-2:e:0===e?-2:e-1},async getModelJson(){const e=this.knn.getClassifierDataset(),t=[];for(const i in e)t.push({label:i,values:Array.from(await e[i].data()),shape:e[i].shape});return JSON.stringify(t)},loadModelFromJson(e){const t=JSON.parse(e),r={};t.forEach((e=>{r[e.label]=i.tensor(e.values,e.shape)})),this.knn.setClassifierDataset(r)},silenceWarnings(){this.consoleWarnSave=console.warn,console.warn=(e,...t)=>{e.includes("[Vue warn]: Component emitted event")||this.consoleWarnSave(e,...t)}},unsilenceWarnings(){console.warn=this.consoleWarnSave}}},o=s.withScopeId("data-v-34778a66");s.pushScopeId("data-v-34778a66");const u={class:"camera-gestures-container"},c={key:0,class:"camera-gestures-loader-container"},d=s.createVNode("div",{class:"camera-gestures-lds-ring"},[s.createVNode("div"),s.createVNode("div"),s.createVNode("div"),s.createVNode("div")],-1);s.popScopeId();const l=o(((e,t,i,r,n,a)=>(s.openBlock(),s.createBlock("div",u,[s.renderSlot(e.$slots,"loading",{loading:e.busyLoadingMobilenet},(()=>[e.busyLoadingMobilenet?(s.openBlock(),s.createBlock("div",c,[d])):s.createCommentVNode("",!0)])),s.withDirectives(s.createVNode("video",{ref:"video",autoplay:"",playsinline:"",class:"camera-gestures-camera-feed",onPlaying:t[1]||(t[1]=t=>e.videoPlaying=!0),onPause:t[2]||(t[2]=t=>e.videoPlaying=!1)},null,544),[[s.vShow,!e.busyLoadingMobilenet&&a.showCameraFeed]]),s.renderSlot(e.$slots,"progress",{inProgress:a.showProgressBar,progress:e.progress},(()=>[s.createVNode("div",{style:{width:227*e.progress+"px"},class:[{invisible:!a.showProgressBar},"camera-gestures-progress-bar"]},null,6)])),s.renderSlot(e.$slots,"instructions",{training:"training"===e.state,verifying:"testing"===e.state,event:a.currentEvent,eventName:a.currentEventName},(()=>[s.withDirectives(s.createVNode("p",{class:"camera-gestures-instructions"},s.toDisplayString(a.currentInstruction),513),[[s.vShow,a.currentInstruction]])]))]))));a.render=l,a.__scopeId="data-v-34778a66","undefined"!=typeof window?window.CameraGesturesComponent=a:"undefined"!=typeof global&&(global.CameraGesturesComponent=a),e.default=a,e.loadMobilenet=n,Object.defineProperty(e,"__esModule",{value:!0}),e[Symbol.toStringTag]="Module"})); 2 | -------------------------------------------------------------------------------- /dist/style.css: -------------------------------------------------------------------------------- 1 | .camera-gestures-container[data-v-34778a66]{width:227px}video.camera-gestures-camera-feed[data-v-34778a66]{transform:rotateY(180deg);-webkit-transform:rotateY(180deg);-moz-transform:rotateY(180deg);width:227px;max-width:100%}.camera-gestures-progress-bar[data-v-34778a66]{height:5px;background:#41b883;border-radius:5px 0 0 5px}.camera-gestures-progress-bar.invisible[data-v-34778a66]{background:0 0}.camera-gestures-instructions[data-v-34778a66]{text-align:center}.camera-gestures-loader-container[data-v-34778a66]{width:227px;height:100px}.camera-gestures-lds-ring[data-v-34778a66]{display:block;position:relative;left:calc(50% - 32px);top:calc(50% - 32px);width:64px;height:64px}.camera-gestures-lds-ring div[data-v-34778a66]{box-sizing:border-box;display:block;position:absolute;width:51px;height:51px;margin:6px;border:6px solid #41b883;border-radius:50%;animation:camera-gestures-lds-ring-34778a66 1.2s cubic-bezier(.5,0,.5,1) infinite;border-color:#41b883 transparent transparent transparent}.camera-gestures-lds-ring div[data-v-34778a66]:nth-child(1){animation-delay:-.45s}.camera-gestures-lds-ring div[data-v-34778a66]:nth-child(2){animation-delay:-.3s}.camera-gestures-lds-ring div[data-v-34778a66]:nth-child(3){animation-delay:-.15s}@keyframes camera-gestures-lds-ring-34778a66{0%{transform:rotate(0)}100%{transform:rotate(360deg)}} -------------------------------------------------------------------------------- /docs/.vitepress/config.js: -------------------------------------------------------------------------------- 1 | module.exports = { 2 | title: 'Vue Camera Gestures', 3 | description: 4 | 'Let users control your Vue app using AI, their camera, and gestures of their choice in just 1 line of HTML!', 5 | head: [ 6 | [ 7 | 'link', 8 | { 9 | rel: 'apple-touch-icon', 10 | sizes: '180x180', 11 | href: '/apple-touch-icon.png', 12 | }, 13 | ], 14 | [ 15 | 'link', 16 | { 17 | rel: 'icon', 18 | type: 'image/png', 19 | sizes: '32x32', 20 | href: '/favicon-32x32.png', 21 | }, 22 | ], 23 | [ 24 | 'link', 25 | { 26 | rel: 'icon', 27 | type: 'image/png', 28 | sizes: '16x16', 29 | href: '/favicon-16x16.png', 30 | }, 31 | ], 32 | ], 33 | themeConfig: { 34 | nav: [ 35 | { text: 'Guide', link: '/guide/' }, 36 | { text: 'API Reference', link: '/api-reference/' }, 37 | ], 38 | algolia: { 39 | appId: 'ADW16RLRMC', 40 | apiKey: '7720237913e955174354861df42d834c', 41 | indexName: 'cameragestures' 42 | }, 43 | lastUpdated: 'Last Updated', 44 | repo: 'danielelkington/vue-camera-gestures', 45 | docsDir: 'docs', 46 | editLinks: true 47 | } 48 | } 49 | -------------------------------------------------------------------------------- /docs/api-reference/index.md: -------------------------------------------------------------------------------- 1 | # API Reference 2 | ::: danger Vue Support 3 | This page is only relevant for Vue 3 only. If you are using Vue 2, please go to [vue2.cameragestures.com](https://vue2.cameragestures.com) 4 | ::: 5 | ## Props 6 | |Prop name | Type | Default Value | Description| 7 | | -------- | ---- | ------------- | ---- | 8 | | __doVerification__ | `Boolean`| `true` | If true, after training gestures the user will be prompted to verify them to establish that the model has been trained correctly. If false, the verification stage will be skipped.| 9 | | __fireOnce__ | `Boolean`| `true` | If true, an event will be fired when a user makes a gesture, but will not be fired again until either a different gesture is detected, or the user first returns to the neutral position.| 10 | | __gestures__ | `array`| `undefined` | See the [Gestures prop](#gestures-prop) notes| 11 | | __model__ | `String`| `undefined` | A trained model, in JSON format. If this is not `null` or `undefined`, training and verification will be skipped, and the provided model will be used.| 12 | | __neutralTrainingPrompt__ | `String` | `'Maintain a neutral position'` | Displayed while training the neutral position.| 13 | | __neutralVerification Prompt__ | `String` | `'Verify neutral position'` | Displayed while training the neutral position.| 14 | | __requiredAccuracy__ | `Number` | `90` | A number between 0 and 100. Each gesture must have at least this percent accuracy during verification, otherwise the training cycle will repeat.| 15 | | __showCameraFeed AfterTrainingCycle__ | `Boolean` | `true` | Whether the camera feed will be displayed after the training cycle has finished.| 16 | | __showCameraFeed DuringTraining__ | `Boolean` | `true` | Whether the camera feed will be displayed while training gestures.| 17 | | __showCameraFeed DuringVerification__ | `Boolean` | `true` | Whether the camera feed will be displayed while verifying gestures.| 18 | | __throttleEvents__ | `Number` | `0` | Only has an effect if `fireOnce` is false. Throttles how often an event gets fired (in milliseconds) if the user persists with a gesture. If 0, the event will be fired each frame the user continues persisting with the gesture.| 19 | | __trainingDelay__ | `Number` | `1000` | The number of milliseconds to wait after first displaying a prompt to train a gesture before the training of that gesture commences. Should be high enough to give the user time to start doing that gesture.| 20 | | __trainingPromptPrefix__ | `String` | `'Perform a gesture: '` | Displayed before a gesture name when training the model.| 21 | | __trainingTime__ | `Number` | `3000` | The number of milliseconds to spend taking snapshots from the camera feed and using them to train a model for a gesture.| 22 | | __trainNeutralLast__ | `Boolean` | `false` | By default, the neutral gesture is trained and verified first. If this prop is true, it will be trained and verified last.| 23 | | __verificationDelay__ | `Number` | `1000` | The number of milliseconds to wait after first displaying a prompt to verify a gesture before the verification of that gesture commences. Should be high enough to give the user time to start doing that gesture.| 24 | | __verificationPrompt Prefix__ | `String` | `'Verify gesture: '` | Displayed before a gesture name when verifying the model.| 25 | | __verificationTime__ | `Number` | `1000` | The number of milliseconds to spend taking snapshots from the camera feed and using them to verify that a gesture has been successfully trained.| 26 | ### gestures prop 27 | The gestures prop is an array of objects for each gesture. Each object can have the following properties: 28 | |Property name | Type | Description| 29 | | -------- | ---- | ---- | 30 | | __event__ | `String`| _Mandatory._ The name of the event that will be fired when this gesture is detected.| 31 | | __fireOnce__ | `Boolean`| If true, an event will be fired when a user makes the gesture, but will not be fired again until either a different gesture is detected, or the user first returns to the neutral position.| 32 | | __name__ | `String`| The name of the gesture shown in prompts.| 33 | | __requiredAccuracy__ | `Number`| A number between 0 and 100. The gesture must have at least this percent accuracy during verification, otherwise the training cycle will repeat.| 34 | | __throttleEvent__ | `Number`| Only has an effect if `fireOnce` is false (on this object or in a prop). Throttles how often an event gets fired (in milliseconds) if the user persists with this gesture. If 0, the event will be fired each frame the user continues persisting with the gesture.| 35 | | __trainingDelay__ | `Number`| How many milliseconds to spend showing the user the prompt before starting to train the gesture.| 36 | | __trainingPrompt__ | `String`| The prompt displayed before and while this gesture is being trained.| 37 | | __trainingTime__ | `Number`| How many milliseconds to spend training the gesture.| 38 | | __verificationDelay__ | `Number`| How many milliseconds to spend showing the user the prompt before starting to verify the gesture.| 39 | | __verificationPrompt__ | `String`| The prompt displayed before and while this gesture is being verified.| 40 | | __verificationTime__ | `Number`| How many milliseconds to spend verifying the gesture.| 41 | ## Events 42 | |Event | Argument | Description| 43 | | -------- | ---- | ------------- | 44 | | __doneTraining__ |The trained model as a JSON string| Emitted when training has finished, and verification is about to start| 45 | | __doneVerification__ |The trained model as a JSON string| Emitted when verification has successfully finished. At this point events will be fired if the user repeats the gestures.| 46 | | __neutral__ ||Emitted when the neutral position is detected. The frequency is determined by the `fireOnce` and `throttleEvents` props.| 47 | | __verificationFailed__ |The failed model as a JSON string| Emitted when verification has failed to meet the required accuracy. The training cycle will begin again.| 48 | 49 | ### Custom events 50 | The `gestures` prop contains an array of objects where the `event` property has the name of an event that will be fired when the user performs that gesture. 51 | 52 | If the `gestures` prop is not provided, gestures will be determined based on any events being listened to that are not one of the reserved events listed above. 53 | ## Slots 54 | |Slot | Slot Props | Description| 55 | | -------- | ---- | ------------- | 56 | | __instructions__ |See the [guide](../guide/#customizing-the-instructions)| Customizes the appearance of the instructions| 57 | | __loading__ |See the [guide](../guide/#customizing-the-initial-loading-indicator)| Customizes the appearance of the loading indicator that initially appears while the Mobilenet model is loading| 58 | | __progress__ |See the [guide](../guide/#customizing-the-progress-bar)| Customizes the appearance of the progress bar| 59 | 60 | 61 | 62 | -------------------------------------------------------------------------------- /docs/components/demo-01.vue: -------------------------------------------------------------------------------- 1 | 15 | 16 | 31 | 32 | -------------------------------------------------------------------------------- /docs/components/load-mobile-net.vue: -------------------------------------------------------------------------------- 1 | 2 | 5 | 13 | -------------------------------------------------------------------------------- /docs/guide/index.md: -------------------------------------------------------------------------------- 1 | # Guide 2 | Vue Camera Gestures is a component that when placed on a page will 3 | - Request access to the user's camera 4 | - Prompt the user to train a number of configurable gestures 5 | - Prompt the user to repeat the gestures to verify the AI model 6 | - Emit events when the user performs the gestures 7 | 8 | ::: warning Vue Support 9 | These are the docs for vue-camera-gestures 2.x, which supports Vue 3. If you are using Vue 2, please go to [vue2.cameragestures.com](https://vue2.cameragestures.com) 10 | ::: 11 | 12 | ## Demo 13 | 14 | 15 | 16 | 17 | 25 | 26 | ```html 27 | 33 | 34 | 43 | ``` 44 | 45 | ## Basic Example 46 | ```html 47 | 48 | ``` 49 | This will 50 | - Prompt the user to train three separate gestures of their choice using their device's camera (as well as a neutral position) 51 | - Once training is completed, the events will be fired when the user performs the gestures. 52 | 53 | Note that __the name and number of the events is completely configurable__ - you can simply subscribe to as many as you need. 54 | ## Installation 55 | ### via npm 56 | ```bash 57 | npm i vue-camera-gestures --save 58 | ``` 59 | Register the component globally 60 | ```js 61 | import CameraGestures from 'vue-camera-gestures' 62 | import 'vue-camera-gestures/dist/style.css' 63 | 64 | app.component('camera-gestures', CameraGestures) 65 | ``` 66 | Or register it in a Single File Component 67 | ```html 68 | 78 | ``` 79 | You can ignore the peer dependency warning as the Tensorflow JS libraries are bundled with the component. You can also [import the Vue Camera Gestures library on its own (only 4KB Gzipped!)](#installing-without-the-bundled-version-of-tensorflow). 80 | ### via CDN 81 | ```html 82 | 83 | 84 | 85 | 92 | ``` 93 | 94 | ::: tip 95 | For production, it is recommended that you link to a specific veresion number to avoid unexpected breakage from newer versions. 96 | ::: 97 | 98 | ## Getting Started 99 | The simplest way to specify both the gestures to be trained and to subscribe to events is to simply subscribe to an event with a name of your choice. Please note that there are a number of [reserved event names](../api-reference/#events). 100 | ```html 101 | 102 | ``` 103 | Event names in camelCase will be split into separate capitalized words. 104 | - When the gestures is being trained the user will see the prompt `Perform a gesture: Fancy Gesture` 105 | - When the AI model is being tested, the user will see the prompt `Verify gesture: Fancy Gesture` 106 | ## Customizing prompts 107 | The following code: 108 | ```html 109 | 116 | 117 | ``` 118 | will display prompts such as: 119 | - `Perform a gesture: Go Left` 120 | - `Verify gesture: Go Left` 121 | 122 | Props are available to let you customize the training and verification prefixes. The following code: 123 | ```html 124 | 133 | 134 | ``` 135 | will display prompts such as: 136 | - `Signal with your hand: Go Left` 137 | - `Again, signal with your hand: Go Left` 138 | 139 | To customize the full prompts for each event: 140 | ```html 141 | 148 | 149 | ``` 150 | The default "neutral" prompts are: 151 | - `Maintain a neutral position` 152 | - `Verify neutral position` 153 | 154 | These can be customized, as shown: 155 | ```html 156 | 159 | 160 | ``` 161 | ## Event frequency 162 | By default, an event will be fired __once__ when the user makes the gesture, and will not be fired again until either 163 | - a different gesture is detected or 164 | - the user first returns to the neutral position. 165 | 166 | This behaviour can be turned off using the `fireOnce` prop. If set to false events will be fired each frame they are detected. 167 | ```html 168 | 169 | ``` 170 | To limit how often the same event can be fired, use the `throttleEvents` prop, specifying a number of milliseconds. If the user persists with a gesture, its event will be fired straight away, and then at the specified interval. 171 | 172 | In the following example, if the user persisted the `up` gesture, the `up()` method would be called immediately, and then once every second. 173 | ```html 174 | 175 | ``` 176 | These settings can also be customized per gesture. 177 | 178 | ```html 179 | 186 | 187 | ``` 188 | ## Customizing the appearance 189 | By default the component displays a feed from the user's camera with a progress bar below it (when training or verifying), and then an instruction. 190 | After the training cycle has completed, only the camera feed is displayed by default. 191 | ### Hiding the camera feed 192 | The camera feed can be turned off at various points using the `showCameraFeedDuringTraining`, `showCameraFeedDuringVerification` and `showCameraFeedAfterTrainingCycle` props. 193 | ```html 194 | 199 | ``` 200 | Note that this will still render the video element in the DOM due to TensorFlow requirements - it will just be hidden. 201 | ### Customizing the progress bar 202 | The progress bar can be customized using the `progress` slot. The slot props object contains the following properties: 203 | - `inProgress`: `Boolean` Whether something is in progress and a progress bar should be displayed 204 | - `progress`: `Number` A number between 0 and 1 indicating the current progress 205 | 206 | For example, to instead render a custom progress bar (if you have elsewhere defined `my-progress-bar`): 207 | ```html 208 | 209 |