├── .github └── workflows │ └── auto-publish.yml ├── .gitignore ├── .pr-preview.json ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE.md ├── Makefile ├── README.md ├── explainer.md ├── favicon-32x32.png ├── favicon-96x96.png ├── favicon.ico ├── images └── hand-layout.svg ├── index.bs ├── package.json └── w3c.json /.github/workflows/auto-publish.yml: -------------------------------------------------------------------------------- 1 | name: Build, and publish spec to GitHub Pages and /TR/ 2 | 3 | on: 4 | pull_request: {} 5 | push: 6 | branches: [main] 7 | paths: 8 | - 'images/**' 9 | - 'index.bs' 10 | 11 | jobs: 12 | main: 13 | name: Build, Validate and Deploy 14 | runs-on: ubuntu-20.04 15 | steps: 16 | - uses: actions/checkout@v2 17 | - uses: w3c/spec-prod@v2 18 | with: 19 | TOOLCHAIN: bikeshed 20 | SOURCE: index.bs 21 | DESTINATION: index.html 22 | GH_PAGES_BRANCH: gh-pages 23 | W3C_ECHIDNA_TOKEN: ${{ secrets.W3C_TR_TOKEN }} 24 | W3C_WG_DECISION_URL: https://lists.w3.org/Archives/Public/public-immersive-web-wg/2021Sep/0004.html 25 | W3C_BUILD_OVERRIDE: | 26 | status: WD 27 | 28 | # not set 'warning' to BUILD_FAIL_ON (not to cause error by bikeshed warning?) 29 | 30 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | index.html -------------------------------------------------------------------------------- /.pr-preview.json: -------------------------------------------------------------------------------- 1 | { 2 | "src_file": "index.bs", 3 | "type": "bikeshed", 4 | "params": { 5 | "force": 1 6 | } 7 | } 8 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Code of Conduct 2 | 3 | All documentation, code and communication under this repository are covered by the [W3C Code of Ethics and Professional Conduct](https://www.w3.org/Consortium/cepc/). 4 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing guidelines for the Immersive Web Working Group 2 | 3 | Contributions to this repository are intended to become part of software or documents licensed under the [Software and Document License](https://www.w3.org/Consortium/Legal/copyright-software). By committing here, you agree to that licensing of your contributions. 4 | 5 | If you are not the sole contributor to a contribution (pull request), please identify all contributors in the pull request comment. 6 | 7 | To add a contributor (other than yourself, that's automatic), mark them one per line as follows: 8 | 9 | ``` 10 | +@github_username 11 | ``` 12 | 13 | If you added a contributor by mistake, you can remove them in a comment with: 14 | 15 | ``` 16 | -@github_username 17 | ``` 18 | 19 | If you are making a pull request on behalf of someone else but you had no part in designing the feature, you can remove yourself with the above syntax. 20 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | All Reports in this Repository are licensed by Contributors 2 | under the 3 | [W3C Software and Document License](http://www.w3.org/Consortium/Legal/2015/copyright-software-and-document). 4 | 5 | Contributions to Specifications are made under the 6 | [W3C CLA](https://www.w3.org/community/about/agreements/cla/). 7 | 8 | Contributions to Test Suites are made under the 9 | [W3C 3-clause BSD License](https://www.w3.org/Consortium/Legal/2008/03-bsd-license.html) 10 | 11 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | .PHONY: all index.html 2 | 3 | all: index.html 4 | 5 | index.html: index.bs 6 | curl https://api.csswg.org/bikeshed/ -F file=@index.bs -F output=err 7 | curl https://api.csswg.org/bikeshed/ -F file=@index.bs -F force=1 > index.html | tee 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # WebXR Hand Input 2 | 3 | The [WebXR Hand Input Specification][this-spec] adds hand input support in WebXR. 4 | Feature lead is Manish Goregaokar ([@Manishearth](https://github.com/Manishearth)). 5 | 6 | ## Taking Part 7 | 8 | 1. Read the [code of conduct][CoC] 9 | 2. See if your issue is being discussed in the [issues][this-spec], or if your idea is being discussed in the [proposals repo][cgproposals]. 10 | 3. We will be publishing the minutes from the bi-weekly calls. 11 | 4. You can also join the working group to participate in these discussions. 12 | 13 | ## Specifications 14 | 15 | * [WebXR Hand Input][this-spec]: Hand input support in WebXR 16 | * [Explainer](explainer.md) 17 | 18 | 19 | ### Related specifications 20 | * [WebXR Device API - Level 1][webxrspec]: Main specification for JavaScript API for accessing VR and AR devices, including sensors and head-mounted displays. 21 | 22 | See also [list of all specifications with detailed status in Working Group and Community Group](https://www.w3.org/immersive-web/list_spec.html). 23 | 24 | ## Relevant Links 25 | 26 | * [Immersive Web Community Group][webxrcg] 27 | * [Immersive Web Working Group Charter][wgcharter] 28 | * [Originating proposal](https://github.com/immersive-web/proposals/issues/48) 29 | 30 | ## Communication 31 | 32 | * [Immersive Web Working Group][webxrwg] 33 | * [Immersive Web Community Group][webxrcg] 34 | * [GitHub issues list](https://github.com/immersive-web/layers/issues) 35 | * [`public-immersive-web` mailing list][publiclist] 36 | 37 | ## Maintainers 38 | 39 | To generate the spec document (`index.html`) from the `index.bs` [Bikeshed][bikeshed] document: 40 | 41 | ```sh 42 | bikeshed spec 43 | ``` 44 | 45 | ## Tests 46 | 47 | For normative changes, a corresponding 48 | [web-platform-tests][wpt] PR is highly appreciated. Typically, 49 | both PRs will be merged at the same time. Note that a test change that contradicts the spec should 50 | not be merged before the corresponding spec change. If testing is not practical, please explain why 51 | and if appropriate [file a web-platform-tests issue][wptissue] 52 | to follow up later. Add the `type:untestable` or `type:missing-coverage` label as appropriate. 53 | 54 | 55 | ## License 56 | 57 | Per the [`LICENSE.md`](LICENSE.md) file: 58 | 59 | > All documents in this Repository are licensed by contributors under the [W3C Software and Document License](https://www.w3.org/Consortium/Legal/copyright-software). 60 | 61 | # Summary 62 | 63 | For more information about this proposal, please read the [explainer](explainer.md) and issues/PRs. 64 | 65 | 66 | [this-spec]: https://immersive-web.github.io/webxr-hand-input 67 | [CoC]: https://immersive-web.github.io/homepage/code-of-conduct.html 68 | [webxrwg]: https://w3.org/immersive-web 69 | [cgproposals]: https://github.com/immersive-web/proposals 70 | [webxrspec]: https://immersive-web.github.io/webxr/ 71 | [webxrcg]: https://www.w3.org/community/immersive-web/ 72 | [wgcharter]: https://www.w3.org/2020/05/immersive-Web-wg-charter.html 73 | [webxrref]: https://immersive-web.github.io/webxr-reference/ 74 | [publiclist]: https://lists.w3.org/Archives/Public/public-immersive-web-wg/ 75 | [bikeshed]: https://github.com/tabatkins/bikeshed 76 | [wpt]: https://github.com/web-platform-tests/wpt 77 | [wptissue]: https://github.com/web-platform-tests/wpt/issues/new 78 | -------------------------------------------------------------------------------- /explainer.md: -------------------------------------------------------------------------------- 1 | # WebXR Device API - Hand Input 2 | 3 | This document describes a design giving developers access to hand-tracking XR systems, building on top of the [WebXR device API](https://immersive-web.github.io/webxr/) 4 | 5 | ## Use cases and scope 6 | 7 | This API primarily exposes the poses of hand skeleton joints. It can be used to render a hand model in VR scenarios, as well perform gesture detection with the hands. It does not provide access to a full hand mesh. 8 | 9 | ## Accessing this API 10 | 11 | This API will only be accessible if a `"hand-tracking"` [XR feature](https://immersive-web.github.io/webxr/#feature-dependencies) is requested. 12 | 13 | This API presents itself as an additional field on `XRInputSource`, `hand`. The `hand` attribute will be non-null if the input source supports hand tracking and the feature has been requested. 14 | 15 | You can determine which hand the input source is associated with by accessing the `handedness` property of `XRInputSource`. 16 | 17 | ```js 18 | navigator.xr.requestSession({optionalFeatures: ["hand-tracking"]}).then(...); 19 | 20 | function renderFrame(session, frame) { 21 | // ... 22 | 23 | for (inputSource of session.inputSources) { 24 | if (inputSource.hand) { 25 | // render a hand model 26 | // perform gesture detection 27 | } 28 | } 29 | } 30 | 31 | 32 | ``` 33 | 34 | ## Hands and joints 35 | 36 | Each hand is made up many bones, connected by _joints_. We name them with their connected bone, for example `index-finger-phalanx-distal` is the joint closer to the wrist connected to the distal phalanx bone of the index finger. The `*-tip` "joints" locate the tips of the fingers. The `wrist` joint is located at the composite joint between the wrist and forearm. 37 | 38 | The joint spaces can be accessed via `XRHand.get()`, for example to access the middle knuckle joint one would use: 39 | 40 | ```js 41 | let joint = inputSource.hand.get("middle-finger-phalanx-distal"); 42 | ``` 43 | 44 | All devices which support hand tracking will support or emulate all joints, so this method will always return a valid object as long as it is supplied with a valid joint name. If a joint is supported but not currently being tracked, the getter will still produce the `XRJointSpace`, but it will return `null` when run through `getPose` (etc). 45 | 46 | Each joint space is an `XRSpace`, with its `-Y` direction pointing perpendicular to the skin, outwards from the palm, and `-Z` direction pointing along their associated bone, away from the wrist. This space will return null poses when the joint loses tracking. 47 | 48 | For `*-tip` joints where there is no associated bone, the `-Z` direction is the same as that for the associated `distal` joint, i.e. the direction is along that of the previous bone. 49 | 50 | 51 | ## Obtaining radii 52 | 53 | If you wish to obtain a radius ("distance from skin") for a joint, instead of using `getPose()`, you can use `getJointPose()` on the joint space. The `radius` can be accessed on the joint pose. 54 | 55 | ```js 56 | let radius = frame.getJointPose(joint, referenceSpace).radius; 57 | ``` 58 | 59 | 60 | ## Displaying hand models using this API 61 | 62 | Ideally, most of this will be handled by an external library or the framework being used by the user. 63 | 64 | A simple skeleton can be displayed as follows: 65 | 66 | ```js 67 | const orderedJoints = [ 68 | ["thumb-metacarpal", "thumb-phalanx-proximal", "thumb-phalanx-distal", "thumb-tip"], 69 | ["index-finger-metacarpal", "index-finger-phalanx-proximal", "index-finger-phalanx-intermediate", "index-finger-phalanx-distal", "index-finger-tip"] 70 | ["middle-finger-metacarpal", "middle-finger-phalanx-proximal", "middle-finger-phalanx-intermediate", "middle-finger-phalanx-distal", "middle-finger-tip"] 71 | ["ring-finger-metacarpal", "ring-finger-phalanx-proximal", "ring-finger-phalanx-intermediate", "ring-finger-phalanx-distal", "ring-finger-tip"] 72 | ["pinky-finger-metacarpal", "pinky-finger-phalanx-proximal", "pinky-finger-phalanx-intermediate", "pinky-finger-phalanx-distal", "pinky-finger-tip"] 73 | ]; 74 | 75 | function renderSkeleton(inputSource, frame, renderer) { 76 | let wrist = inputSource.hand.get("wrist"); 77 | if (!wrist) { 78 | // this code is written to assume that the wrist joint is exposed 79 | return; 80 | } 81 | let wristPose = frame.getJointPose(wrist, renderer.referenceSpace); 82 | renderer.drawSphere(frame, wristPose.transform, wristPose.radius); 83 | for (finger of orderedJoints) { 84 | let previous = wristPose; 85 | for (joint of finger) { 86 | let joint = inputSource.hand.get(joint); 87 | if (joint) { 88 | let pose = frame.getJointPose(joint, renderer.referenceSpace); 89 | drawSphere(frame, pose.transform, pose.radius); 90 | drawCylinder(frame, 91 | /* from */ previous.transform, 92 | /* to */ pose.transform, 93 | /* specify a thinner radius */ pose.radius / 3); 94 | previous = pose; 95 | } 96 | } 97 | } 98 | } 99 | ``` 100 | 101 | ## Hand interaction using this API 102 | 103 | It's useful to be able to use individual fingers for interacting with objects. For example, it's possible to have fingers interacting with spherical buttons in space: 104 | 105 | ```js 106 | const buttons = [ 107 | {position: [1, 0, 0, 1], radius: 0.1, pressed: false, 108 | onpress: function() { ... }, onrelease: function() { ... }}, 109 | // ... 110 | ]; 111 | 112 | function checkInteraction(button, inputSource, frame, renderer) { 113 | let tip = frame.getPose(inputSource.hand.get("index-finger-tip"), renderer.referenceSpace); 114 | let distance = calculateDistance(tip.transform.position, button.position); 115 | if (distance < button.radius) { 116 | if (!button.pressed) { 117 | button.pressed = true; 118 | button.onpress(); 119 | } 120 | } else { 121 | if (button.pressed) { 122 | button.pressed = false; 123 | button.onrelease(); 124 | } 125 | } 126 | } 127 | 128 | function onFrame(frame, renderer) { 129 | // ... 130 | for (button of buttons) { 131 | for (inputSource of frame.session.inputSources) { 132 | checkInteraction(button, inputSource, frame, renderer); 133 | } 134 | } 135 | } 136 | 137 | ``` 138 | 139 | ## Gesture detection using this API 140 | 141 | One can do gesture detection using the position and orientation values of the various fingers. This can get pretty complicated and stateful, but a straightforward example below would be simplistically detecting that the user has made a fist gesture: 142 | 143 | ```js 144 | function checkFistGesture(inputSource, frame, renderer) { 145 | for (finger of [["index-finger-tip", "index-finger-metacarpal"], 146 | ["middle-finger-tip", "middle-finger-metacarpal"], 147 | ["ring-finger-tip", "ring-finger-metacarpal"], 148 | ["pinky-finger-tip", "pinky-finger-metacarpal"]]) { 149 | let tip = finger[0]; 150 | let metacarpal = finger[1]; 151 | let tipPose = frame.getPose(inputSource.hand.get(tip), renderer.referenceSpace); 152 | let metacarpalPose = frame.getPose(inputSource.hand.get(metacarpal), renderer.referenceSpace) 153 | if (calculateDistance(tipPose.position, metacarpalPose.position) > minimumDistance || 154 | !checkOrientation(tipPose.orientation, metacarpalPose.orientation)) { 155 | return false 156 | } 157 | } 158 | return true; 159 | } 160 | 161 | function checkOrientation(tipOrientation, metacarpalOrientation) { 162 | let tipDirection = applyOrientation(tipOrientation, [0, 0, -1]); // -Z axis of tip 163 | let palmDirection = applyOrientation(metacarpalOrientation, [0, -1, 0]) // -Y axis of metacarpal 164 | 165 | if (1 - dotProduct(tipDirection, palmDirection) < minimumDeviation) { 166 | return true; 167 | } else { 168 | return false; 169 | } 170 | } 171 | ``` 172 | 173 | ## Efficiently obtaining hand poses 174 | 175 | Each use of `getPose()` allocates one short-lived `XRPose` object, one `XRRigidTransform` object, and at least one `DOMPointReadOnly` or `Float32Array`. For 25 joints per hand and two hands, this is 150-250 objects created per frame. This can have noticeable performance implications, especially around garbage collection. 176 | 177 | To avoid that, we provide a `fillPoses()` (and `fillJointRadii`) API which can be used to efficiently obtain all the transforms for a hand at once. 178 | 179 | ```js 180 | let poses1 = new Float32Array(16 * 25); 181 | let radii1 = new Float32Array(25); 182 | function onFrame(frame, renderer) { 183 | let hand1 = frame.session.inputSources[0].hand; 184 | frame.fillPoses(hand1.values(), renderer.referenceSpace, poses1); 185 | frame.fillJointRadii(hand1.values(), radii1); 186 | renderer.drawHand(poses1, radii1); 187 | // do something similar for second hand 188 | } 189 | ``` 190 | 191 | ## Privacy and Security Considerations 192 | 193 | The concept of exposing hand input could pose a risk to users’ privacy. For example, data produced by some hand-tracking systems could potentially enable sites to infer users’ gestural behaviors or approximate hand size, make it apparent to sites that a user is missing fingers or parts of fingers, or allow detection of tremors or other medical conditions. 194 | 195 | Implementations are required to employ strategies to mitigate these risks, such as: 196 | - Reducing the precision and sampling rate of data 197 | - Adding noise or rounding data 198 | - Return the same hand geometry/size for all users 199 | - Emulating values for joints if the implementation isn’t capable of detecting them or the user does not have them. 200 | 201 | This specification requires implementations to include sufficient mitigations to protect users’ privacy. 202 | 203 | ## Appendix: Proposed IDL 204 | 205 | ```webidl 206 | partial interface XRInputSource { 207 | readonly attribute XRHand? hand; 208 | } 209 | 210 | partial interface XRFrame { 211 | XRJointPose? getJointPose(XRJointSpace joint, XRSpace relativeTo); 212 | boolean fillJointRadii(sequence jointSpaces, Float32Array radii); 213 | boolean fillPoses(sequence spaces, XRSpace baseSpace, Float32Array transforms); 214 | } 215 | 216 | interface XRJointPose: XRPose { 217 | readonly attribute float? radius; 218 | } 219 | 220 | interface XRJointSpace: XRSpace { 221 | readonly attribute XRHandJoint jointName; 222 | } 223 | 224 | enum XRHandJoint { 225 | "wrist", 226 | 227 | "thumb-metacarpal", 228 | "thumb-phalanx-proximal", 229 | "thumb-phalanx-distal", 230 | "thumb-tip", 231 | 232 | "index-finger-metacarpal", 233 | "index-finger-phalanx-proximal", 234 | "index-finger-phalanx-intermediate", 235 | "index-finger-phalanx-distal", 236 | "index-finger-tip", 237 | 238 | "middle-finger-metacarpal", 239 | "middle-finger-phalanx-proximal", 240 | "middle-finger-phalanx-intermediate", 241 | "middle-finger-phalanx-distal", 242 | "middle-finger-tip", 243 | 244 | "ring-finger-metacarpal", 245 | "ring-finger-phalanx-proximal", 246 | "ring-finger-phalanx-intermediate", 247 | "ring-finger-phalanx-distal", 248 | "ring-finger-tip", 249 | 250 | "pinky-finger-metacarpal", 251 | "pinky-finger-phalanx-proximal", 252 | "pinky-finger-phalanx-intermediate", 253 | "pinky-finger-phalanx-distal", 254 | "pinky-finger-tip" 255 | }; 256 | 257 | interface XRHand { 258 | iterable; 259 | 260 | readonly attribute unsigned long size; 261 | XRJointSpace get(XRHandJoint key); 262 | }; 263 | 264 | 265 | ``` 266 | -------------------------------------------------------------------------------- /favicon-32x32.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/immersive-web/webxr-hand-input/ae350052ea2143b05afa14efda1e11739ac46774/favicon-32x32.png -------------------------------------------------------------------------------- /favicon-96x96.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/immersive-web/webxr-hand-input/ae350052ea2143b05afa14efda1e11739ac46774/favicon-96x96.png -------------------------------------------------------------------------------- /favicon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/immersive-web/webxr-hand-input/ae350052ea2143b05afa14efda1e11739ac46774/favicon.ico -------------------------------------------------------------------------------- /images/hand-layout.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 19 | 21 | 43 | 45 | 46 | 48 | image/svg+xml 49 | 51 | 52 | 53 | 54 | 55 | 60 | 66 | 69 | 74 | 80 | 86 | 87 | 90 | 96 | 0 109 | 110 | 113 | 119 | 1 134 | 135 | 138 | 144 | 2 159 | 160 | 163 | 169 | 3 184 | 190 | 4 205 | 206 | 209 | 215 | 5 230 | 231 | 234 | 240 | 6 255 | 258 | 264 | 7 279 | 282 | 288 | 8 303 | 304 | 305 | 306 | 309 | 315 | 9 330 | 331 | 334 | 337 | 339 | 345 | 10 360 | 361 | 364 | 370 | 11 383 | 384 | 385 | 386 | 389 | 395 | 12 408 | 411 | 417 | 13 430 | 433 | 439 | 14 452 | 453 | 454 | 455 | 458 | 464 | 15 477 | 478 | 481 | 487 | 16 500 | 501 | 504 | 510 | 17 523 | 524 | 527 | 533 | 18 546 | 547 | 550 | 556 | 19 569 | 570 | 573 | 579 | 20 592 | 593 | 596 | 602 | 21 615 | 616 | 619 | 625 | 22 638 | 639 | 642 | 648 | 23 661 | 662 | 665 | 671 | 24 684 | 685 | 696 | 697 | 698 | -------------------------------------------------------------------------------- /index.bs: -------------------------------------------------------------------------------- 1 | 
  2 | Shortname: webxr-hand-input
  3 | Title: WebXR Hand Input Module - Level 1
  4 | Group: immersivewebwg
  5 | Status: ED
  6 | TR: https://www.w3.org/TR/webxr-hand-input-1/
  7 | ED: https://immersive-web.github.io/webxr-hand-input/
  8 | Previous Version: https://www.w3.org/TR/2020/WD-webxr-hand-input-1-20210824/
  9 | Repository: immersive-web/webxr-hand-input
 10 | Level: 1
 11 | Mailing List Archives: https://lists.w3.org/Archives/Public/public-immersive-web-wg/
 12 | 
 13 | !Participate: File an issue (open issues)
 14 | !Participate: Mailing list archive
 15 | !Participate: W3C's #immersive-web IRC
 16 | 
 17 | Editor: Manish Goregaokar 109489, Google [Mozilla until 2020], manishsmail@gmail.com
 18 | 
 19 | Abstract: The WebXR Hand Input module expands the WebXR Device API with the functionality to track articulated hand poses.
 20 | 
 21 | Status Text: This WebXR Augmented Reality Module is designed as a module to be implemented in addition to WebXR Device API, and is originally included in WebXR Device API which was divided into core and modules.
 22 |
23 | 24 | 25 | 26 | 27 | 28 | 81 | 82 | 83 | 
 84 | spec: webxr-1;
 85 |     type: dfn; text: xr device; for: /
 86 |     type: dfn; text: xr device; for: XRSession
 87 |     type: dfn; text: animation frame; for: XRSession
 88 |     type: dfn; text: explicit consent
 89 |     type: dfn; text: native origin
 90 |     type: dfn; text: feature descriptor
 91 |     type: dfn; text: populate the pose
 92 |     type: dfn; text: session; for: XRFrame
 93 |     type: dfn; text: time; for: XRFrame
 94 |     type: dfn; text: session; for: XRSpace
 95 |     type: dfn; text: input profile name; for: XRInputSource
 96 |     type: dfn; text: primary action; for: /
 97 |     type: dfn; text: primary squeeze action; for: /
 98 | spec:infra; type:dfn; text:list
 99 | spec:webxr-ar-module-1; type:enum-value; text:"immersive-ar"
100 | spec:webidl;
101 |     type:dfn; text:value pairs
102 |
103 | 104 | 
105 | spec:html; urlPrefix: https://html.spec.whatwg.org/multipage/
106 |     type: dfn; text: browsing context; url: browsers.html#browsing-context
107 |
108 | 109 | 110 | Introduction {#intro} 111 | ============ 112 | 113 |
114 | On some [=/XR devices=] it is possible to get fully articulated information about the user's hands when they are used as input sources. 115 | 116 | This API exposes the poses of each of the users' hand [=skeleton joints=]. This can be used to do gesture detection or to render a hand model in VR scenarios. 117 | 118 |
119 | 120 | 121 | Initialization {#initialization} 122 | ============== 123 | 124 | If an application wants to view articulated hand pose information during a session, 125 | the session MUST be requested with an appropriate [=feature descriptor=]. The string "hand-tracking" is introduced 126 | by this module as a new valid [=feature descriptor=] for articulated hand tracking. 127 | 128 | The "[=hand-tracking=]" [=feature descriptor=] should only be granted for an {{XRSession}} when its [=XRSession/XR device=] has [=physical hand input sources=] that [=support hand tracking=]. 129 | 130 | The user agent MAY gate support for hand based {{XRInputSource|XRInputSources}} based upon this [=feature descriptor=]. 131 | 132 | NOTE: This means that if an {{XRSession}} does not request the "[=hand-tracking=]" [=feature descriptor=], the user agent may choose to not support input controllers that are hand based. 133 | 134 | Physical Hand Input Sources {#physical-hand} 135 | =========================== 136 | 137 | An {{XRInputSource}} is a physical hand input source if it tracks a physical hand. A [=physical hand input source=] supports hand tracking if it supports reporting the poses of one or more [=skeleton joints=] defined in this specification. 138 | 139 | [=Physical hand input sources=] MUST include the [=XRInputSource/input profile name=] of "generic-hand-select" in their {{XRInputSource/profiles}}. 140 | 141 | For many [=physical hand input sources=], there can be overlap between the gestures used for the [=primary action=] and the squeeze action. For example, a pinch gesture may indicate both a "select" and "squeeze" event, depending on whether you are interacting with nearby or far away objects. Since content may assume that these are independent events, user agents MAY, instead of surfacing the squeeze action as the [=primary squeeze action=], surface it as an additional "grasp button", using an input profile derived from the "generic-hand-select-grasp" profile. 142 | 143 | XRInputSource {#xrinputsource-interface} 144 | ------------- 145 | 146 | 
147 | partial interface XRInputSource {
148 |    [SameObject] readonly attribute XRHand? hand;
149 | };
150 |
151 | 152 | The hand attribute on a [=physical hand input source=] that [=supports hand tracking=] will be an {{XRHand}} object giving access to the underlying hand-tracking capabilities. {{XRInputSource/hand}} will have its [=input source=] set to [=this=]. 153 | 154 | If the {{XRInputSource}} belongs to an {{XRSession}} that has not been requested with the "[=hand-tracking=]" [=feature descriptor=], {{XRInputSource/hand}} MUST be null. 155 | 156 | Skeleton Joints {#skeleton-joints-section} 157 | --------------- 158 | 159 | A [=physical hand input source=] is made up of many skeleton joints. 160 | 161 | A [=skeleton joint=] for a given hand can be uniquely identified by a skeleton joint name, which is an enum of type {{XRHandJoint}}. 162 | 163 | A [=skeleton joint=] may have an associated bone that it is named after and used to orient its -Z axis. The [=associated bone=] of a [=skeleton joint=] is the bone that comes after the joint when moving towards the fingertips. The tip and wrist joints have no [=associated bones=]. 164 | 165 | A [=skeleton joint=] has a radius which is the radius of a sphere placed at its center so that it roughly touches the skin on both sides of the hand. The "tip" [=skeleton joints=] SHOULD have an appropriate nonzero radius so that collisions with the fingertip may work. Implementations MAY offset the origin of the tip joint so that it can have a spherical shape with nonzero radius. 166 | 167 | This list of joints defines the following [=skeleton joints=] and their order: 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 |
[=Skeleton joint=][=Skeleton joint name=]Index
Wrist{{XRHandJoint/wrist}}0
ThumbMetacarpal{{XRHandJoint/thumb-metacarpal}}1
Proximal Phalanx{{XRHandJoint/thumb-phalanx-proximal}}2
Distal Phalanx{{XRHandJoint/thumb-phalanx-distal}}3
Tip{{XRHandJoint/thumb-tip}}4
Index fingerMetacarpal{{XRHandJoint/index-finger-metacarpal}}5
Proximal Phalanx{{XRHandJoint/index-finger-phalanx-proximal}}6
Intermediate Phalanx{{XRHandJoint/index-finger-phalanx-intermediate}}7
Distal Phalanx{{XRHandJoint/index-finger-phalanx-distal}}8
Tip{{XRHandJoint/index-finger-tip}}9
Middle fingerMetacarpal{{XRHandJoint/middle-finger-metacarpal}}10
Proximal Phalanx{{XRHandJoint/middle-finger-phalanx-proximal}}11
Intermediate Phalanx{{XRHandJoint/middle-finger-phalanx-intermediate}}12
Distal Phalanx{{XRHandJoint/middle-finger-phalanx-distal}}13
Tip{{XRHandJoint/middle-finger-tip}}14
Ring fingerMetacarpal{{XRHandJoint/ring-finger-metacarpal}}15
Proximal Phalanx{{XRHandJoint/ring-finger-phalanx-proximal}}16
Intermediate Phalanx{{XRHandJoint/ring-finger-phalanx-intermediate}}17
Distal Phalanx{{XRHandJoint/ring-finger-phalanx-distal}}18
Tip{{XRHandJoint/ring-finger-tip}}19
Little fingerMetacarpal{{XRHandJoint/pinky-finger-metacarpal}}20
Proximal Phalanx{{XRHandJoint/pinky-finger-phalanx-proximal}}21
Intermediate Phalanx{{XRHandJoint/pinky-finger-phalanx-intermediate}}22
Distal Phalanx{{XRHandJoint/pinky-finger-phalanx-distal}}23
Tip{{XRHandJoint/pinky-finger-tip}}24
206 | 207 | Visual aid demonstrating joint layout 208 | 209 | XRHand {#xrhand-interface} 210 | ------ 211 | 212 | 
213 | enum XRHandJoint {
214 |   "wrist",
215 | 
216 |   "thumb-metacarpal",
217 |   "thumb-phalanx-proximal",
218 |   "thumb-phalanx-distal",
219 |   "thumb-tip",
220 | 
221 |   "index-finger-metacarpal",
222 |   "index-finger-phalanx-proximal",
223 |   "index-finger-phalanx-intermediate",
224 |   "index-finger-phalanx-distal",
225 |   "index-finger-tip",
226 | 
227 |   "middle-finger-metacarpal",
228 |   "middle-finger-phalanx-proximal",
229 |   "middle-finger-phalanx-intermediate",
230 |   "middle-finger-phalanx-distal",
231 |   "middle-finger-tip",
232 | 
233 |   "ring-finger-metacarpal",
234 |   "ring-finger-phalanx-proximal",
235 |   "ring-finger-phalanx-intermediate",
236 |   "ring-finger-phalanx-distal",
237 |   "ring-finger-tip",
238 | 
239 |   "pinky-finger-metacarpal",
240 |   "pinky-finger-phalanx-proximal",
241 |   "pinky-finger-phalanx-intermediate",
242 |   "pinky-finger-phalanx-distal",
243 |   "pinky-finger-tip"
244 | };
245 | 
246 | [Exposed=Window]
247 | interface XRHand {
248 |     iterable<XRHandJoint, XRJointSpace>;
249 | 
250 |     readonly attribute unsigned long size;
251 |     XRJointSpace get(XRHandJoint key);
252 | };
253 |
254 | 255 | The {{XRHandJoint}} enum defines the various joints that each {{XRHand}} MUST contain. 256 | 257 | Every {{XRHand}} has an associated input source, which is the [=physical hand input source=] that it tracks. 258 | 259 | NOTE: The handedness property of {{XRInputSource}} describes which hand the XR input source is associated with, if any. 260 | 261 |
262 | Each {{XRHand}} object has a \[[joints]] internal slot, 263 | which is an [=ordered map=] of pairs with the key of type {{XRHandJoint}} and the value of type {{XRJointSpace}}. 264 | 265 | The ordering of the {{[[joints]]}} internal slot is given by the [=list of joints=] under [=skeleton joints=]. 266 | 267 | {{[[joints]]}} MUST NOT change over the course of a session. 268 |
269 | 270 |
271 | The [=value pairs to iterate over=] for an {{XRHand}} object are the list of [=value pairs=] with the key being 272 | the {{XRHandJoint}} and the value being the {{XRJointSpace}} corresponding to that {{XRHandJoint}}, ordered by [=list of joints=] 273 | under [=skeleton joints=]. 274 |
275 | 276 | If an individual device does not support a joint defined in this specification, it MUST emulate it instead. 277 | 278 | The size attribute MUST return the number 25. 279 | 280 |
281 | The get(|jointName|) method when invoked on an {{XRHand}} [=this=] MUST run the following steps: 282 | 283 | 1. Let |joints| be the value of [=this=]'s {{[[joints]]}} internal slot. 284 | 2. Return |joints|[|jointName|]. (This implies returning undefined for unknown |jointName|.) 285 | 286 |
287 | 288 | XRJointSpace {#xrjointspace-interface} 289 | ------------- 290 | 291 | 
292 | [Exposed=Window]
293 | interface XRJointSpace: XRSpace {
294 |   readonly attribute XRHandJoint jointName;
295 | };
296 |
297 | 298 | The [=native origin=] of an {{XRJointSpace}} is the position and orientation of the underlying [=XRJointSpace/joint=]. 299 | 300 | The [=native origin=] of the {{XRJointSpace}} may only be reported when [=native origins=] of all other {{XRJointSpace}}s on the same [=XRJointSpace/hand=] are being reported. When a hand is partially obscured the user agent MUST either emulate the obscured joints, or report null poses for all of the joints. 301 | 302 | Note: This means that when fetching poses you will either get an entire hand or none of it. 303 | 304 | Issue: This by default precludes faithfully exposing polydactyl/oligodactyl hands, however for fingerprinting concerns it will likely need to be a separate opt-in, anyway. See Issue 11 for more details. 305 | 306 | The [=native origin=] has its -Y direction pointing perpendicular to the skin, outwards from the palm, and -Z direction pointing along their associated bone, away from the wrist. 307 | 308 | For tip [=skeleton joints=] where there is no [=associated bone=], the -Z direction is the same as that for the associated distal joint, i.e. the direction is along that of the previous bone. For wrist [=skeleton joints=] the -Z direction SHOULD point roughly towards the center of the palm. 309 | 310 | Every {{XRJointSpace}} has an associated hand, which is the {{XRHand}} that created it. 311 | 312 | jointName returns the joint name of the joint it tracks. 313 | 314 | Every {{XRJointSpace}} has an associated joint, which is the [=skeleton joint=] corresponding to the [=XRJointSpace/jointName=]. 315 | 316 | 317 | Frame Loop {#frame-loop} 318 | ========== 319 | 320 | XRFrame {#xrframe-interface} 321 | ------- 322 | 323 | 
324 | partial interface XRFrame {
325 |     XRJointPose? getJointPose(XRJointSpace joint, XRSpace baseSpace);
326 |     boolean fillJointRadii(sequence<XRJointSpace> jointSpaces, Float32Array radii);
327 | 
328 |     boolean fillPoses(sequence<XRSpace> spaces, XRSpace baseSpace, Float32Array transforms);
329 | };
330 |
331 | 332 |
333 | 334 | The getJointPose(XRJointSpace |joint|, XRSpace |baseSpace|) method provides the pose of |joint| relative to |baseSpace| as an {{XRJointPose}}, at the {{XRFrame}}'s [=XRFrame/time=]. 335 | 336 | When this method is invoked, the user agent MUST run the following steps: 337 | 338 | 1. Let |frame| be [=this=]. 339 | 1. Let |session| be |frame|'s {{XRFrame/session}} object. 340 | 1. If |frame|'s [=active=] boolean is false, throw an {{InvalidStateError}} and abort these steps. 341 | 1. If |baseSpace|'s [=XRSpace/session=] or |joint|'s [=XRSpace/session=] are different from [=this=] {{XRFrame/session}}, throw an {{InvalidStateError}} and abort these steps. 342 | 1. Let |pose| be a [=new=] {{XRJointPose}} object in the [=relevant realm=] of |session|. 343 | 1. [=Populate the pose=] of |joint| in |baseSpace| at the time represented by |frame| into |pose|, with force emulation set to false. 344 | 1. If |pose| is null return null. 345 | 1. Set |pose|'s {{XRJointPose/radius}} to the [=skeleton joint/radius=] of |joint|, emulating it if necessary. 346 | 1. Return |pose|. 347 | 348 |
349 | 350 |
351 | 352 | The fillJointRadii(sequence<XRJointSpace> |jointSpaces|, Float32Array |radii|) method populates |radii| with the radii of the |jointSpaces|, and returns a boolean indicating whether all of the spaces have a valid pose. 353 | 354 | When this method is invoked on an {{XRFrame}} |frame|, the user agent MUST run the following steps: 355 | 356 | 1. Let |frame| be [=this=]. 357 | 1. Let |session| be |frame|'s {{XRFrame/session}} object. 358 | 1. If |frame|'s [=active=] boolean is false, throw an {{InvalidStateError}} and abort these steps. 359 | 1. For each |joint| in the |jointSpaces|: 360 | 1. If |joint|'s [=XRSpace/session=] is different from |session|, throw an {{InvalidStateError}} and abort these steps. 361 | 1. If the length of |jointSpaces| is larger than the number of elements in |radii|, throw a {{TypeError}} and abort these steps. 362 | 1. let |offset| be a new number with the initial value of 0. 363 | 1. Let |allValid| be true. 364 | 1. For each |joint| in the |jointSpaces|: 365 | 1. Set the float value of |radii| at |offset| as follows: 366 |
367 |
If the user agent can determine the poses of all the joints belonging to the |joint|'s [=XRJointSpace/hand=]: 368 |
Set the float value of |radii| at |offset| to that [=skeleton joint/radius=]. 369 |
Otherwise 370 |
Set the float value of |radii| at |offset| to NaN. 371 |
Set |allValid| to false. 372 |
373 | 1. Increase |offset| by 1. 374 | 1. Return |allValid|. 375 | 376 |
377 | 378 | NOTE: if the user agent can't determine the pose of any of the spaces belonging to the same {{XRHand}}, all the spaces of that {{XRHand}} must also not have a pose. 379 | 380 |
381 | 382 | The fillPoses(sequence<XRSpace> |spaces|, XRSpace |baseSpace|, Float32Array |transforms|) method populates |transforms| with the matrices of the poses of the |spaces| relative to the |baseSpace|, and returns a boolean indicating whether all of the spaces have a valid pose. 383 | 384 | When this method is invoked on an {{XRFrame}} |frame|, the user agent MUST run the following steps: 385 | 386 | 1. Let |frame| be [=this=]. 387 | 1. Let |session| be |frame|'s {{XRFrame/session}} object. 388 | 1. If |frame|'s [=active=] boolean is false, throw an {{InvalidStateError}} and abort these steps. 389 | 1. For each |space| in the |spaces| sequence: 390 | 1. If |space|'s [=XRSpace/session=] is different from |session|, throw an {{InvalidStateError}} and abort these steps. 391 | 1. If |baseSpace|'s [=XRSpace/session=] is different from |session|, throw an {{InvalidStateError}} and abort these steps. 392 | 1. If the length of |spaces| multiplied by 16 is larger than the number of elements in |transforms|, throw a {{TypeError}} and abort these steps. 393 | 1. let |offset| be a new number with the initial value of 0. 394 | 1. Initialize |pose| as follows: 395 |
396 |
If {{XRFrame/fillPoses()}} was called previously, the user agent MAY: 397 |
Let |pose| be the same object as used by an earlier call. 398 |
Otherwise 399 |
Let |pose| be a [=new=] {{XRPose}} object in the [=relevant realm=] of |session|. 400 |
401 | 1. Let |allValid| be true. 402 | 1. For each |space| in the |spaces| sequence: 403 | 1. [=Populate the pose=] of |space| in |baseSpace| at the time represented by |frame| into |pose|. 404 | 1. If |pose| is null, perform the following steps: 405 | 1. Set 16 consecutive elements of the |transforms| array starting at |offset| to NaN. 406 | 1. Set |allValid| to false. 407 | 1. If |pose| is not null, copy all elements from |pose|'s {{XRRigidTransform/matrix}} member to the |transforms| array starting at |offset|. 408 | 1. Increase |offset| by 16. 409 | 1. Return |allValid|. 410 | 411 |
412 | 413 | NOTE: if any of the spaces belonging to the same {{XRHand}} return null when [=Populate the pose|populating the pose=], all the spaces of that {{XRHand}} must also return null when [=Populate the pose|populating the pose=] 414 | 415 | XRJointPose {#xrjointpose-interface} 416 | ----------- 417 | 418 | An {{XRJointPose}} is an {{XRPose}} with additional information about the size of the [=skeleton joint=] it represents. 419 | 420 | 
421 | [Exposed=Window]
422 | interface XRJointPose: XRPose {
423 |     readonly attribute float radius;
424 | };
425 |
426 | 427 | The radius attribute returns the [=skeleton joint/radius=] of the [=skeleton joint=] in meters. 428 | 429 | The user-agent MUST set {{XRJointPose/radius}} to an emulated value if the [=/XR device=] does not have the capability of determining this value, either in general or in the current [=XRSession/animation frame=] (e.g. when the [=skeleton joint=] is partially obscured). 430 | 431 | Privacy & Security Considerations {#privacy-security} 432 | ================================= 433 | The WebXR Hand Input API is a powerful feature that carries significant privacy risks. 434 | 435 | Since this feature returns new sensor data, the User Agent MUST ask for [=explicit consent=] from the user at session creation time. 436 | 437 | Data returned from this API, MUST NOT be so specific that one can detect individual users. 438 | If the underlying hardware returns data that is too precise, the User Agent MUST anonymize this data 439 | before revealing it through the WebXR Hand Input API. 440 | 441 | This API MUST only be supported in XRSessions created with XRSessionMode of {{XRSessionMode/"immersive-vr"}} 442 | or {{XRSessionMode/"immersive-ar"}}. {{XRSessionMode/"inline"}} sessions MUST not support this API. 443 | 444 |
445 | When anonymizing the hands data, the UA can follow these guidelines: 446 | * Noising is discouraged in favour of rounding. 447 | * If the UA uses rounding, each joint must not be rounded independently. Instead the correct way to round is to map each hand to a static hand-model. 448 | * If noising, the noised data must not reveal any information over time: 449 | - Each new WebXR session in the same [=browsing context=] must use the same noise to make sure that the data cannot be de-noised by creating multiple sessions. 450 | - Each new [=browsing context=] must use a different noise vector. 451 | - Any seed used to initialize the noise must not be predictable. 452 | * Anonymization must be done in a trusted environment. 453 | 454 |
455 | 456 | 457 |

458 | Changes

459 | 460 |

461 | Changes from the First Public Working Draft 22 October 2020

462 | 463 | - Mention grasp profile (GitHub #68) 464 | - Change from constants to enums + change XRHand into a map (GitHub #71) 465 | - Added additional clarification in security section (GitHub #87) 466 | - Marked hand as sameobject + added a clarifying note (GitHub #93) 467 | - Nonzero radius for tip (GitHub #111) 468 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "webxr-ar-module", 3 | "description": "WebXR Augmented Reality Module", 4 | "version": "0.0.1", 5 | "devDependencies": { 6 | "browser-sync": "^2.18.8" 7 | }, 8 | "scripts": { 9 | "start": "npm run dev", 10 | "dev": "cross-env NODE_ENV=development npm run server", 11 | "prod": "cross-env NODE_ENV=production npm run server", 12 | "server": "browser-sync start --config scripts/browsersync-config.js", 13 | "build": "make" 14 | }, 15 | "dependencies": { 16 | "cross-env": "^4.0.0" 17 | } 18 | } 19 | -------------------------------------------------------------------------------- /w3c.json: -------------------------------------------------------------------------------- 1 | { 2 | "group": 109735 3 | , "contacts": ["dontcallmedom", "himorin"] 4 | , "shortName": "webxr-hand-input" 5 | , "repo-type": "rec-track" 6 | } --------------------------------------------------------------------------------