├── README.md
├── index.bs
├── index.html
└── media
├── brow_lowerer_left.png
├── brow_lowerer_right.png
├── cheek_puff_left.png
├── cheek_puff_right.png
├── cheek_raiser_left.png
├── cheek_raiser_right.png
├── cheek_suck_left.png
├── cheek_suck_right.png
├── chin_raiser_bottom.png
├── chin_raiser_top.png
├── dimpler_left.png
├── dimpler_right.png
├── eyes_closed_left.png
├── eyes_closed_right.png
├── eyes_look_down_left.png
├── eyes_look_down_right.png
├── eyes_look_left_left.png
├── eyes_look_left_right.png
├── eyes_look_right_left.png
├── eyes_look_right_right.png
├── eyes_look_up_left.png
├── eyes_look_up_right.png
├── inner_brow_raiser_left.png
├── inner_brow_raiser_right.png
├── jaw_drop.png
├── jaw_sideways_left.png
├── jaw_sideways_right.png
├── jaw_thrust.png
├── lid_tightener_left.png
├── lid_tightener_right.png
├── lip_corner_depressor_left.png
├── lip_corner_depressor_right.png
├── lip_corner_puller_left.png
├── lip_corner_puller_right.png
├── lip_funneler_left_bottom.png
├── lip_funneler_left_top.png
├── lip_funneler_right_bottom.png
├── lip_funneler_right_top.png
├── lip_pressor_left.png
├── lip_pressor_right.png
├── lip_pucker_left.png
├── lip_pucker_right.png
├── lip_stretcher_left.png
├── lip_stretcher_right.png
├── lip_suck_left_bottom.png
├── lip_suck_left_top.png
├── lip_suck_right_bottom.png
├── lip_suck_right_top.png
├── lip_tightener_left.png
├── lip_tightener_right.png
├── lips_toward.png
├── lips_toward_right_top.png
├── lower_lip_depressor_left.png
├── lower_lip_depressor_right.png
├── mouth_left.png
├── mouth_right.png
├── neutral.png
├── nose_wrinkler_left.png
├── nose_wrinkler_right.png
├── outer_brow_raiser_left.png
├── outer_brow_raiser_right.png
├── upper_lid_raiser_left.png
├── upper_lid_raiser_right.png
├── upper_lip_raiser_left.png
├── upper_lip_raiser_leftt.png
└── upper_lip_raiser_right.png
/README.md:
--------------------------------------------------------------------------------
1 | # WebXR Expression Tracking
2 |
3 | ## Introduction
4 | Expression tracking is a technology that reports a set of expressions of the current user. The expressions report how much a portion of the face or an eye move. They don't report actual positions or other real world data.
5 |
6 | ## Use cases
7 | The main use case is to render the user's face and eyes using an avatar for increased social presence.
8 | This technology will NOT:
9 | * allow rendering of the user's actual face
10 | * give precise information where the user is looking
11 | * allow virtual interaction with the user's face (ie Makeup, hats, etc)
12 |
13 | ## Proposed API shape
14 | This API will define an extensive set of expression and will on a per frame basis, report which ones were detected and how strong they are.
15 | For instance, if the device's eye tracking is functional and the user looks left, it will return that both eyes are looking left.
16 |
17 | ```webidl
18 | enum XRExpression {
19 | "brow_lowerer_left",
20 | "brow_lowerer_right",
21 | "cheek_puff_left",
22 | "cheek_puff_right",
23 | "cheek_raiser_left",
24 | "cheek_raiser_right",
25 | "cheek_suck_left",
26 | "cheek_suck_right",
27 | "chin_raiser_bottom",
28 | "chin_raiser_top",
29 | "dimpler_left",
30 | "dimpler_right",
31 | "eyes_closed_left",
32 | "eyes_closed_right",
33 | "eyes_look_down_left",
34 | "eyes_look_down_right",
35 | "eyes_look_left_left",
36 | "eyes_look_left_right",
37 | "eyes_look_right_left",
38 | "eyes_look_right_right",
39 | "eyes_look_up_left",
40 | "eyes_look_up_right",
41 | "inner_brow_raiser_left",
42 | "inner_brow_raiser_right",
43 | "jaw_drop",
44 | "jaw_sideways_left",
45 | "jaw_sideways_right",
46 | "jaw_thrust",
47 | "lid_tightener_left",
48 | "lid_tightener_right",
49 | "lip_corner_depressor_left",
50 | "lip_corner_depressor_right",
51 | "lip_corner_puller_left",
52 | "lip_corner_puller_right",
53 | "lip_funneler_left_bottom",
54 | "lip_funneler_left_top",
55 | "lip_funneler_right_bottom",
56 | "lip_funneler_right_top",
57 | "lip_pressor_left",
58 | "lip_pressor_right",
59 | "lip_pucker_left",
60 | "lip_pucker_right",
61 | "lip_stretcher_left",
62 | "lip_stretcher_right",
63 | "lip_suck_left_bottom",
64 | "lip_suck_left_top",
65 | "lip_suck_right_bottom",
66 | "lip_suck_right_top",
67 | "lip_tightener_left",
68 | "lip_tightener_right",
69 | "lips_toward",
70 | "lower_lip_depressor_left",
71 | "lower_lip_depressor_right",
72 | "mouth_left",
73 | "mouth_right",
74 | "nose_wrinkler_left",
75 | "nose_wrinkler_right",
76 | "outer_brow_raiser_left",
77 | "outer_brow_raiser_right",
78 | "upper_lid_raiser_left",
79 | "upper_lid_raiser_right",
80 | "upper_lip_raiser_left",
81 | "upper_lip_raiser_right"
82 | };
83 |
84 | interface XRExpressions {
85 | iterable;
86 |
87 | readonly attribute unsigned long size;
88 | float get(XRExpression key);
89 | };
90 |
91 | partial interface XRFrame {
92 | readonly attribute XRExpressions? expressions;
93 | }
94 |
95 | ```
96 |
97 | ## Background/Rationale
98 | This API is inspired by the OpenXR face extensions.
99 |
100 | ## Security/privacy implications
101 | Face tracking exposes sensitive sensor data.
102 | Because of this, the user agent must ask the user for their permission when a session is asking for this feature (much like WebXR and WebXR hand tracking).
103 | In addition, sensitive values such as eye position must be rounded.
104 |
--------------------------------------------------------------------------------
/index.bs:
--------------------------------------------------------------------------------
1 |
2 | Shortname: webxr-expression-tracking
3 | Title: WebXR Expression Tracking - Level 1
4 | Group: immersivewebwg
5 | Status: UD
6 |
7 | ED: https://cabanier.github.io/webxr-expression-tracking-1/
8 | Repository: cabanier/webxr-expression-tracking-1
9 | Level: 1
10 | Mailing List Archives: https://lists.w3.org/Archives/Public/public-immersive-web-wg/
11 |
12 | Editor: Rik Cabanier, Meta, cabanier@meta.com
13 |
14 | Abstract: The WebXR Expression Tracking module expands the WebXR Device API with the functionality to track expressions.
15 |
16 | Status Text: This WebXR Module is designed as a module to be implemented in addition to WebXR Device API, and is originally included in WebXR Device API which was divided into core and modules.
17 |
109 |
110 | Introduction {#intro}
111 | ============
112 |
113 |
114 | This API exposes the expressions of the user's face and the position of their eyes. This can be used to render a more immersive avatar.
115 |
116 |
117 |
118 | Initialization {#initialization}
119 | ==============
120 |
121 | If an application wants to get expressions during an immersive session, the session MUST be requested with an appropriate [=feature descriptor=]. The string "expression-tracking" is introduced
122 | by this module as a new valid [=feature descriptor=] for face expressions.
123 |
124 | The "[=expression-tracking=]" [=feature descriptor=] should only be granted for an {{XRSession}} when its [=XRSession/XR device=] has sensor data to support reporting of expressions.
125 |
126 | Access to expressions {#expression-access}
127 | =====================
128 |
129 |
130 | Expressions {#expressions}
131 | -----------
132 |
133 | An expression MUST be one of the {{XRExpression}} types.
134 | A user agent MAY support a subset of {{XRExpression}} types and this subset is allowed to change during an {{XRSession}}
135 |
136 | Following is the list of expressions and their order:
137 |
607 |
608 | The {{XRExpression}} enum defines the various expressions that could be reported by the user agent.
609 |
610 |
611 | Each {{XRExpressions}} object has a \[[expressions]] internal slot,
612 | which is an [=ordered map=] of pairs with the key of type {{XRExpression}} and the value of type float. Each {{XRExpression}} MUST have a value between 0 and 1 with 0 being undected (or rest pose) and 1 the maximum expression.
613 |
614 | The ordering of the {{[[expressions]]}} internal slot is given by the [=list of expressions=].
615 |
616 | {{[[expressions]]}} MAY change over the course of a session but MUST stay the same during the {{XRFrame}}.
617 |
618 |
619 |
620 | The [=value pairs to iterate over=] for an {{XRExpressions}} object are the list of [=value pairs=] with the key being
621 | the {{XRExpression}} and the value being the float corresponding to that {{XRExpression}}, ordered by [=list of expressions=].
622 |
623 |
624 | If the user agent does not support or can report an expression defined, it MUST NOT be reported.
625 |
626 | Frame Loop {#frame-loop}
627 | ==========
628 |
629 | XRFrame {#xrframe-interface}
630 | -------
631 |
632 |
637 |
638 | Privacy & Security Considerations {#privacy-security}
639 | =================================
640 | The WebXR Expression Tracking API is a powerful feature that carries significant privacy risks.
641 |
642 | Since this feature returns new sensor data, the User Agent MUST ask for [=explicit consent=] from the user at session creation time.
643 |
644 | Data returned from this API, MUST NOT be so specific that one can detect individual users.
645 | If the underlying hardware returns data that is too precise, the User Agent MUST anonymize this data
646 | before revealing it through the WebXR Expression Tracking API.
647 |
648 | This API MUST only be supported in XRSessions created with XRSessionMode of {{XRSessionMode/"immersive-vr"}}
649 | or {{XRSessionMode/"immersive-ar"}}. {{XRSessionMode/"inline"}} sessions MUST not support this API.
650 |
651 |
652 | When anonymizing the expression data, the UA can follow these guidelines:
653 | * Noising is discouraged in favour of rounding.
654 | * If the UA uses rounding, each expression must not be rounded independently.
655 | * If noising, the noised data must not reveal any information over time:
656 | - Each new WebXR session in the same [=browsing context=] must use the same noise to make sure that the data cannot be de-noised by creating multiple sessions.
657 | - Each new [=browsing context=] must use a different noise vector.
658 | - Any seed used to initialize the noise must not be predictable.
659 | * Anonymization must be done in a trusted environment.
660 |
661 |
The WebXR Expression Tracking module expands the WebXR Device API with the functionality to track expressions.
671 |
672 |
Status of this document
673 |
674 |
675 |
This WebXR Module is designed as a module to be implemented in addition to WebXR Device API, and is originally included in WebXR Device API which was divided into core and modules.
676 |
677 |
678 |
717 |
718 |
1. Introduction
719 | This API exposes the expressions of the user’s face and the position of their eyes. This can be used to render a more immersive avatar.
720 |
2. Initialization
721 |
If an application wants to get expressions during an immersive session, the session MUST be requested with an appropriate feature descriptor. The string "expression-tracking" is introduced
722 | by this module as a new valid feature descriptor for face expressions.
An expression MUST be one of the XRExpression types.
727 | A user agent MAY support a subset of XRExpression types and this subset is allowed to change during an XRSession
728 |
Following is the list of expressions and their order:
The XRExpression enum defines the various expressions that could be reported by the user agent.
1063 |
1064 | Each XRExpressions object has a [[expressions]] internal slot,
1065 | which is an ordered map of pairs with the key of type XRExpression and the value of type float. Each XRExpression MUST have a value between 0 and 1 with 0 being undected (or rest pose) and 1 the maximum expression.
1066 |
1079 | The WebXR Expression Tracking API is a powerful feature that carries significant privacy risks.
1080 |
Since this feature returns new sensor data, the User Agent MUST ask for explicit consent from the user at session creation time.
1081 |
Data returned from this API, MUST NOT be so specific that one can detect individual users.
1082 | If the underlying hardware returns data that is too precise, the User Agent MUST anonymize this data
1083 | before revealing it through the WebXR Expression Tracking API.
1084 |
This API MUST only be supported in XRSessions created with XRSessionMode of "immersive-vr" or "immersive-ar". "inline" sessions MUST not support this API.
1085 |
1086 | When anonymizing the expression data, the UA can follow these guidelines:
1087 |
1088 |
1089 |
Noising is discouraged in favour of rounding.
1090 |
1091 |
If the UA uses rounding, each expression must not be rounded independently.
1092 |
1093 |
If noising, the noised data must not reveal any information over time:
1094 |
1095 |
1096 |
Each new WebXR session in the same browsing context must use the same noise to make sure that the data cannot be de-noised by creating multiple sessions.
Any seed used to initialize the noise must not be predictable.
1101 |
1102 |
1103 |
Anonymization must be done in a trusted environment.
1104 |
1105 |
1106 |
1107 |
1108 |
Conformance
1109 |
Document conventions
1110 |
Conformance requirements are expressed
1111 | with a combination of descriptive assertions
1112 | and RFC 2119 terminology.
1113 | The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL”
1114 | in the normative parts of this document
1115 | are to be interpreted as described in RFC 2119.
1116 | However, for readability,
1117 | these words do not appear in all uppercase letters in this specification.
1118 |
All of the text of this specification is normative
1119 | except sections explicitly marked as non-normative, examples, and notes. [RFC2119]
1120 |
Examples in this specification are introduced with the words “for example”
1121 | or are set apart from the normative text
1122 | with class="example",
1123 | like this:
1124 |
1125 |
1126 |
This is an example of an informative example.
1127 |
1128 |
Informative notes begin with the word “Note”
1129 | and are set apart from the normative text
1130 | with class="note",
1131 | like this:
1132 |
Note, this is an informative note.
1133 |
Conformant Algorithms
1134 |
Requirements phrased in the imperative as part of algorithms
1135 | (such as "strip any leading space characters"
1136 | or "return false and abort these steps")
1137 | are to be interpreted with the meaning of the key word
1138 | ("must", "should", "may", etc)
1139 | used in introducing the algorithm.
1140 |
Conformance requirements phrased as algorithms or specific steps
1141 | can be implemented in any manner,
1142 | so long as the end result is equivalent.
1143 | In particular, the algorithms defined in this specification
1144 | are intended to be easy to understand
1145 | and are not intended to be performant.
1146 | Implementers are encouraged to optimize.
799 | 
803 | 
807 | 
811 | 
815 | 
819 | 
823 | 
827 | 
831 | 
835 | 
839 | 
843 | 
847 | 
851 | 
855 | 
859 | 
863 | 
867 | 
871 | 
875 | 
879 | 
883 | 
887 | 
891 | 
895 | 
899 | 
903 | 
907 | 
911 | 
915 | 
919 | 
923 | 
927 | 
931 | 
935 | 
939 | 
943 | 
947 | 
951 | 
955 | 
959 | 
963 | 
967 | 
971 | 
975 | 
979 | 
983 | 
987 | 
991 | 
995 | 
999 | 
1003 | 
1007 | 
1011 | 
1015 | 
1019 | 
1023 | 
1027 | 
1031 | 
1035 | 
1039 | 
1043 | 
1047 | 
1051 |