├── LICENSE
├── README.md
├── app
    ├── .gitignore
    ├── build
    │   └── icon.png
    ├── global.d.ts
    ├── images
    │   ├── icon1.png
    │   └── icon2.png
    ├── index.html
    ├── main.js
    ├── package-lock.json
    ├── package.json
    ├── preload.js
    ├── rtcfilesystem-client.js
    ├── rtcfilesystem-server.js
    ├── style.css
    ├── tsconfig.json
    └── webrtc-rdp.js
├── design.png
├── index.html
├── screenshot-xr.png
└── webxr
    ├── aframe-rdp.js
    └── index.html


/LICENSE:
--------------------------------------------------------------------------------
 1 | MIT License
 2 | 
 3 | Copyright (c) 2022 Kosuke Kawahira
 4 | 
 5 | Permission is hereby granted, free of charge, to any person obtaining a copy
 6 | of this software and associated documentation files (the "Software"), to deal
 7 | in the Software without restriction, including without limitation the rights
 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 | 
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 | 
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | # WebRTC + WebXR Remote Desktop
  2 | 
  3 | WebRTCを使ったブラウザ上で動作するリモートデスクトップです．
  4 | 
  5 | ![Screenshot](screenshot-xr.png)
  6 | 
  7 | 最近の Chrome や Edge で動作します．通常のWeb UIの他にWebXRを利用することでVR/AR空間内にデスクトップを表示することができます．
  8 | また、タスクトレイに常駐するアプリ版もあります．
  9 | 
 10 | Demo URL:
 11 | - https://binzume.github.io/webrtc-rdp/ (Web)
 12 | - https://binzume.github.io/vr-workspace/#app:app-webrtc-rdp (WebXR)
 13 | 
 14 | 
 15 | ## Usage
 16 | 
 17 | 無駄に色々な構成で動作しますが、ホスト側のPCにElectron App版をインストールして、ブラウザからデスクトップにアクセスするのが一番実用的です。
 18 | 
 19 | 1. ブラウザで https://binzume.github.io/webrtc-rdp/ にアクセスするか，または下記のアプリを起動してください
 20 | 2. ホスト・ゲスト：片方のブラウザでPINを生成し，もう一方のブラウザでそのPINを入力してください (初回のみ)
 21 | 3. ホスト側：「Share Desktop」または「Share Camera」ボタンで共有したいストリームを選択してください (ブラウザ版のみ)
 22 | 4. ゲスト側：「Open Remote Desktop」リンクをクリックすると，相手側のデスクトップに接続します．複数のストリームがある場合は選択画面が表示されます
 23 | 
 24 | - [WebXR](https://binzume.github.io/vr-workspace/#app:app-webrtc-rdp) リンクから VR モードでPCのデスクトップに接続できます (Oculus Quest用)
 25 | - 最低限の実装なので，本格利用する場合は色々いじってください．
 26 | 
 27 | | Features  | Web         | WebXR | Electron App |
 28 | |-----------|-------------|-------|--------------|
 29 | | Screen    | send/recv   | recv  | send/recv    |
 30 | | Mouse/Kbd | send/(recv) | send  | send/recv    |
 31 | | File      | send/(recv) | recv  | send         |
 32 | 
 33 | ### Mouse/Keyboard
 34 | 
 35 | マウスやキーボードの操作をしたい場合は，下記のElectron App版を使ってください．
 36 | 
 37 | どうしてもブラウザ経由でマウスを動かしたい場合は，ホスト側のPCで https://github.com/binzume/inputproxy を起動し表示されたURLをフォームに入力することで使えるようになります．
 38 | 
 39 | ### Share files
 40 | 
 41 | ホスト側の画面にファイルやディレクトリをドラッグ＆ドロップすると共有されます．
 42 | 
 43 | - WebXR版のクライアントでデスクトップに接続すると `Storage` に追加されます
 44 | - VRではないWeb UIは別リポジトリの https://github.com/binzume/webrtcfs-web にあります
 45 | 
 46 | ## Electron App
 47 | 
 48 | Chromeを起動していなくても単体で動くアプリケーションです．マウスやキーボードも使えます．
 49 | 
 50 | [Releases](https://github.com/binzume/webrtc-rdp/releases/latest)ページからWindows用のインストーラがダウンロードできます．
 51 | 
 52 | インストールせずに利用する場合や開発時は以下のように起動してください．
 53 | 
 54 | ```
 55 | git clone https://github.com/binzume/webrtc-rdp.git
 56 | cd webrtc-rdp/app
 57 | npm install
 58 | npx electron-rebuild
 59 | npm start
 60 | ```
 61 | 
 62 | Node.jsのネイティブモジュールのビルドができる環境が必要です．
 63 | `npm run build-win` で実行ファイルをビルドできます．
 64 | MacOSやLinuxでも動くように実装していますが，動作確認はWindowsのみでしています．
 65 | 
 66 | # Design
 67 | 
 68 | ![Design](design.png)
 69 | 
 70 | ## WebRTC
 71 | 
 72 | - WebRTC Signaling Serverは[OpenAyame/ayame](https://github.com/OpenAyame/ayame)を使います
 73 | - デモの実装では[Ayame Labo](https://ayame-labo.shiguredo.jp/)に接続します．本格利用する場合は自分でAyameを動かしたほうが良いです
 74 | 
 75 | ## VR
 76 | 
 77 | - [A-Frame](https://aframe.io/)を使っています．
 78 | - [単体](https://binzume.github.io/webrtc-rdp/webxr/)でも使えますが，[binzume/vr-workspace](https://github.com/binzume/vr-workspace)内のアプリとして読み込む前提の作りになっています
 79 | - VRモードは Oculus Quest 2 の Oculus Browser で動作確認しています．
 80 | 
 81 | Oculus Touchコントローラーのボタン割当:
 82 | 
 83 | - Trigger: 左クリック
 84 | - Aボタン: 中クリック
 85 | - Bボタン: 右クリック
 86 | - Grip＆ドラッグ: デスクトップ内のウインドウを分離して表示 (ホストがElectron Appの場合のみ)
 87 | 
 88 | ## Security
 89 | 
 90 | - P2Pなので，同じネットワーク内で使う場合は共有している映像や音声などはインターネットを経由しません．
 91 | - デモの実装ではAyame Laboを使って接続します．セキュアな接続が必要な場合は自分の環境でAyameを起動して使ってください．
 92 | - 接続にAyame Laboを使っている場合，何らかの理由でRoomIdが漏れると他者が接続できる可能性があるので，接続を待機した状態で放置しないでください．
 93 | - RoomIdはPINの交換時にランダムな文字列から生成して共有します．
 94 | - ファイルの送受信は、PIN確認時に共有したtokenと接続時のDTLSのfingerprintを使って認証するので少しセキュアです(気休め程度)
 95 | 
 96 | ## TODO
 97 | 
 98 | - クリップボード共有機能
 99 | - UIを分かりやすくする
100 | 
101 | ## License
102 | 
103 | MIT
104 | 


--------------------------------------------------------------------------------
/app/.gitignore:
--------------------------------------------------------------------------------
1 | node_modules
2 | dist/
3 | 


--------------------------------------------------------------------------------
/app/build/icon.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/binzume/webrtc-rdp/9904ba09c1bbdbf777cae74b82e3bedc89329adb/app/build/icon.png


--------------------------------------------------------------------------------
/app/global.d.ts:
--------------------------------------------------------------------------------
 1 | 
 2 | declare var Ayame: typeof import('@open-ayame/ayame-web-sdk')
 3 | 
 4 | declare type KeyAction = { target: { id: string, name?: string, dispaly_id?: string }, action: string, key: string, modifiers: string[] };
 5 | declare type MouseAction = { target: { id: string, name?: string, dispaly_id?: string }, action: string, button: number, x: number, y: number };
 6 | 
 7 | declare var RDP: {
 8 |     getDisplayStreams(types: string[]): Promise<{ id: string, name: string, dispaly_id: string }[]>
 9 |     sendMouse(mouse: MouseAction): Promise<void>
10 |     sendKey(key: KeyAction): Promise<void>
11 |     streamFromPoint(params: { target: any, x: number, y: number }): Promise<any>
12 | }
13 | 
14 | declare interface StreamSpec {
15 |     id: string
16 |     name: string
17 |     hasAudio?: boolean
18 | }
19 | 
20 | declare interface ConnectionInfo {
21 |     id: number
22 |     name: string
23 |     conn: PublisherConnection
24 |     permanent: boolean
25 | }
26 | 
27 | declare interface StreamProvider {
28 |     startStream: ((cm: ConnectionManager, spec: StreamSpec, permanent: boolean) => Promise<ConnectionInfo>)
29 |     getStreams?: (() => Promise<StreamSpec[]>)
30 | }
31 | 
32 | declare interface DataChannelInfo {
33 |     onmessage?: ((ch: RTCDataChannel, ev: MessageEvent) => void)
34 |     onopen?: ((ch: RTCDataChannel, ev: Event) => void)
35 |     onclose?: ((ch: RTCDataChannel, ev: Event) => void)
36 |     ch?: RTCDataChannel | null
37 | }
38 | 
39 | declare interface DeviceSettings {
40 |     name?: string
41 |     roomId: string
42 |     publishRoomId?: string | null
43 |     localToken?: string,
44 |     signalingKey: string | null
45 |     userAgent: string
46 |     token: string
47 |     services?: string[]
48 | }
49 | 
50 | 
51 | declare interface FileInfo {
52 |     type: string;
53 |     name: string;
54 |     size: number;
55 |     path: string;
56 |     updatedTime: number;
57 |     tags?: string[];
58 |     thumbnail?: { [k: string]: any };
59 |     remove?(): any;
60 |     [k: string]: any;
61 | }
62 | 
63 | declare interface FilesResult {
64 |     name?: string;
65 |     items: FileInfo[];
66 |     next: any;
67 |     more?: boolean;
68 | }
69 | 
70 | declare interface Folder {
71 |     getFiles(offset: any, limit: number, options: object, signal: AbortSignal): Promise<FilesResult>;
72 | }
73 | 
74 | declare interface FolderResolver {
75 |     getFolder(path: string, prefix?: string): Folder;
76 |     parsePath(path: string): string[][];
77 | }
78 | 
79 | declare var storageAccessors: Record<string, FolderResolver & { name: string, [k: string]: any; }> | undefined;
80 | 


--------------------------------------------------------------------------------
/app/images/icon1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/binzume/webrtc-rdp/9904ba09c1bbdbf777cae74b82e3bedc89329adb/app/images/icon1.png


--------------------------------------------------------------------------------
/app/images/icon2.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/binzume/webrtc-rdp/9904ba09c1bbdbf777cae74b82e3bedc89329adb/app/images/icon2.png


--------------------------------------------------------------------------------
/app/index.html:
--------------------------------------------------------------------------------
 1 | <html>
 2 | 
 3 | <head>
 4 |   <link rel="stylesheet" type="text/css" href="style.css" />
 5 |   <title>WebRTC Remote Desktop</title>
 6 |   <script src="node_modules/@open-ayame/ayame-web-sdk/dist/ayame.min.js"></script>
 7 |   <script src="rtcfilesystem-server.js?20220520"></script>
 8 |   <script src="webrtc-rdp.js?20220520"></script>
 9 | </head>
10 | 
11 | <body>
12 |   <h1 class="noplayer browser">WebRTC Remote Desktop</h1>
13 | 
14 |   <div id="content" class="noplayer">
15 |   <div id="pairing" class="block" style="display:none;">
16 |     <h2>Add new device</h2>
17 |     <button id="generatePin">Generate PIN</button> / <button id="inputPin">Input PIN</button>
18 |     <div id="pinDisplayBox" style="display:none;">
19 |       <div class="pin">
20 |         PIN: <span id="pin" class="selectable"></span>
21 |       </div>
22 |       <select id="modeSelect">
23 |         <option value="twoway" selected>two-way</option>
24 |         <option value="host">host</option>
25 |         <option value="guest">guest</option>
26 |       </select><br />
27 |       Waiting.... Visit this page on peer device, and input the PIN.
28 |     </div>
29 |     <form id="pinInputBox">
30 |       <div class="pin">
31 |         PIN:<input type="number" inputmode="numeric" maxlength="8" id="pinInput"
32 |           style="width:200pt;height:50pt;" /><button type='submit' style="width:80pt;height:50pt;">OK</button>
33 |       </div>
34 |       Visit this page on peer device, and generate PIN.
35 |     </form>
36 |   </div>
37 | 
38 |   <div id="publishOrPlay" style="display:none;text-align: center">
39 |     <div style="text-align:right;">
40 |       <button id="addDeviceButton">Add Device</button>
41 |       <button class="browser" id="clearSettingsButton">Reset Settings</button>
42 |     </div>
43 |     <div id="devices"></div>
44 |     <ul id="files" class="streamlist"></ul>
45 |     <div class="browser block">
46 |       Input proxy: <input id="inputProxyUrl" />
47 |       <button id="connectInputButton">Connect</button> <br />
48 |       If you want to cntrol mouse, set <a href="https://github.com/binzume/inputproxy" target="_blank">inputproxy</a>
49 |       url.
50 |     </div>
51 |   </div>
52 |   </div>
53 | 
54 |   <div id="player" style="width: 100%; height: 100vh; position: relative; overflow: hidden;">
55 |     <div id="player-menu">
56 |       <button id="closePlayerButton">Back</button>
57 |       <button id="playButton">Reset</button>
58 |       <button id="toggleShiftButton" class='onoff'>Shift</button>
59 |       <button id="toggleControlButton" class='onoff'>Ctrl</button>
60 |       <button id="toggleAltButton" class='onoff'>Alt</button>
61 |       <button id="muteButton" class='onoff'>Mute</button>
62 |       <button id="selectWindowButton" class='onoff'>Select Window</button>
63 |       <button id="fullscreenButton" class='onoff'>Fullscreen</button>
64 |     </div>
65 |     <video id="screen" autoplay playsinline></video>
66 |     <div id="connectingBox" style="margin: 50pt 0;font-size: 40pt;">
67 |       <div class="loading-spinner"></div>
68 |       Connecting...
69 |     </div>
70 |   </div>
71 | 
72 |   <div class="browser noplayer" style="margin: 30pt 0;text-align: center">
73 |     <div><a href="https://binzume.github.io/vr-workspace/#app:app-webrtc-rdp">WebXR Client</a></div>
74 |     <div><a href="https://github.com/binzume/webrtc-rdp/blob/master/README.md">README.md</a></div>
75 |   </div>
76 | 
77 | </body>
78 | 
79 | </html>


--------------------------------------------------------------------------------
/app/main.js:
--------------------------------------------------------------------------------
  1 | // @ts-check
  2 | const { app, ipcMain, BrowserWindow, Tray, Menu, desktopCapturer, screen, systemPreferences } = require('electron');
  3 | const path = require('path');
  4 | const karakuri = require('karakurijs');
  5 | 
  6 | // https://github.com/electron/electron/issues/28422
  7 | app.commandLine.appendSwitch('enable-experimental-web-platform-features');
  8 | 
  9 | class InputManager {
 10 |   constructor() {
 11 |     /** @type {Record<string, Electron.Display>} */
 12 |     this.displays = {};
 13 |     /** @type {Electron.DesktopCapturerSource[]} */
 14 |     this.sources = [];
 15 |   }
 16 |   async updateSources(types = ['screen']) {
 17 |     this.sources = await desktopCapturer.getSources({ types: types, thumbnailSize: { width: 0, height: 0 }, fetchWindowIcons: false });
 18 |     if (types.includes('screen')) {
 19 |       this.displays = {};
 20 |       for (let d of screen.getAllDisplays()) {
 21 |         this.displays[d.id] = d;
 22 |       }
 23 |     }
 24 |   }
 25 |   getSourceInfos() {
 26 |     return this.sources.map(s => ({
 27 |       id: s.id,
 28 |       name: s.name,
 29 |       display_id: s.display_id,
 30 |     }));
 31 |   }
 32 |   _getWindowId(target) {
 33 |     let m = target?.id?.match(/^window:(\d+):/);
 34 |     return m ? m[1] : null;
 35 |   }
 36 |   async sendMouse(mouseMessage) {
 37 |     let { target, action, x, y, button } = mouseMessage;
 38 |     let windowId = this._getWindowId(target);
 39 |     if (windowId) {
 40 |       action != 'move' && await karakuri.setForegroundWindow(windowId);
 41 |       this.moveMouse_window(windowId, x, y);
 42 |     } else {
 43 |       let d = this.displays[target.display_id];
 44 |       this.moveMouse_display(d || screen.getPrimaryDisplay(), x, y);
 45 |     }
 46 |     if (action == 'click') {
 47 |       karakuri.click(button);
 48 |     } else if (action == 'mouseup' || action == 'up') {
 49 |       karakuri.toggleMouseButton(button, false);
 50 |     } else if (action == 'mousedown' || action == 'down') {
 51 |       karakuri.toggleMouseButton(button, true);
 52 |     }
 53 |   }
 54 |   moveMouse_display(d, x, y) {
 55 |     let p = this._toScreenPoint(d, x, y);
 56 |     karakuri.setMousePos(p.x, p.y)
 57 |   }
 58 |   moveMouse_window(windowId, x, y) {
 59 |     let bounds = karakuri.getWindowBounds(windowId);
 60 |     if (bounds) {
 61 |       karakuri.setMousePos(bounds.x + bounds.width * x, bounds.y + bounds.height * y)
 62 |     }
 63 |   }
 64 |   async sendKey(keyMessage) {
 65 |     let { target, action, key, modifiers } = keyMessage;
 66 |     let windowId = this._getWindowId(target);
 67 |     windowId && karakuri.setForegroundWindow(windowId);
 68 |     if (action == 'press') {
 69 |       karakuri.tapKey(key, modifiers);
 70 |     } else {
 71 |       karakuri.toggleKey(key, action == 'down', modifiers);
 72 |     }
 73 |   }
 74 |   streamFromPoint(target, x, y) {
 75 |     let d = this.displays[target.display_id];
 76 |     if (d == null && target.id?.startsWith('screen:0:')) {
 77 |       console.log('primary display?', target.id);
 78 |       d = screen.getPrimaryDisplay();
 79 |     }
 80 |     if (d == null) {
 81 |       console.log('invalid target: ', target);
 82 |       return null;
 83 |     }
 84 |     let p = this._toScreenPoint(d, x, y);
 85 |     let hWnd = karakuri.windowFromPoint(p.x, p.y);
 86 |     if (!hWnd) {
 87 |       return null;
 88 |     }
 89 |     let bounds = karakuri.getWindowBounds(hWnd);
 90 |     let r = null;
 91 |     if (bounds) {
 92 |       let p0 = this._fromScreenPoint(d, bounds.x, bounds.y);
 93 |       let p1 = this._fromScreenPoint(d, bounds.x + bounds.width, bounds.y + bounds.height);
 94 |       r = { x: p0.x, y: p0.y, width: p1.x - p0.x, height: p1.y - p0.y };
 95 |     }
 96 |     return { id: `window:${hWnd}:0`, rect: r, rawBounds: bounds };
 97 |   }
 98 |   _toScreenPoint(d, x, y) {
 99 |     let p = { x: d.bounds.x + d.bounds.width * x, y: d.bounds.y + d.bounds.height * y };
100 |     return process.platform == 'win32' ? screen.dipToScreenPoint(p) : p;
101 |   }
102 |   _fromScreenPoint(d, x, y) {
103 |     let p = process.platform == 'win32' ? screen.screenToDipPoint({ x: x, y: y }) : { x: x, y: y };
104 |     return { x: (p.x - d.bounds.x) / d.bounds.width, y: (p.y - d.bounds.y) / d.bounds.height };
105 |   }
106 | }
107 | 
108 | class RDPApp {
109 |   constructor() {
110 |     this.mainWindow = null;
111 |     this.tray = this.createTray();
112 |     this.inputManager = new InputManager();
113 |     this.createWindow();
114 |   }
115 |   createWindow() {
116 |     if (this.mainWindow) {
117 |       return;
118 |     }
119 |     const window = new BrowserWindow({
120 |       width: 640,
121 |       height: 480,
122 |       webPreferences: {
123 |         preload: path.join(__dirname, 'preload.js'),
124 |       }
125 |     });
126 | 
127 |     const isMac = process.platform === 'darwin'
128 |     let menuTemplate = [
129 |       ...(isMac ? [
130 |         { role: 'appMenu' }
131 |       ] : [
132 |         { role: 'fileMenu' }
133 |       ]),
134 |       {
135 |         label: 'Debug',
136 |         submenu: [
137 |           { role: 'reload' },
138 |           { role: 'forceReload' },
139 |           { role: 'toggleDevTools' },
140 |         ]
141 |       },
142 |     ];
143 |     // @ts-ignore
144 |     const menu = Menu.buildFromTemplate(menuTemplate);
145 |     Menu.setApplicationMenu(menu);
146 | 
147 |     window.addListener('close', (ev) => {
148 |       if (this.mainWindow) {
149 |         window.hide();
150 |         ev.preventDefault();
151 |       }
152 |     });
153 |     window.addListener('closed', (ev) => {
154 |       this.mainWindow = null;
155 |     });
156 |     app.addListener('before-quit', (ec) => {
157 |       this.mainWindow = null;
158 |     });
159 | 
160 |     window.loadFile('index.html');
161 |     if (process.argv.includes('--debug')) {
162 |       window.webContents.openDevTools();
163 |     }
164 |     this.mainWindow = window;
165 |   }
166 |   createTray() {
167 |     let iconPath = __dirname + '/images/icon1.png';
168 |     let contextMenu = Menu.buildFromTemplate([
169 |       { label: 'Settings', click: () => this.mainWindow.show() },
170 |       { label: 'Reload', click: () => this.mainWindow.reload() },
171 |       { type: 'separator' },
172 |       { label: 'Quit', role: 'quit' },
173 |     ]);
174 |     let tray = new Tray(iconPath);
175 |     tray.setContextMenu(contextMenu);
176 |     tray.setToolTip(app.name);
177 |     tray.on('click', () => tray.popUpContextMenu());
178 |     return tray;
179 |   }
180 | }
181 | 
182 | let rdp;
183 | app.whenReady().then(() => {
184 |   if (systemPreferences.getMediaAccessStatus('screen') != 'granted') {
185 |     console.log('ERROR: No screen capture permission');
186 |     karakuri.requestPermission('screenCapture');
187 |   }
188 |   if (!karakuri.requestPermission('accessibility')) {
189 |     console.log('ERROR: No accessibility permission');
190 |   }
191 | 
192 |   rdp = new RDPApp();
193 | 
194 |   let inputManager = rdp.inputManager;
195 |   ipcMain.handle('getDisplayStreams', async (event, types) => {
196 |     await inputManager.updateSources(types);
197 |     return inputManager.getSourceInfos();
198 |   });
199 |   ipcMain.handle('sendMouse', (event, mouseMessage) => {
200 |     return inputManager.sendMouse(mouseMessage);
201 |   });
202 |   ipcMain.handle('sendKey', (event, keyMessage) => {
203 |     return inputManager.sendKey(keyMessage);
204 |   });
205 |   ipcMain.handle('streamFromPoint', async (event, params) => {
206 |     return inputManager.streamFromPoint(params.target, params.x, params.y);
207 |   });
208 | 
209 |   app.on('activate', () => {
210 |     // On macOS it's common to re-create a window in the app when the
211 |     // dock icon is clicked and there are no other windows open.
212 |     if (BrowserWindow.getAllWindows().length === 0) rdp.createWindow();
213 |   });
214 | })
215 | 
216 | app.on('window-all-closed', () => {
217 |   if (process.platform !== 'darwin') app.quit()
218 | });
219 | 


--------------------------------------------------------------------------------
/app/package.json:
--------------------------------------------------------------------------------
 1 | {
 2 |   "name": "webrtc-rdp-app",
 3 |   "version": "0.0.5",
 4 |   "description": "WebRTC RDP application",
 5 |   "main": "main.js",
 6 |   "scripts": {
 7 |     "start": "electron .",
 8 |     "start-debug": "electron . --debug",
 9 |     "prepare": "npx electron-rebuild",
10 |     "build-win": "npx electron-builder --win --x64",
11 |     "build-mac": "npx electron-builder --mac --x64 --dir",
12 |     "build-mac-arm64": "npx electron-builder --mac --arm64 --dir"
13 |   },
14 |   "repository": "https://github.com/binzume/webrtc-rdp",
15 |   "keywords": [
16 |     "WebRTC",
17 |     "RemoteDesktop",
18 |     "Electron"
19 |   ],
20 |   "author": "binzume",
21 |   "license": "MIT",
22 |   "devDependencies": {
23 |     "electron": "^19.0.0",
24 |     "electron-builder": "^23.6.0",
25 |     "@electron/rebuild": "^3.2.10"
26 |   },
27 |   "dependencies": {
28 |     "@open-ayame/ayame-web-sdk": "^2022.1.0",
29 |     "karakurijs": "^0.0.4"
30 |   },
31 |   "build": {
32 |     "appId": "net.binzume.webrtc-rdp",
33 |     "productName": "WebRTC Remote Desktop",
34 |     "npmRebuild": false,
35 |     "files": [
36 |       "!**/*.{vcxproj,ilk,exp,lib,tlog,ts,c,cc,mm,h}",
37 |       "!**/.*",
38 |       "!node_modules/karakurijs/karakurijs-*",
39 |       "!node_modules/ffi-napi/{deps,src,bin,prebuilds,build/deps}",
40 |       "!node_modules/ref-napi/{docs,include}",
41 |       "!**/node-addon-api/*"
42 |     ]
43 |   }
44 | }


--------------------------------------------------------------------------------
/app/preload.js:
--------------------------------------------------------------------------------
 1 | const { ipcRenderer, contextBridge } = require('electron')
 2 | 
 3 | 
 4 | if (location.origin != 'file://') {
 5 |   throw location.origin
 6 | }
 7 | 
 8 | contextBridge.exposeInMainWorld('RDP', {
 9 |   getDisplayStreams: (spec) => ipcRenderer.invoke('getDisplayStreams', spec),
10 |   sendMouse: (params) => ipcRenderer.invoke('sendMouse', params),
11 |   sendKey: (params) => ipcRenderer.invoke('sendKey', params),
12 |   streamFromPoint: (params) => ipcRenderer.invoke('streamFromPoint', params),
13 | });
14 | 


--------------------------------------------------------------------------------
/app/rtcfilesystem-client.js:
--------------------------------------------------------------------------------
  1 | // @ts-check
  2 | 
  3 | /** @typedef {{name: string, type: string, size: number, updatedTime: number, [k:string]: any}} RTCFileSystemFileStat */
  4 | 
  5 | class RTCFileSystemClient {
  6 |     constructor() {
  7 |         /** @type {(WebSocket | RTCDataChannel)[]} */
  8 |         this.sockets = [];
  9 |         this.available = false;
 10 |         this._onAvailable = null;
 11 |         this.disconnectDelayMs = 5000;
 12 |         this.ondisconnected = null;
 13 |         this._disconnectTimer = null;
 14 |         this._seq = 0;
 15 |         /** @type {Record<string, {resolve:any, reject:any}>} */
 16 |         this._req = {};
 17 |         this.setAvailable(false);
 18 |     }
 19 |     /** @returns {Promise<RTCFileSystemFileStat>} */
 20 |     async stat(path) {
 21 |         return await this._request({ op: 'stat', path: path });
 22 |     }
 23 |     /** @returns {Promise<RTCFileSystemFileStat[]>} */
 24 |     async files(path, offset = 0, limit = -1, options = null) {
 25 |         return await this._request({ op: 'files', path: path, p: offset, l: limit, options: options });
 26 |     }
 27 |     /** @returns {Promise<ArrayBuffer>} */
 28 |     async read(path, offset, len) {
 29 |         return await this._request({ op: 'read', path: path, p: offset, l: len });
 30 |     }
 31 |     /** @returns {Promise<number>} */
 32 |     async write(path, offset, data) {
 33 |         return await this._request({ op: 'write', path: path, p: offset, b: data });
 34 |     }
 35 |     /** @returns {Promise<number>} */
 36 |     async writeBytes(path, offset, data) {
 37 |         let b64 = btoa(String.fromCharCode(...data));
 38 |         return await this._request({ op: 'write', path: path, p: offset, b: b64 });
 39 |     }
 40 |     /** @returns {Promise<boolean>} */
 41 |     async truncate(path, pos) {
 42 |         return await this._request({ op: 'truncate', path: path, p: pos });
 43 |     }
 44 |     /** @returns {Promise<boolean>} */
 45 |     async remove(path) {
 46 |         return await this._request({ op: 'remove', path: path });
 47 |     }
 48 |     /** @returns {Promise<boolean>} */
 49 |     async rename(path, path2) {
 50 |         return await this._request({ op: 'rename', path: path, path2: path2 });
 51 |     }
 52 |     /** @returns {Promise<boolean>} */
 53 |     async mkdir(path) {
 54 |         return await this._request({ op: 'mkdir', path: path });
 55 |     }
 56 | 
 57 |     readStream(path, pos, end) {
 58 |         const blockSize = 32768;
 59 |         let queue = [];
 60 |         let prefetch = () => {
 61 |             if (pos < end) {
 62 |                 let sz = Math.min(end - pos, blockSize);
 63 |                 queue.push(this.read(path, pos, sz));
 64 |                 pos += sz;
 65 |             }
 66 |         };
 67 |         return new ReadableStream({
 68 |             // @ts-ignore
 69 |             type: 'bytes',
 70 |             start: (_controller) => {
 71 |                 for (let i = 0; i < 16; i++) {
 72 |                     prefetch();
 73 |                 }
 74 |             },
 75 |             pull: async (controller) => {
 76 |                 let buf = await queue.shift();
 77 |                 if (buf.byteLength > 0) {
 78 |                     controller.enqueue(new DataView(buf));
 79 |                     prefetch();
 80 |                 }
 81 |                 if (queue.length == 0) {
 82 |                     controller.close();
 83 |                 }
 84 |             }
 85 |         });
 86 |     }
 87 | 
 88 |     writeStream(path, options = {}) {
 89 |         const blockSize = 32768 / 4 * 3; // BASE64
 90 |         let pos = options.start || 0;
 91 |         return new WritableStream({
 92 |             start: async (_controller) => {
 93 |                 if (!options.keepExistingData) {
 94 |                     this.truncate(path, 0);
 95 |                 }
 96 |             },
 97 |             write: async (/** @type {Uint8Array&{type: string, [key:string]:any}} */ chunk, _controller) => {
 98 |                 if (chunk.type == 'seek') {
 99 |                     pos = chunk.position;
100 |                     return;
101 |                 }
102 |                 let l = chunk.byteLength;
103 |                 for (let p = 0; p < l; p += blockSize) {
104 |                     // TODO: prevent memcopy
105 |                     await this.writeBytes(path, pos + p, chunk.slice(p, p + blockSize));
106 |                 }
107 |                 pos += l;
108 |             }
109 |         });
110 |     }
111 | 
112 |     async _request(req) {
113 |         if (this.sockets.length == 0) { throw 'no_connection'; }
114 |         let rid = ++this._seq;
115 |         req.rid = rid;
116 |         return new Promise((resolve, reject) => {
117 |             this._req[rid] = { resolve, reject };
118 |             this.sockets[0].send(JSON.stringify(req));
119 |         });
120 |     }
121 | 
122 |     /**
123 |      * @param {MessageEvent} ev
124 |      */
125 |     async handleEvent(ev) {
126 |         if (typeof ev.data === "string") {
127 |             await this._handleResponse(JSON.parse(ev.data));
128 |         } else {
129 |             /** @type {ArrayBuffer} */
130 |             let buf = ev.data;
131 |             if (buf.byteLength >= 8) {
132 |                 let v = new DataView(buf);
133 |                 await this._handleResponse({ rid: v.getUint32(4, true), data: buf.slice(8) });
134 |             }
135 |         }
136 |     }
137 | 
138 |     async _handleResponse(msg) {
139 |         let req = this._req[msg.rid];
140 |         if (req) {
141 |             delete this._req[msg.rid];
142 |             msg.error ? req.reject(msg.error) : req.resolve(msg.data);
143 |         }
144 |     }
145 | 
146 |     addSocket(socket, ready = true) {
147 |         socket.binaryType = 'arraybuffer';
148 |         clearTimeout(this._disconnectTimer);
149 |         this._disconnectTimer = 0;
150 |         this.sockets.push(socket);
151 |         ready && this.setAvailable(true);
152 |     }
153 |     setAvailable(available) {
154 |         this.available = available;
155 |         if (available) {
156 |             this._onAvailable && this._onAvailable(0);
157 |         } else {
158 |             this._waitSocket = new Promise(r => this._onAvailable = r);
159 |         }
160 |     }
161 |     async wait() {
162 |         this.available || await this._waitSocket;
163 |     }
164 |     removeSocket(socket) {
165 |         this.sockets = this.sockets.filter(s => s != socket);
166 |         if (this.sockets.length == 0) {
167 |             this.reset();
168 |             this._disconnectTimer = setTimeout(() => {
169 |                 this.setAvailable(false);
170 |                 this.ondisconnected?.();
171 |             }, this.disconnectDelayMs);
172 |         }
173 |     }
174 |     reset() {
175 |         for (let r of Object.values(this._req)) { r.reject('reset'); }
176 |         this._req = {};
177 |         this.sockets = [];
178 |     }
179 | }
180 | 
181 | /**
182 |  * @implements {Folder}
183 |  */
184 | class RTCFileSystemClientFolder {
185 |     /**
186 |      * @param {RTCFileSystemClient} client 
187 |      * @param {string} path 
188 |      * @param {string} prefix
189 |      */
190 |     constructor(client, path, prefix) {
191 |         this._client = client;
192 |         this.path = path;
193 |         this._pathPrefix = prefix || '';
194 |         this.size = -1; // unknown size
195 |         this.onupdate = null;
196 |     }
197 | 
198 |     /** @returns {Promise<{items: FileInfo[], next: number}>} */
199 |     async getFiles(offset, limit = 100, options = null, signal = null) {
200 |         let filesopt = options && options.sortField ? { sort: (options.sortOrder == 'd' ? '-' : '') + options.sortField } : null;
201 |         let client = this._client;
202 |         await client.wait();
203 |         signal?.throwIfAborted();
204 |         let files = await client.files(this.path, offset, limit, filesopt);
205 |         let items = files.map(f => this._procFile(f));
206 |         let sz = offset + items.length + (items.length >= limit ? 1 : 0);
207 |         if (sz > this.size) {
208 |             this.size = sz;
209 |             this.onupdate?.();
210 |         }
211 |         return {
212 |             items: items,
213 |             next: items.length >= limit ? offset + limit : null,
214 |         };
215 |     }
216 |     mkdir(name, options = {}) {
217 |         return this._client.mkdir((this.path != '' ? this.path + '/' : '') + name);
218 |     }
219 |     async writeFile(name, blob, options = {}) {
220 |         let path = (this.path != '' ? this.path + '/' : '') + name;
221 |         await blob.stream().pipeTo(this._client.writeStream(path));
222 |     }
223 |     _procFile(f) {
224 |         let client = this._client;
225 |         let dir = this.path != '' ? this.path + '/' : '';
226 |         return ({
227 |             name: f.name,
228 |             type: f.type == 'directory' ? 'folder' : f.type,
229 |             size: f.size,
230 |             lastModified: f.updatedTime,
231 |             updatedTime: f.updatedTime,
232 |             tags: f.metadata?.tags || [],
233 |             path: this._pathPrefix + dir + f.name,
234 |             stream(start = 0, end = -1) { return client.readStream(dir + f.name, start, end < 0 ? f.size : end); },
235 |             async createWritable(options = {}) { return client.writeStream(dir + f.name, options); },
236 |             async fetch(start = 0, end = -1) {
237 |                 return new Response(client.readStream(dir + f.name, start, end < 0 ? f.size : end), { headers: { 'Content-Type': f.type, 'Content-Length': '' + f.size } });
238 |             },
239 |             update(blob) { return blob.stream().pipeTo(client.writeStream(dir + f.name)); },
240 |             remove() { return client.remove(dir + f.name); },
241 |             rename(name) { return client.rename(dir + f.name, dir + name); },
242 |             thumbnail: f.metadata?.thumbnail ? {
243 |                 type: 'image/jpeg',
244 |                 async fetch(start = 0, end = -1) {
245 |                     return new Response(client.readStream(dir + f.name + f.metadata?.thumbnail, start, end < 0 ? 32768 : end), { headers: { 'Content-Type': 'image/jpeg' } });
246 |                 }
247 |             } : null,
248 |         });
249 |     }
250 | 
251 |     /**
252 |      * @returns {string}
253 |      */
254 |     getParentPath() {
255 |         if (this.path == '' || this.path == '/') {
256 |             return null;
257 |         }
258 |         return this._pathPrefix + this.path.substring(0, this.path.lastIndexOf('/'));
259 |     }
260 | }
261 | 
262 | class RTCFileSystemManager {
263 |     constructor() {
264 |         /** @type {Record<string,RTCFileSystemClient>} */
265 |         this._clients = {};
266 |     }
267 | 
268 |     /**
269 |      * @param {string} id host unique string (roomId)
270 |      * @param {string} name volume name
271 |      * @returns 
272 |      */
273 |     getRtcChannelSpec(id, name) {
274 |         return {
275 |             onopen: (ch, _ev) => {
276 |                 ch.binaryType = 'arraybuffer';
277 |                 if (!this._clients[id]) {
278 |                     this._clients[id] = new RTCFileSystemClient();
279 |                     console.log('FileSystemClient: connected ' + id);
280 |                     if (globalThis.storageAccessors) {
281 |                         globalThis.storageAccessors[id] = {
282 |                             name: name,
283 |                             root: '',
284 |                             getFolder: (path, prefix) => new RTCFileSystemClientFolder(this._clients[id], path, prefix),
285 |                             parsePath: (path) => path ? path.split('/').map(p => [p]) : [],
286 |                         };
287 |                     }
288 |                     this._clients[id].ondisconnected = () => {
289 |                         console.log('FileSystemClient: disconnected ' + id);
290 |                         this._clients[id].ondisconnected = null;
291 |                         delete this._clients[id];
292 |                         if (globalThis.storageAccessors) {
293 |                             delete globalThis.storageAccessors[id];
294 |                         }
295 |                     };
296 |                 }
297 |                 this._clients[id].addSocket(ch);
298 |             },
299 |             onclose: (ch, _ev) => {
300 |                 this._clients[id]?.removeSocket(ch);
301 |             },
302 |             onmessage: (_ch, ev) => this._clients[id].handleEvent(ev)
303 |         };
304 |     }
305 |     static _registered = {};
306 |     registerAll(connectionFactory, roomIdPrefix = '') {
307 |         globalThis.storageAccessors ||= {};
308 |         function add(roomId, signalingKey, password, name) {
309 |             if (RTCFileSystemManager._registered[roomId]) {
310 |                 return;
311 |             }
312 |             RTCFileSystemManager._registered[roomId] = true;
313 |             let client = new RTCFileSystemClient();
314 |             /** @type {PlayerConnection|null} */
315 |             let player = null;
316 |             let id = roomId.startsWith(roomIdPrefix) ? roomId.substring(roomIdPrefix.length) : roomId;
317 |             globalThis.storageAccessors[id] = {
318 |                 name: name,
319 |                 detach: () => player && player.dispose(),
320 |                 getFolder(path, prefix) {
321 |                     if (player == null) {
322 |                         player = connectionFactory(signalingKey, roomId);
323 |                         player.authToken = password;
324 |                         player.dataChannels['fileServer'] = {
325 |                             onopen: (ch, _ev) => client.addSocket(ch, false),
326 |                             onclose: (ch, _ev) => client.removeSocket(ch),
327 |                             onmessage: (_ch, ev) => client.handleEvent(ev),
328 |                         };
329 |                         player.onauth = (ok) => {
330 |                             if (!ok) {
331 |                                 player.disconnect();
332 |                                 return;
333 |                             }
334 |                             client.setAvailable(true);
335 |                         };
336 |                         player.onstatechange = (state, oldState, reason) => {
337 |                             if (state == 'disconnected' && reason != 'redirect') {
338 |                                 player = null;
339 |                             }
340 |                         };
341 |                         player.connect();
342 |                     }
343 |                     return new RTCFileSystemClientFolder(client, path, prefix);
344 |                 },
345 |                 parsePath: (path) => path ? path.split('/').map(p => [p]) : [],
346 |             };
347 |         }
348 | 
349 |         // see https://github.com/binzume/webrtc-rdp
350 |         let config = JSON.parse(localStorage.getItem('webrtc-rdp-settings') || 'null') || { devices: [] };
351 |         let devices = config.devices != null ? config.devices : [config];
352 |         for (let device of devices) {
353 |             let name = (device.name || device.userAgent || device.roomId).substring(0, 64);
354 |             add(device.roomId, device.signalingKey, device.token, name);
355 |         }
356 |     }
357 | }
358 | 
359 | // Install storage accessor for WebXR client storage.
360 | // player.dataChannels['fileServer'] = rtcFileSystemManager.getRtcChannelSpec(roomId, name);
361 | globalThis.rtcFileSystemManager = new RTCFileSystemManager();
362 | 


--------------------------------------------------------------------------------
/app/rtcfilesystem-server.js:
--------------------------------------------------------------------------------
  1 | 
  2 | // @ts-check
  3 | 
  4 | /** @typedef {FileSystemHandleArray} FileSystemHandle2 */
  5 | 
  6 | class FileSystemHandleArray {
  7 |     constructor(name = '') {
  8 |         this.name = name;
  9 |         /** @type {FileSystemHandleKind} */
 10 |         this.kind = 'directory';
 11 |         /** @type {Record<string,FileSystemHandle2>} */
 12 |         this._entries = {};
 13 |     }
 14 | 
 15 |     /**
 16 |      * @param {FileSystemHandle2} ent 
 17 |      * @param {string} name 
 18 |      */
 19 |     addEntry(ent, name = null) { this._entries[name || ent.name] = ent; }
 20 | 
 21 |     // FileSystemHandle methods
 22 |     isSameEntry(ent) { return Promise.resolve(ent === this); }
 23 |     queryPermission(_options = null) { return Promise.resolve('granted'); }
 24 |     /**
 25 |      * @param {{mode: string}} _options 
 26 |      * @returns {Promise<string>}
 27 |      */
 28 |     async requestPermission(_options = null) {
 29 |         let ok = true;
 30 |         for (let handle of Object.values(this._entries)) {
 31 |             ok = (await handle.requestPermission({ mode: 'readwrite' }) == 'granted') && ok;
 32 |         }
 33 |         return ok ? 'granted' : 'denied';
 34 |     }
 35 | 
 36 |     // FileSystemDirectoryHandle methods
 37 |     /** @returns { {[Symbol.asyncIterator](): AsyncGenerator<FileSystemHandle2, void, unknown>}} */
 38 |     values() { return this._asyncIterator(Object.values(this._entries)); }
 39 |     keys() { return this._asyncIterator(Object.keys(this._entries)); }
 40 |     entries() { return this._asyncIterator(Object.entries(this._entries)); }
 41 |     async getFileHandle(name, _options = null) { return this._entries[name]; }
 42 |     async getDirectoryHandle(name, _options = null) { return this._entries[name]; }
 43 |     async removeEntry(name) { delete this._entries[name]; }
 44 |     async resolve(possibleDescendant) { return []; }
 45 | 
 46 |     // FileSystemFileHandle methods
 47 |     /**  @return {Promise<File>}     */
 48 |     async getFile() { throw "not a file"; }
 49 |     /**  @return {Promise<WritableStream&{seek:any, write:any, truncate:any}>} */
 50 |     async createWritable(_options) { throw "not a file"; }
 51 | 
 52 |     _asyncIterator(array) {
 53 |         return { async *[Symbol.asyncIterator]() { for (let ent of array) { yield ent; } } }
 54 |     }
 55 | }
 56 | 
 57 | class FileSystemWrapper {
 58 |     /**
 59 |      * @param {FileSystemHandle2} handle 
 60 |      */
 61 |     constructor(handle) {
 62 |         this.writable = false;
 63 |         this.handle = handle;
 64 |     }
 65 | 
 66 |     async setWritable(writable) {
 67 |         this.writable = writable;
 68 |         if (writable) {
 69 |             return await this.handle.requestPermission({ mode: 'readwrite' }) === 'granted';
 70 |         }
 71 |     }
 72 |     async stat(path) {
 73 |         return await this.statInternal(await this.resolvePath(path));
 74 |     }
 75 |     async files(path, offset = 0, limit = -1) {
 76 |         let h = await this.resolvePath(path, 'directory');
 77 |         if (limit == 0) { return []; }
 78 |         let fileTasks = [];
 79 |         let pos = 0;
 80 |         for await (let ent of h.values()) {
 81 |             if (pos++ < offset) { continue; }
 82 |             fileTasks.push(this.statInternal(ent));
 83 |             if (limit > 0 && fileTasks.length >= limit) { break; }
 84 |         }
 85 |         return Promise.all(fileTasks);
 86 |     }
 87 |     async read(path, offset = 0, len) {
 88 |         if (path.endsWith('#thumbnail.jpeg')) {
 89 |             let file = await this.resolveFile(path.substring(0, path.lastIndexOf('#')));
 90 |             let blob = await this.createThumbnail(file);
 91 |             return blob.slice(offset, offset + len);
 92 |         }
 93 |         let file = await this.resolveFile(path);
 94 |         return file.slice(offset, offset + len);
 95 |     }
 96 |     async write(path, offset = 0, data) {
 97 |         if (!this.writable) { throw 'readonly'; }
 98 |         let handle = await this.resolvePath(path, 'file', true);
 99 |         let writer = await handle.createWritable({ keepExistingData: true });
100 |         await writer.seek(offset);
101 |         await writer.write(data);
102 |         await writer.close();
103 |         return data.length;
104 |     }
105 |     async truncate(path, size = 0) {
106 |         if (!this.writable) { throw 'readonly'; }
107 |         let handle = await this.resolvePath(path, 'file', true);
108 |         let writer = await handle.createWritable({ keepExistingData: true });
109 |         await writer.truncate(size);
110 |         await writer.close();
111 |         return true;
112 |     }
113 |     async mkdir(path) {
114 |         if (!this.writable) { throw 'readonly'; }
115 |         let handle = await this.resolvePath(path, 'directory', true);
116 |         return handle != null;
117 |     }
118 |     async remove(path) {
119 |         if (!this.writable) { throw 'readonly'; }
120 |         let dir = '', name = path;
121 |         let p = path.lastIndexOf('/');
122 |         if (p > 0) {
123 |             dir = path.substring(0, p);
124 |             name = path.substring(p + 1);
125 |         }
126 |         let hdir = await this.resolvePath(dir);
127 |         await hdir.removeEntry(name);
128 |         return true;
129 |     }
130 |     async rename(path, path2) {
131 |         if (!this.writable) { throw 'readonly'; }
132 |         let handle = await this.resolvePath(path, 'file');
133 |         // @ts-ignore
134 |         if (handle && handle.move) {
135 |             // @ts-ignore
136 |             await handle.move(path2);
137 |             return true;
138 |         }
139 |         return false;
140 |     }
141 | 
142 |     /**
143 |      * @param {string} path
144 |      * @return {Promise<FileSystemHandle2>}
145 |      */
146 |     async resolvePath(path, kind = null, create = false) {
147 |         let p = path.split('/');
148 |         let h = this.handle;
149 |         let wrap = async (/** @type {Promise<FileSystemHandle2>} */ t) => { try { return await t; } catch { } };
150 |         for (let i = 0; i < p.length; i++) {
151 |             if (p[i] == '' || p[i] == '.') { continue; }
152 |             let c = await ((i == p.length - 1 && kind == 'file') ? wrap(h.getFileHandle(p[i], { create })) : wrap(h.getDirectoryHandle(p[i], { create })));
153 |             if (!c && kind == null) { c = await wrap(h.getFileHandle(p[i], { create })); }
154 |             if (!c) throw 'noent';
155 |             h = c;
156 |         }
157 |         return h;
158 |     }
159 |     /**
160 |      * @param {string} path
161 |      * @return {Promise<File>}
162 |      */
163 |     async resolveFile(path) {
164 |         return await (await this.resolvePath(path, 'file')).getFile();
165 |     }
166 | 
167 |     /**
168 |      * @param {FileSystemHandle2} handle 
169 |      */
170 |     async statInternal(handle) {
171 |         if (handle.kind == 'file') {
172 |             let f = await handle.getFile();
173 |             let stat = { type: f.type || this._typeFromName(f.name), name: f.name, size: f.size, updatedTime: f.lastModified }
174 |             if (["image/jpeg", "image/png", "image/gif", "image/webp", "image/bmp"].includes(f.type)) {
175 |                 stat.metadata = { thumbnail: "#thumbnail.jpeg" };
176 |             }
177 |             return stat;
178 |         } else {
179 |             return { type: 'directory', size: 0, name: handle.name, updatedTime: null }
180 |         }
181 |     }
182 | 
183 |     _typeFromName(name) {
184 |         return {
185 |             // video
186 |             ".mp4": "video/mp4",
187 |             ".m4v": "video/mp4",
188 |             ".f4v": "video/mp4",
189 |             ".mov": "video/mp4",
190 |             ".webm": "video/webm",
191 |             ".ogv": "video/ogv",
192 |             // image
193 |             ".jpeg": "image/jpeg",
194 |             ".jpg": "image/jpeg",
195 |             ".gif": "image/gif",
196 |             ".png": "image/png",
197 |             ".bmp": "image/bmp",
198 |             ".webp": "image/webp",
199 |             // audio
200 |             ".aac": "audio/aac",
201 |             ".mp3": "audio/mp3",
202 |             ".ogg": "audio/ogg",
203 |             ".mid": "audio/midi",
204 |         }[name.split('.').pop()] || '';
205 |     }
206 | 
207 |     /**
208 |      * @param {Blob} file 
209 |      * @returns {Promise<Blob>}
210 |      */
211 |     async createThumbnail(file, maxWidth = 200, maxHeight = 200) {
212 |         let canvas = document.createElement('canvas');
213 |         let drawThumbnail = (image, w, h) => {
214 |             if (w > maxWidth) {
215 |                 h = h * maxWidth / w;
216 |                 w = maxWidth;
217 |             }
218 |             if (h > maxHeight) {
219 |                 w = w * maxHeight / h;
220 |                 h = maxHeight;
221 |             }
222 |             canvas.width = w;
223 |             canvas.height = h;
224 |             canvas.getContext('2d').drawImage(image, 0, 0, w, h);
225 |         };
226 |         let objectUrl = URL.createObjectURL(file);
227 |         let media;
228 |         try {
229 |             if (file.type.startsWith('video')) {
230 |                 // TODO: detect background tab
231 |                 media = document.createElement('video');
232 |                 media.muted = true;
233 |                 media.autoplay = true;
234 |                 await new Promise((resolve, reject) => {
235 |                     media.onloadeddata = resolve;
236 |                     media.onerror = reject;
237 |                     media.src = objectUrl;
238 |                     setTimeout(reject, 3000);
239 |                 });
240 |                 await new Promise((resolve, _reject) => {
241 |                     media.onseeked = resolve;
242 |                     media.currentTime = 3;
243 |                     setTimeout(resolve, 500);
244 |                 });
245 |                 drawThumbnail(media, media.videoWidth, media.videoHeight);
246 |             } else {
247 |                 media = new Image();
248 |                 await new Promise((resolve, reject) => {
249 |                     media.onload = resolve;
250 |                     media.onerror = reject;
251 |                     media.src = objectUrl;
252 |                     setTimeout(reject, 5000);
253 |                 });
254 |                 drawThumbnail(media, media.naturalWidth, media.naturalHeight);
255 |             }
256 |         } finally {
257 |             if (media) { media.src = ''; }
258 |             URL.revokeObjectURL(objectUrl);
259 |         }
260 |         return await new Promise((resolve, _) => canvas.toBlob(resolve, 'image/jpeg', 0.8));
261 |     }
262 | }
263 | 
264 | class FileServer {
265 |     /**
266 |      * @param {FileSystemHandle2} handle 
267 |      */
268 |     constructor(handle) {
269 |         this.fs = new FileSystemWrapper(handle);
270 |     }
271 |     /**
272 |      * @param {MessageEvent} ev 
273 |      * @param {RTCDataChannel | WebSocket} socket 
274 |      */
275 |     async handleEvent(ev, socket) {
276 |         let cmd = JSON.parse(ev.data);
277 |         let fs = this.fs;
278 |         try {
279 |             switch (cmd.op) {
280 |                 case 'stat':
281 |                     socket.send(JSON.stringify({ rid: cmd.rid, data: await fs.stat(cmd.path) }));
282 |                     break;
283 |                 case 'files':
284 |                     socket.send(JSON.stringify({ rid: cmd.rid, data: await fs.files(cmd.path, cmd.p, cmd.l) }));
285 |                     break;
286 |                 case 'read':
287 |                     let data = await fs.read(cmd.path, cmd.p, cmd.l);
288 |                     socket.send(await new Blob([Uint32Array.from([0, cmd.rid]), data]).arrayBuffer()); //TODO: endian
289 |                     break;
290 |                 case 'write':
291 |                     let buf = new Uint8Array([...atob(cmd.b)].map(s => s.charCodeAt(0)));
292 |                     let l = await fs.write(cmd.path, cmd.p, buf);
293 |                     socket.send(JSON.stringify({ rid: cmd.rid, data: l }));
294 |                     break;
295 |                 case 'truncate':
296 |                     socket.send(JSON.stringify({ rid: cmd.rid, data: await fs.truncate(cmd.path, cmd.p) }));
297 |                     break;
298 |                 case 'mkdir':
299 |                     socket.send(JSON.stringify({ rid: cmd.rid, data: await fs.mkdir(cmd.path) }));
300 |                     break;
301 |                 case 'remove':
302 |                     socket.send(JSON.stringify({ rid: cmd.rid, data: await fs.remove(cmd.path) }));
303 |                     break;
304 |                 case 'rename':
305 |                     socket.send(JSON.stringify({ rid: cmd.rid, data: await fs.rename(cmd.path, cmd.path2) }));
306 |                     break;
307 |                 default:
308 |                     throw 'unknown_operation';
309 |             }
310 |         } catch (e) {
311 |             if (cmd.rid) {
312 |                 socket.send(JSON.stringify({ rid: cmd.rid, error: (typeof e == 'string') ? e : 'internal_error' }));
313 |             }
314 |             if (typeof e != 'string') { throw e; }
315 |         }
316 |     }
317 | 
318 |     getRtcChannelSpec() {
319 |         return {
320 |             onopen: (ch, _ev) => { ch.binaryType = 'arraybuffer'; },
321 |             /**
322 |              * @param {RTCDataChannel} ch 
323 |              * @param {MessageEvent} ev
324 |              */
325 |             onmessage: (ch, ev) => this.handleEvent(ev, ch)
326 |         };
327 |     }
328 | }
329 | 


--------------------------------------------------------------------------------
/app/style.css:
--------------------------------------------------------------------------------
  1 | body {
  2 |     margin: 0;
  3 |     padding: 0;
  4 |     background-color: white;
  5 | }
  6 | 
  7 | h1 {
  8 |     text-align: center;
  9 | }
 10 | 
 11 | body.standalone .browser {
 12 |     display: none;
 13 | }
 14 | 
 15 | body.standalone :not(input):not(textarea) {
 16 |     user-select: none;
 17 |     cursor: default;
 18 | }
 19 | 
 20 | body.standalone .selectable {
 21 |     user-select: all !important;
 22 |     cursor: auto !important;
 23 | }
 24 | 
 25 | body.player .noplayer {
 26 |     display: none;
 27 | }
 28 | 
 29 | #player {
 30 |     display: none;
 31 |     text-align: center;
 32 | }
 33 | 
 34 | body.player #player {
 35 |     display: block;
 36 | }
 37 | 
 38 | #player-menu {
 39 |     height: 20px;
 40 |     line-height: 1;
 41 | }
 42 | 
 43 | #player-menu button {
 44 |     margin: 0px;
 45 |     height: 20px;
 46 |     min-height: 20px;
 47 |     box-sizing: border-box;
 48 | }
 49 | 
 50 | #player-menu .onoff.active {
 51 |     background-color: lightgreen;
 52 | }
 53 | 
 54 | #screen {
 55 |     display: none;
 56 |     width: 100%;
 57 |     height: calc(100vh - 20px);
 58 |     padding: 0;
 59 |     margin: 0;
 60 |     border: none;
 61 |     background-color: #ccc;
 62 | }
 63 | 
 64 | #content {
 65 |     margin: 0 auto;
 66 |     max-width: 400pt;
 67 | }
 68 | 
 69 | .block {
 70 |     border: 2px solid #ccc;
 71 |     border-radius: 4pt;
 72 |     margin: 12pt auto;
 73 | }
 74 | 
 75 | #pairing {
 76 |     padding: 8pt 0;
 77 |     text-align: center;
 78 | }
 79 | 
 80 | #pairing .pin {
 81 |     font-size: 40pt;
 82 | }
 83 | 
 84 | #pairing .pin * {
 85 |     font-size: 40pt;
 86 | }
 87 | 
 88 | #pairing h2 {
 89 |     font-size: 20pt;
 90 |     text-align: center;
 91 |     margin: 0 0 10pt 0;
 92 |     padding: 0;
 93 | }
 94 | 
 95 | #pairing input::-webkit-outer-spin-button,
 96 | #pairing input::-webkit-inner-spin-button {
 97 |     -webkit-appearance: none;
 98 |     margin: 0;
 99 | }
100 | 
101 | #pairing input[type=number] {
102 |     -moz-appearance: textfield;
103 | }
104 | 
105 | #publishOrPlay {
106 |     text-align: center;
107 | }
108 | 
109 | #player video {
110 |     touch-action: none;
111 | }
112 | 
113 | #player .streamrect {
114 |     position: absolute;
115 |     overflow: hidden;
116 |     pointer-events: none;
117 |     box-sizing: border-box;
118 |     border: 2px solid red;
119 |     color: green;
120 |     text-align: center;
121 |     top: 0;
122 |     left: 0;
123 | }
124 | 
125 | .streamlist li {
126 |     list-style-type: none;
127 |     text-align: left;
128 | }
129 | 
130 | .streamlist .connectionstate {
131 |     margin: 0 10pt;
132 | }
133 | 
134 | .streamlist .connectionstate_ready {
135 |     color: green;
136 | }
137 | 
138 | .streamlist .connectionstate_connected {
139 |     color: red;
140 | }
141 | 
142 | .streamlist .connectionstate_disconnected {
143 |     color: darkred;
144 | }
145 | 
146 | #devices>div {
147 |     border: solid 1px #ccc;
148 |     min-height: 50pt;
149 |     border-radius: 6pt;
150 |     padding-bottom: 0 5pt;
151 |     margin-bottom: 8pt;
152 | }
153 | 
154 | #devices>div>span {
155 |     display: block;
156 |     border-top-left-radius: 4pt;
157 |     border-top-right-radius: 4pt;
158 |     width: 100%;
159 |     background-color: #e0e8f8;
160 |     font-weight: bold;
161 |     margin-bottom: 4pt;
162 | }
163 | 
164 | #devices>div>span>span {
165 |     display: inline-block;
166 |     width: calc(100% - 30pt);
167 |     white-space: nowrap;
168 |     overflow: hidden;
169 |     text-overflow: ellipsis;
170 | }
171 | 
172 | #devices>div>span>button {
173 |     float: right;
174 | }
175 | 
176 | .nodevices-msg {
177 |     display: none;
178 | }
179 | 
180 | .nodevices .nodevices-msg {
181 |     display: block;
182 | }
183 | 
184 | .loading-spinner {
185 | 	display: block;
186 |     padding: 10px;
187 | 	text-align: center;
188 | }
189 | 
190 | .loading-spinner::before {
191 | 	display: inline-block;
192 | 	width: 30px;
193 | 	height: 30px;
194 | 	border: 2px solid #aaa;
195 | 	content: "";
196 | 	animation: loading-animation 1s linear infinite;
197 | }
198 | @keyframes loading-animation {
199 | 	0%   { transform: rotate(0deg); }
200 | 	100% { transform: rotate(360deg); }
201 | }
202 | 


--------------------------------------------------------------------------------
/app/tsconfig.json:
--------------------------------------------------------------------------------
 1 | {
 2 |     "compilerOptions": {
 3 |         "target": "es2020",
 4 |         "module": "esnext",
 5 |         "strictNullChecks": false,
 6 |         "moduleResolution": "node",
 7 |         "noEmit": true,
 8 |     },
 9 |     "files": [
10 |         "global.d.ts",
11 |         "webrtc-rdp.js",
12 |         "rtcfilesystem-client.js",
13 |         "rtcfilesystem-server.js"
14 |     ]
15 | }
16 | 


--------------------------------------------------------------------------------
/app/webrtc-rdp.js:
--------------------------------------------------------------------------------
   1 | // @ts-check
   2 | 'use strict';
   3 | 
   4 | // Please replace with your id and signalingKey!
   5 | const signalingUrl = 'wss://ayame-labo.shiguredo.app/signaling';
   6 | const signalingKey = 'VV69g7Ngx-vNwNknLhxJPHs9FpRWWNWeUzJ9FUyylkD_yc_F';
   7 | const roomIdPrefix = 'binzume@rdp-room-';
   8 | const roomIdPinPrefix = 'binzume@rdp-pin-';
   9 | 
  10 | class Settings {
  11 |     static settingsKey = 'webrtc-rdp-settings';
  12 |     /** @type {((devices:DeviceSettings[])=>any)|null} */
  13 |     static onsettingsupdate = null;
  14 | 
  15 |     /**
  16 |      * @param {DeviceSettings} deviceInfo 
  17 |      */
  18 |     static addPeerDevice(deviceInfo) {
  19 |         let devices = this.getPeerDevices();
  20 |         let idx = devices.findIndex(d => d.roomId == deviceInfo.roomId);
  21 |         if (idx < 0) {
  22 |             devices.push(deviceInfo);
  23 |         } else {
  24 |             devices[idx] = deviceInfo;
  25 |         }
  26 |         this._save(devices);
  27 |     }
  28 | 
  29 |     /**
  30 |      * @returns {DeviceSettings[]}
  31 |      */
  32 |     static getPeerDevices() {
  33 |         try {
  34 |             let s = localStorage.getItem(this.settingsKey);
  35 |             if (!s) { return []; }
  36 |             let settings = JSON.parse(s);
  37 |             return settings.version == 2 ? (settings.devices || []) : [settings];
  38 |         } catch (e) {
  39 |             console.log(e);
  40 |             return [];
  41 |         }
  42 |     }
  43 | 
  44 |     /**
  45 |      * @param {DeviceSettings} deviceInfo 
  46 |      */
  47 |     static removePeerDevice(deviceInfo) {
  48 |         let devices = this.getPeerDevices();
  49 |         let filtered = devices.filter(d => d.roomId != deviceInfo.roomId);
  50 |         if (filtered.length != devices.length) {
  51 |             this._save(filtered);
  52 |         }
  53 |     }
  54 | 
  55 |     static clear() {
  56 |         this._save([]);
  57 |     }
  58 | 
  59 |     static _save(devices) {
  60 |         if (devices.length == 0) {
  61 |             localStorage.removeItem(this.settingsKey);
  62 |         } else if (devices.length == 1) {
  63 |             localStorage.setItem(this.settingsKey, JSON.stringify(devices[0])); // compat
  64 |         } else {
  65 |             localStorage.setItem(this.settingsKey, JSON.stringify({ devices: devices, version: 2 }));
  66 |         }
  67 |         this.onsettingsupdate && this.onsettingsupdate(devices);
  68 |     }
  69 | }
  70 | 
  71 | class BaseConnection {
  72 |     /**
  73 |      * @param {string} signalingUrl 
  74 |      * @param {string|undefined} signalingKey 
  75 |      * @param {string} roomId 
  76 |      */
  77 |     constructor(signalingUrl, signalingKey, roomId) {
  78 |         this.signalingUrl = signalingUrl;
  79 |         this.roomId = roomId;
  80 |         this.conn = null;
  81 |         /** @type {MediaStream|null} */
  82 |         this.mediaStream = null;
  83 |         this.stopTracksOnDisposed = true;
  84 |         /** @type {Record<string, DataChannelInfo>} */
  85 |         this.dataChannels = {};
  86 |         this.onstatechange = null;
  87 |         /** @type {'disconnected' | 'connecting' | 'waiting' | 'disposed' | 'connected'} */
  88 |         this.state = 'disconnected';
  89 |         this.options = Object.assign({}, Ayame.defaultOptions);
  90 |         this.options.video = Object.assign({}, this.options.video);
  91 |         this.options.audio = Object.assign({}, this.options.audio);
  92 |         this.options.signalingKey = signalingKey;
  93 |         this.reconnectWaitMs = -1;
  94 |         this.connectTimeoutMs = -1;
  95 |     }
  96 |     async connect() {
  97 |         if (this.conn || this.state == 'disposed') {
  98 |             throw 'invalid operation';
  99 |         }
 100 |         await this.setupConnection().connect(this.mediaStream, null);
 101 |     }
 102 |     setupConnection() {
 103 |         console.log("connecting..." + this.signalingUrl + " " + this.roomId);
 104 |         this.updateState('connecting');
 105 |         clearTimeout(this._connectTimer);
 106 |         if (this.connectTimeoutMs > 0) {
 107 |             this._connectTimer = setTimeout(() => this.disconnect(), this.connectTimeoutMs);
 108 |         }
 109 | 
 110 |         let conn = this.conn = Ayame.connection(this.signalingUrl, this.roomId, this.options, false);
 111 |         conn.on('open', async (e) => {
 112 |             for (let c of Object.keys(this.dataChannels)) {
 113 |                 this._handleDataChannel(await conn.createDataChannel(c));
 114 |             }
 115 |             this.updateState('waiting');
 116 |         });
 117 |         conn.on('connect', (e) => {
 118 |             clearTimeout(this._connectTimer);
 119 |             this.updateState('connected');
 120 |         });
 121 |         conn.on('datachannel', (channel) => {
 122 |             this._handleDataChannel(channel);
 123 |         });
 124 |         conn.on('disconnect', (e) => {
 125 |             this.conn = null;
 126 |             this.disconnect(e.reason);
 127 |         });
 128 |         return conn;
 129 |     }
 130 |     /**
 131 |      * @param {string|null} reason 
 132 |      */
 133 |     disconnect(reason = null) {
 134 |         console.log('disconnect', reason);
 135 |         clearTimeout(this._connectTimer);
 136 |         if (this.conn) {
 137 |             this.conn.on('disconnect', () => { });
 138 |             this.conn.disconnect();
 139 |             this.conn.stream = null;
 140 |             this.conn = null;
 141 |         }
 142 |         if (reason != 'dispose' && this.state != 'disconnected' && this.reconnectWaitMs >= 0) {
 143 |             this._connectTimer = setTimeout(() => this.connect(), this.reconnectWaitMs);
 144 |         }
 145 |         for (let c of Object.values(this.dataChannels)) {
 146 |             c.ch = null;
 147 |         }
 148 |         this.updateState('disconnected', reason);
 149 |     }
 150 |     dispose() {
 151 |         this.disconnect('dispose');
 152 |         this.updateState('disposed');
 153 |         this.stopTracksOnDisposed && this.mediaStream?.getTracks().forEach(t => t.stop());
 154 |         this.mediaStream = null;
 155 |     }
 156 |     /**
 157 |      * @param {'disconnected' | 'connecting' | 'waiting' | 'disposed' | 'connected'} s
 158 |      * @param {string|null} reason 
 159 |      */
 160 |     updateState(s, reason = null) {
 161 |         if (s != this.state && this.state != 'disposed') {
 162 |             console.log(this.roomId, s);
 163 |             let oldState = this.state;
 164 |             this.state = s;
 165 |             this.onstatechange && this.onstatechange(s, oldState, reason);
 166 |         }
 167 |     }
 168 |     /**
 169 |      * @param {RTCDataChannel|null} ch
 170 |      */
 171 |     _handleDataChannel(ch) {
 172 |         if (!ch) return;
 173 |         let c = this.dataChannels[ch.label];
 174 |         if (c && !c.ch) {
 175 |             console.log('datachannel', ch.label);
 176 |             c.ch = ch;
 177 |             ch.onmessage = (ev) => c.onmessage?.(ch, ev);
 178 |             // NOTE: dataChannel.onclose = null in Ayame web sdk.
 179 |             ch.addEventListener('open', (ev) => c.onopen?.(ch, ev));
 180 |             ch.addEventListener('close', (ev) => c.onclose?.(ch, ev));
 181 |         }
 182 |     }
 183 |     getFingerprint(remote = false) {
 184 |         let pc = this.conn._pc;
 185 |         let m = pc && (remote ? pc.currentRemoteDescription : pc.currentLocalDescription).sdp.match(/a=fingerprint:\s*([\w-]+ [a-f0-9:]+)/i);
 186 |         return m && m[1];
 187 |     }
 188 |     async hmacSha256(password, fingerprint) {
 189 |         let enc = new TextEncoder();
 190 |         let key = await crypto.subtle.importKey('raw', enc.encode(password),
 191 |             { name: 'HMAC', hash: { name: 'SHA-256' } }, false, ['sign']);
 192 |         let sign = await crypto.subtle.sign('HMAC', key, enc.encode(fingerprint));
 193 |         return btoa(String.fromCharCode(...new Uint8Array(sign)));
 194 |     }
 195 | }
 196 | 
 197 | class PairingConnection extends BaseConnection {
 198 |     /**
 199 |      * @param {string} signalingUrl 
 200 |      * @param {*} signalingKey 
 201 |      */
 202 |     constructor(signalingUrl, signalingKey = undefined) {
 203 |         super(signalingUrl, signalingKey, '');
 204 |         this.pinLength = 6;
 205 |         this.userAgent = navigator.userAgent;
 206 |         this.localServices = null;
 207 |         this.pinTimeoutMs = 3600000;
 208 |         this.version = 1;
 209 |     }
 210 | 
 211 |     validatePin(pin) {
 212 |         return pin && pin.length == this.pinLength;
 213 |     }
 214 | 
 215 |     async startPairing() {
 216 |         this.disconnect();
 217 |         this.connectTimeoutMs = this.pinTimeoutMs;
 218 |         let localRoomId = roomIdPrefix + this._generateSecret(16);
 219 |         let localToken = this._generateSecret(16);
 220 |         let pin = this._generatePin();
 221 | 
 222 |         this.dataChannels['secretExchange'] = {
 223 |             onopen: (ch, ev) => {
 224 |                 ch.send(JSON.stringify(this._getPairingInfo('hello', localRoomId, localToken)));
 225 |             },
 226 |             onmessage: (_ch, ev) => {
 227 |                 console.log('pairing event', ev.data);
 228 |                 let msg = JSON.parse(ev.data);
 229 |                 if (msg.type == 'credential') {
 230 |                     this._finishPairing(msg, localRoomId, localToken);
 231 |                 }
 232 |             },
 233 |         };
 234 |         this.roomId = roomIdPinPrefix + pin;
 235 |         await this.connect();
 236 |         return pin;
 237 |     }
 238 | 
 239 |     async sendPin(pin) {
 240 |         if (!this.validatePin(pin)) {
 241 |             throw "invalid pin";
 242 |         }
 243 |         this.disconnect();
 244 |         this.connectTimeoutMs = 10000;
 245 |         let localRoomId = roomIdPrefix + this._generateSecret(16);
 246 |         let localToken = this._generateSecret(16);
 247 | 
 248 |         this.dataChannels['secretExchange'] = {
 249 |             onmessage: (ch, ev) => {
 250 |                 console.log('pairing event', ev.data);
 251 |                 let msg = JSON.parse(ev.data);
 252 |                 if (msg.type == 'hello') {
 253 |                     if (msg.version && msg.version != this.version) {
 254 |                         console.log('Unsupported version: ' + msg.version);
 255 |                         this.disconnect();
 256 |                     }
 257 |                     ch.send(JSON.stringify(this._getPairingInfo('credential', localRoomId, localToken)));
 258 |                     this._finishPairing(msg, localRoomId, localToken);
 259 |                 }
 260 |             },
 261 |         };
 262 |         this.roomId = roomIdPinPrefix + pin;
 263 |         await this.connect();
 264 |     }
 265 | 
 266 |     _getPairingInfo(type, localRoomId, localToken) {
 267 |         let isGuest = this.localServices?.length == 0;
 268 |         return {
 269 |             type: type,
 270 |             roomId: localRoomId,
 271 |             token: isGuest ? null : localToken,
 272 |             signalingKey: isGuest ? null : signalingKey,
 273 |             services: this.localServices,
 274 |             userAgent: this.userAgent,
 275 |             version: this.version,
 276 |         };
 277 |     }
 278 | 
 279 |     _finishPairing(msg, localRoomId, localToken) {
 280 |         let isGuest = this.localServices?.length == 0;
 281 |         let isHost = this.localServices?.includes('no-client');
 282 |         Settings.addPeerDevice({
 283 |             roomId: msg.roomId || localRoomId,
 284 |             publishRoomId: localRoomId,
 285 |             localToken: localToken,
 286 |             token: msg.token,
 287 |             signalingKey: msg.signalingKey,
 288 |             services: msg.services || (isGuest ? ['no-client', 'screen', 'file'] : isHost ? [] : null),
 289 |             userAgent: msg.userAgent,
 290 |             name: msg.name,
 291 |         });
 292 |         this.disconnect();
 293 |     }
 294 | 
 295 |     _generatePin() {
 296 |         return (Math.floor(Math.random() * 1000000) + "000000").substring(0, this.pinLength);
 297 |     }
 298 | 
 299 |     _generateSecret(n) {
 300 |         const chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
 301 |         return Array.from(crypto.getRandomValues(new Uint8Array(n))).map((c) => chars[c % chars.length]).join('');
 302 |     }
 303 | }
 304 | 
 305 | class PublisherConnection extends BaseConnection {
 306 |     /**
 307 |      * @param {string} signalingUrl 
 308 |      * @param {string} roomId 
 309 |      * @param {MediaStream} mediaStream 
 310 |      * @param {DataChannelInfo|null} messageHandler 
 311 |      */
 312 |     constructor(signalingUrl, signalingKey, roomId, mediaStream, messageHandler = null) {
 313 |         super(signalingUrl, signalingKey, roomId);
 314 |         if (mediaStream) {
 315 |             this.options.video.direction = 'sendonly';
 316 |             this.options.audio.direction = 'sendonly';
 317 |             this.mediaStream = mediaStream;
 318 |         }
 319 |         this.reconnectWaitMs = 3000;
 320 |         this.dataChannels['controlEvent'] = messageHandler || {};
 321 |         this.onauth = null;
 322 |         this._originalWidth = 0;
 323 |         this._originalHeight = 0;
 324 |     }
 325 |     async auth(msg, ch, password, reply = false) {
 326 |         let result = false;
 327 |         if (msg.hmac) {
 328 |             let hmac = await this.hmacSha256(password, this.getFingerprint(true));
 329 |             result = msg.hmac == hmac;
 330 |         } else {
 331 |             result = msg.token == password;
 332 |         }
 333 |         console.log('Auth result', result);
 334 |         result && this.onauth?.();
 335 |         reply && ch.send(JSON.stringify({ type: 'authResult', result: result }));
 336 |         return result;
 337 |     }
 338 |     async updateVideoResolution(preferredWidth) {
 339 |         let video = this.mediaStream.getTracks().find(t => t.kind == 'video');
 340 |         if (video == null) {
 341 |             return false;
 342 |         }
 343 |         if (this._originalWidth <= 0 || this._originalHeight <= 0) {
 344 |             let settings = video.getSettings();
 345 |             this._originalWidth = settings.width;
 346 |             this._originalHeight = settings.height;
 347 |             if (this._originalWidth <= 0 || this._originalHeight <= 0) {
 348 |                 return false;
 349 |             }
 350 |         }
 351 |         let scale2 = Math.round(Math.log2(preferredWidth / this._originalWidth));
 352 |         let scale = Math.pow(2, scale2);
 353 |         console.log('video scale: ', scale);
 354 |         await video.applyConstraints({
 355 |             width: this._originalWidth * scale,
 356 |             height: this._originalHeight * scale,
 357 |         });
 358 |         return true;
 359 |     }
 360 | }
 361 | 
 362 | class PlayerConnection extends BaseConnection {
 363 |     /**
 364 |      * @param {string} signalingUrl 
 365 |      * @param {string} roomId 
 366 |      * @param {HTMLVideoElement|null} videoEl
 367 |      */
 368 |     constructor(signalingUrl, signalingKey, roomId, videoEl) {
 369 |         super(signalingUrl, signalingKey, roomId);
 370 |         if (videoEl) {
 371 |             this.options.video.direction = 'recvonly';
 372 |             this.options.audio.direction = 'recvonly';
 373 |         }
 374 |         this.videoEl = videoEl;
 375 |         this._rpcResultHandler = {};
 376 |         this.authToken = null;
 377 |         this.services = null;
 378 |         this.onauth = null;
 379 |         this.dataChannels['controlEvent'] = {
 380 |             onopen: async (ch, ev) => {
 381 |                 if (window.crypto?.subtle) {
 382 |                     let localFingerprint = this.getFingerprint(false);
 383 |                     if (!localFingerprint) {
 384 |                         console.log("Failed to get DTLS cert fingerprint");
 385 |                         return;
 386 |                     }
 387 |                     console.log("local fingerprint:", localFingerprint);
 388 |                     let hmac = this.authToken && await this.hmacSha256(this.authToken, localFingerprint);
 389 |                     ch.send(JSON.stringify({
 390 |                         type: "auth",
 391 |                         requestServices: videoEl ? ['screen', 'file'] : ['file'],
 392 |                         fingerprint: localFingerprint,
 393 |                         hmac: hmac
 394 |                     }));
 395 |                 } else {
 396 |                     ch.send(JSON.stringify({ type: "auth", token: this.authToken }));
 397 |                 }
 398 |             },
 399 |             onmessage: (ch, ev) => {
 400 |                 let msg = JSON.parse(ev.data);
 401 |                 if (msg.type == 'redirect' && msg.roomId) {
 402 |                     this.disconnect('redirect');
 403 |                     this.roomId = msg.roomId;
 404 |                     this.connect();
 405 |                 } else if (msg.type == 'auth') {
 406 |                     // player and player error
 407 |                     this.disconnect();
 408 |                 } else if (msg.type == 'authResult') {
 409 |                     this.services = msg.services;
 410 |                     this.onauth?.(msg.result);
 411 |                 } else if (msg.type == 'rpcResult') {
 412 |                     this._rpcResultHandler[msg.reqId]?.(msg);
 413 |                 }
 414 |             }
 415 |         };
 416 |     }
 417 |     setupConnection() {
 418 |         let conn = super.setupConnection();
 419 |         conn.on('addstream', (ev) => {
 420 |             this.mediaStream = ev.stream;
 421 |             if (this.videoEl) {
 422 |                 this.videoEl.srcObject = ev.stream;
 423 |             }
 424 |         });
 425 |         return conn;
 426 |     }
 427 |     /**
 428 |      * @param {string|null} reason 
 429 |      */
 430 |     disconnect(reason = null) {
 431 |         if (this.videoEl && this.videoEl.srcObject == this.mediaStream) {
 432 |             this.videoEl.srcObject = null;
 433 |         }
 434 |         super.disconnect(reason);
 435 |     }
 436 |     sendRpcAsync(name, params, timeoutMs = 10000) {
 437 |         let reqId = Date.now(); // TODO: monotonic
 438 |         this.dataChannels['controlEvent'].ch?.send(JSON.stringify({ type: 'rpc', name: name, reqId: reqId, params: params }));
 439 |         return new Promise((resolve, reject) => {
 440 |             let timer = setTimeout(() => {
 441 |                 delete this._rpcResultHandler[reqId];
 442 |                 reject('timeout');
 443 |             }, timeoutMs);
 444 |             this._rpcResultHandler[reqId] = (res) => {
 445 |                 clearTimeout(timer);
 446 |                 delete this._rpcResultHandler[reqId];
 447 |                 resolve(res.value);
 448 |             };
 449 |         });
 450 |     }
 451 |     sendMouseEvent(action, x, y, button) {
 452 |         if (this.state == 'connected') {
 453 |             this.dataChannels['controlEvent'].ch?.send(JSON.stringify({ type: 'mouse', action: action, x: x, y: y, button: button }));
 454 |         }
 455 |     }
 456 |     sendKeyEvent(action, key, code, shift = false, ctrl = false, alt = false) {
 457 |         if (this.state == 'connected') {
 458 |             this.dataChannels['controlEvent'].ch?.send(JSON.stringify({ type: 'key', action: action, key: key, code: code, shift: shift, ctrl: ctrl, alt: alt }));
 459 |         }
 460 |     }
 461 | }
 462 | 
 463 | class ConnectionManager {
 464 |     /**
 465 |      * @param {DeviceSettings} settings 
 466 |      */
 467 |     constructor(settings) {
 468 |         this.settings = settings;
 469 |         this.onadded = null;
 470 |         this.disableAuth = !settings.localToken;
 471 |         /**  @type {ConnectionInfo[]} */
 472 |         this._connections = [];
 473 |     }
 474 | 
 475 |     /**
 476 |      * @param {MediaStream} mediaStream 
 477 |      * @param {string|null} name 
 478 |      * @param {DataChannelInfo|null} messageHandler 
 479 |      * @param {boolean} connect 
 480 |      * @param {boolean} permanent 
 481 |      * @returns {ConnectionInfo}
 482 |      */
 483 |     addStream(mediaStream, messageHandler = null, name = null, connect = true, permanent = true) {
 484 |         let id = this._genId();
 485 |         let roomId = this.settings.publishRoomId || this.settings.roomId;
 486 |         name = name || mediaStream.getVideoTracks()[0]?.label || mediaStream.id;
 487 |         let conn = new PublisherConnection(signalingUrl, signalingKey, roomId + (id != 1 ? '.' + id : ''), mediaStream, messageHandler);
 488 |         conn.connectTimeoutMs = permanent ? -1 : 30000;
 489 |         conn.reconnectWaitMs = permanent ? 2000 : -1;
 490 | 
 491 |         let info = { conn: conn, id: id, name: name, permanent: permanent };
 492 |         this._connections.push(info);
 493 |         this.onadded?.(info);
 494 |         if (connect) {
 495 |             conn.connect();
 496 |         }
 497 |         return info;
 498 |     }
 499 | 
 500 |     removeStream(id) {
 501 |         let index = this._connections.findIndex(c => c.id == id);
 502 |         if (index >= 0) {
 503 |             this._connections[index].conn.dispose();
 504 |             this._connections.splice(index, 1);
 505 |         }
 506 |     }
 507 |     async auth(msg, ch, reply = false) {
 508 |         let c = this._connections.find(c => c.conn.dataChannels['controlEvent']?.ch == ch);
 509 |         return await c?.conn.auth(msg, ch, this.settings.localToken, reply);
 510 |     }
 511 |     _genId() {
 512 |         let n = 1;
 513 |         while (this._connections.some(c => c.id == n)) n++;
 514 |         return n;
 515 |     }
 516 |     dispose() {
 517 |         this._connections.forEach((c) => c.conn.dispose());
 518 |         this._connections = [];
 519 |     }
 520 | }
 521 | 
 522 | class StreamSelectScreen {
 523 |     /**
 524 |      * @param {StreamProvider} streamProvider 
 525 |      */
 526 |     constructor(streamProvider) {
 527 |         let canvas = this.canvasEl = document.createElement('canvas');
 528 |         canvas.width = 640;
 529 |         canvas.height = 400;
 530 |         this.streamProvider = streamProvider;
 531 |         this.ctx = canvas.getContext('2d');
 532 |         /** @type {StreamSpec[]} */
 533 |         this._streams = [];
 534 |         this.buttonSpec = { width: 520, height: 20, font: 'bold 18px sans-serif', color: 'black' };
 535 |         this.buttonLayout = { top: 24, left: (canvas.width - this.buttonSpec.width) / 2, spacing: 6 };
 536 |         this._attachCount = 0;
 537 |         this._updateTimer = null;
 538 |     }
 539 |     async update() {
 540 |         let streams = await this.streamProvider.getStreams();
 541 |         this._streams = streams;
 542 |         let canvas = this.canvasEl;
 543 |         let ctx = this.ctx;
 544 |         ctx.fillStyle = 'white';
 545 |         ctx.fillRect(0, 0, canvas.width, canvas.height);
 546 |         ctx.font = 'normal 18px sans-serif';
 547 |         ctx.textBaseline = 'top';
 548 |         ctx.textAlign = 'center';
 549 |         ctx.fillStyle = 'black';
 550 |         ctx.fillText('Available screens (Click to select)', canvas.width / 2, 0);
 551 |         if (streams.length == 0) {
 552 |             ctx.fillStyle = 'red';
 553 |             ctx.fillText('No Available Screen', canvas.width / 2, canvas.height / 2);
 554 |         }
 555 | 
 556 |         let button = this.buttonSpec, layout = this.buttonLayout;
 557 |         ctx.font = button.font;
 558 |         streams.forEach((s, i) => {
 559 |             let x = layout.left, y = layout.top + i * (button.height + layout.spacing);
 560 |             ctx.strokeStyle = '#888';
 561 |             ctx.strokeRect(x, y, button.width, button.height);
 562 |             ctx.fillStyle = button.color;
 563 |             ctx.fillText(s.name, canvas.width / 2, y, button.width);
 564 |         });
 565 |     }
 566 | 
 567 |     /**
 568 |      * @param {ConnectionManager} cm
 569 |      * @param {StreamSpec} s 
 570 |      * @param {boolean} permanent 
 571 |      */
 572 |     async startStream(cm, s, permanent) {
 573 |         let self = this;
 574 |         let mediaStream = this.canvasEl.captureStream(1);
 575 |         let dataChannelInfo = {
 576 |             onopen(ch, _ev) { self._attach(cm, ch); },
 577 |             onclose(_ch, _ev) { self._detach(); },
 578 |             onmessage(ch, ev) { self._handleMessage(cm, ch, JSON.parse(ev.data)) },
 579 |         };
 580 |         return await cm.addStream(mediaStream, dataChannelInfo, s.name, true, permanent);
 581 |     }
 582 | 
 583 |     /**
 584 |      * @param {ConnectionManager} cm 
 585 |      * @param {RTCDataChannel} ch 
 586 |      * @param {object} msg 
 587 |      */
 588 |     async _handleMessage(cm, ch, msg) {
 589 |         if (msg.type == 'mouse' && (msg.action == 'click' || msg.action == 'up')) {
 590 |             let x = msg.x * this.canvasEl.width, y = msg.y * this.canvasEl.height;
 591 |             let layout = this.buttonLayout;
 592 |             if (x < layout.left || x > layout.left + this.buttonSpec.width || y < layout.top) {
 593 |                 return;
 594 |             }
 595 |             let n = Math.floor((y - layout.top) / (this.buttonSpec.height + layout.spacing));
 596 |             if (this._streams[n]) {
 597 |                 this._redirect(cm, ch, n);
 598 |             }
 599 |         } else if (msg.type == 'auth') {
 600 |             let authResult = await cm.auth(msg, ch);
 601 |             if (authResult && msg.requestServices && !msg.requestServices.includes('screen')) {
 602 |                 let c = await cm.addStream(null, {
 603 |                     onmessage: async (ch, ev) => {
 604 |                         let msg = JSON.parse(ev.data);
 605 |                         if (msg.type == 'auth') {
 606 |                             await cm.auth(msg, ch, true);
 607 |                         }
 608 |                     },
 609 |                 }, 'file server', true, false);
 610 |                 if (c) {
 611 |                     ch.send(JSON.stringify({ type: 'redirect', 'roomId': c.conn.roomId }));
 612 |                 }
 613 |             } else if (authResult) {
 614 |                 if (this._attachCount == 1) {
 615 |                     await this.update();
 616 |                 }
 617 |                 this._streams.length == 1 && this._redirect(cm, ch, 0);
 618 |             }
 619 |         }
 620 |     }
 621 |     async _attach(cm, ch) {
 622 |         this._attachCount++;
 623 |         if (this._attachCount == 1) {
 624 |             this._updateTimer = setInterval(() => this.update(), 1000);
 625 |         }
 626 |     }
 627 |     _detach() {
 628 |         this._attachCount--;
 629 |         if (this._attachCount == 0) {
 630 |             clearInterval(this._updateTimer);
 631 |             this.ctx.fillRect(0, 0, this.canvasEl.width, this.canvasEl.height);
 632 |         }
 633 |     }
 634 |     async _redirect(cm, ch, n) {
 635 |         let c = await this.streamProvider.startStream(cm, this._streams[n], false);
 636 |         if (c) {
 637 |             ch.send(JSON.stringify({ type: 'redirect', 'roomId': c.conn.roomId }));
 638 |         }
 639 |     }
 640 | }
 641 | 
 642 | class StreamRedirector {
 643 |     /**
 644 |      * @param {StreamProvider} streamProvider 
 645 |      * @param {StreamSpec} target 
 646 |      */
 647 |     constructor(streamProvider, target) {
 648 |         this.streamProvider = streamProvider;
 649 |         this.target = target;
 650 |     }
 651 | 
 652 |     /**
 653 |      * @param {ConnectionManager} cm
 654 |      * @param {StreamSpec} s 
 655 |      * @param {boolean} permanent 
 656 |      */
 657 |     async startStream(cm, s, permanent) {
 658 |         let dataChannelInfo = {
 659 |             onopen: async (ch, ev) => {
 660 |                 // TODO: timeout
 661 |                 let c = await this.streamProvider.startStream(cm, this.target, false);
 662 |                 ch.send(JSON.stringify({ type: 'redirect', 'roomId': c.conn.roomId }));
 663 |             }
 664 |         };
 665 |         return await cm.addStream(null, dataChannelInfo, s.name, true, permanent);
 666 |     }
 667 | }
 668 | 
 669 | class BrowserStreamProvider {
 670 |     constructor() {
 671 |         /** @type {((target: any, ev: Record<string,any>) => void)|null} */
 672 |         this.sendInputEvent = null;
 673 |         /** @type {Record<string, StreamProvider>} */
 674 |         this.pseudoStreams = {};
 675 |         /** @type {{spec:StreamSpec, isCamera: boolean, mediaStream: MediaStream}[]} */
 676 |         this._streams = [];
 677 |         this._idSeq = 0;
 678 |     }
 679 | 
 680 |     /**
 681 |      * @param {boolean} camera 
 682 |      * @param {boolean} registerStream
 683 |      * @returns {Promise<StreamSpec>}
 684 |      */
 685 |     async addMediaStream(camera = false, registerStream = true) {
 686 |         let mediaStream = await (camera ? navigator.mediaDevices.getUserMedia({ audio: true, video: true }) : navigator.mediaDevices.getDisplayMedia({ audio: true, video: true }));
 687 |         let name = mediaStream.getVideoTracks()[0]?.label || "?";
 688 |         let s = { spec: { id: 'BrowserStreamProvider_' + (++this._idSeq), name: name }, mediaStream: mediaStream, isCamera: camera };
 689 |         registerStream && this._streams.push(s);
 690 |         return s.spec;
 691 |     }
 692 | 
 693 |     async getStreams() {
 694 |         return this._streams.map(s => s.spec);
 695 |     }
 696 | 
 697 |     /**
 698 |      * @param {ConnectionManager} cm
 699 |      * @param {StreamSpec} s 
 700 |      * @param {boolean} permanent 
 701 |      */
 702 |     async startStream(cm, s, permanent = false) {
 703 |         if (this.pseudoStreams[s.id]) {
 704 |             return await this.pseudoStreams[s.id].startStream(cm, s, permanent);
 705 |         }
 706 |         let stream = this._streams.find(st => st.spec.id == s.id);
 707 |         if (!stream) {
 708 |             return null;
 709 |         }
 710 |         let target = stream.isCamera ? null : this._getTarget(stream.mediaStream);
 711 |         let authorized = cm.disableAuth;
 712 |         let c = cm.addStream(stream.mediaStream, {
 713 |             onmessage: async (ch, ev) => {
 714 |                 let msg = JSON.parse(ev.data);
 715 |                 if (msg.type == 'auth') {
 716 |                     authorized = await cm.auth(msg, ch, true);
 717 |                 } else if (authorized && this.sendInputEvent) {
 718 |                     this.sendInputEvent(target, msg);
 719 |                 }
 720 |             },
 721 |         }, null, true, permanent);
 722 |         c.conn.stopTracksOnDisposed = false; // Reuse media streams.
 723 |         return c;
 724 |     }
 725 | 
 726 |     /**
 727 |      * @param {StreamSpec} s 
 728 |      */
 729 |     remove(s) {
 730 |         this._streams.filter(ss => ss.spec.id == s.id).forEach(ss => ss.mediaStream.getTracks().forEach(t => t.stop()));
 731 |         this._streams = this._streams.filter(ss => ss.spec.id != s.id);
 732 |     }
 733 | 
 734 |     _getTarget(mediaStream) {
 735 |         let surface = mediaStream.getVideoTracks()[0]?.getSettings().displaySurface;
 736 |         let label = mediaStream.getVideoTracks()[0]?.label;
 737 |         if (surface == null || label == null) {
 738 |             // TODO: Firefox
 739 |             return { type: 'monitor', id: 0 };
 740 |         }
 741 |         if (surface == 'monitor') {
 742 |             let m = label.match(/^screen:(\d+):\d+/);
 743 |             if (m) {
 744 |                 return { type: surface, id: m[1] | 0 };
 745 |             }
 746 |         } else if (surface == 'window') {
 747 |             let m = label.match(/^window:(\d+):\d+/);
 748 |             if (m) {
 749 |                 return { type: surface, id: m[1] | 0 };
 750 |             }
 751 |         }
 752 |         return null;
 753 |     }
 754 | }
 755 | 
 756 | class ElectronStreamProvider {
 757 |     constructor(types = ['screen', 'window']) {
 758 |         this._lastMouseMoveTime = 0;
 759 |         this.streamTypes = types;
 760 |         /** @type {Record<string, StreamProvider>} */
 761 |         this.pseudoStreams = {};
 762 |     }
 763 |     /**
 764 |      * @returns {Promise<StreamSpec[]>}
 765 |      */
 766 |     async getStreams(all = false) {
 767 |         return await RDP.getDisplayStreams(all ? ['screen', 'window'] : this.streamTypes);
 768 |     }
 769 |     /**
 770 |      * @param {StreamSpec} s 
 771 |      */
 772 |     async getMediaStream(s) {
 773 |         return await navigator.mediaDevices.getUserMedia({
 774 |             video: {
 775 |                 // @ts-ignore
 776 |                 mandatory: {
 777 |                     chromeMediaSource: 'desktop',
 778 |                     chromeMediaSourceId: s.id,
 779 |                     maxWidth: 1920,
 780 |                     maxHeight: 1080,
 781 |                 }
 782 |             },
 783 |             audio: s.hasAudio ? {
 784 |                 // @ts-ignore
 785 |                 mandatory: {
 786 |                     chromeMediaSource: 'desktop',
 787 |                 }
 788 |             } : false
 789 |         });
 790 |     }
 791 |     /**
 792 |      * @param {ConnectionManager} cm
 793 |      * @param {StreamSpec} s 
 794 |      * @param {boolean} permanent 
 795 |      */
 796 |     async startStream(cm, s, permanent = false) {
 797 |         try {
 798 |             if (this.pseudoStreams[s.id]) {
 799 |                 return await this.pseudoStreams[s.id].startStream(cm, s, permanent);
 800 |             }
 801 |             let mediaStream = await this.getMediaStream(s);
 802 |             let authorized = cm.disableAuth;
 803 |             let c = cm.addStream(mediaStream, {
 804 |                 onmessage: async (ch, ev) => {
 805 |                     let msg = JSON.parse(ev.data);
 806 |                     if (msg.type == 'auth') {
 807 |                         authorized = await cm.auth(msg, ch, true);
 808 |                         if (s.name && s.name != 'unknown' && authorized) {
 809 |                             ch.send(JSON.stringify({ type: 'streamInfo', title: s.name }));
 810 |                         }
 811 |                     } else if (authorized) {
 812 |                         this._handleMessage(cm, s, ch, msg, c)
 813 |                     }
 814 |                 }
 815 |             }, s.name, true, permanent);
 816 |             return c;
 817 |         } catch (e) {
 818 |             console.log(e);
 819 |             return null;
 820 |         }
 821 |     }
 822 |     /**
 823 |      * @param {ConnectionManager} cm 
 824 |      * @param {StreamSpec} s 
 825 |      * @param {RTCDataChannel} ch 
 826 |      * @param {ConnectionInfo} c
 827 |      * @param {object} msg 
 828 |      */
 829 |     async _handleMessage(cm, s, ch, msg, c) {
 830 |         if (msg.type == 'mouse') {
 831 |             let now = Date.now();
 832 |             if (now - this._lastMouseMoveTime < 10 && msg.action == 'move') {
 833 |                 return;
 834 |             }
 835 |             this._lastMouseMoveTime = now;
 836 |             await RDP.sendMouse({ target: s, action: msg.action, button: msg.button, x: msg.x, y: msg.y });
 837 |         } else if (msg.type == 'key') {
 838 |             let modifiers = msg.modifiers || [];
 839 |             msg.ctrl && modifiers.push('Control');
 840 |             msg.alt && modifiers.push('Alt');
 841 |             msg.shift && modifiers.push('Shift');
 842 |             await RDP.sendKey({ target: s, action: msg.action, key: msg.key, modifiers: modifiers });
 843 |         } else if (msg.type == 'rpc' && msg.name == 'getStreams') {
 844 |             let streams = await this.getStreams();
 845 |             ch.send(JSON.stringify({ type: 'rpcResult', name: msg.name, reqId: msg.reqId, value: streams.map(s => ({ id: s.id, name: s.name })) }));
 846 |         } else if (msg.type == 'rpc' && msg.name == 'streamFromPoint') {
 847 |             let si = await RDP.streamFromPoint({ target: s, x: msg.params.x, y: msg.params.y });
 848 |             ch.send(JSON.stringify({ type: 'rpcResult', name: msg.name, reqId: msg.reqId, value: si }));
 849 |         } else if (msg.type == 'rpc' && msg.name == 'play') {
 850 |             let streams = await this.getStreams(true);
 851 |             let s = streams.find(s => s.id == msg.params.streamId) || { id: msg.params.streamId, name: 'unknown' };
 852 |             let c = await this.startStream(cm, s);
 853 |             if (msg.params.redirect) {
 854 |                 ch.send(JSON.stringify({ type: 'redirect', reqId: msg.reqId, roomId: c?.conn.roomId }));
 855 |             }
 856 |             ch.send(JSON.stringify({ type: 'rpcResult', name: msg.name, reqId: msg.reqId, value: { roomId: c?.conn.roomId } }));
 857 |         } else if (msg.type == 'rpc' && msg.name == 'setResolution') {
 858 |             let r = await c.conn.updateVideoResolution(msg.params.preferredWidth);
 859 |             ch.send(JSON.stringify({ type: 'rpcResult', name: msg.name, reqId: msg.reqId, value: r }));
 860 |         } else {
 861 |             console.log("drop:", msg);
 862 |         }
 863 |     }
 864 | }
 865 | 
 866 | 
 867 | function initPairing() {
 868 |     let pairing = new PairingConnection(signalingUrl, signalingKey);
 869 |     document.getElementById('addDeviceButton').addEventListener('click', (ev) => {
 870 |         pairing.disconnect();
 871 |         document.getElementById("pairing").style.display =
 872 |             document.getElementById("pairing").style.display == 'none' ? 'block' : 'none';
 873 |         document.getElementById("pinDisplayBox").style.display = "none";
 874 |         document.getElementById("pinInputBox").style.display = "block";
 875 |     });
 876 |     document.getElementById('inputPin').addEventListener('click', (ev) => {
 877 |         document.getElementById("pinDisplayBox").style.display = "none";
 878 |         document.getElementById("pinInputBox").style.display = "block";
 879 |     });
 880 |     document.getElementById('pinInputBox').addEventListener('submit', (ev) => {
 881 |         let inputEl = /** @type {HTMLInputElement} */(document.getElementById("pinInput"));
 882 |         let pin = inputEl.value.trim();
 883 |         if (pairing.validatePin(pin)) {
 884 |             document.getElementById("pinInputBox").style.display = "none";
 885 |             inputEl.value = '';
 886 |             pairing.localServices = null;
 887 |             pairing.sendPin(pin);
 888 |         }
 889 |         ev.preventDefault();
 890 |     });
 891 |     document.getElementById('generatePin').addEventListener('click', async (ev) => {
 892 |         document.getElementById("pinDisplayBox").style.display = "block";
 893 |         document.getElementById("pinInputBox").style.display = "none";
 894 |         let pinEl = document.getElementById("pin");
 895 |         pinEl.innerText = "......";
 896 |         pinEl.innerText = await pairing.startPairing();
 897 |         pairing.onstatechange = (state) => {
 898 |             let mode = /** @type {HTMLSelectElement} */(document.getElementById('modeSelect')).value;
 899 |             pairing.localServices = mode == 'guest' ? [] :
 900 |                 mode == 'host' ? ['no-client', 'screen', 'file'] : ['screen', 'file'];
 901 |             if (state == "disconnected") {
 902 |                 pinEl.innerText = "......";
 903 |             }
 904 |         };
 905 |     });
 906 | }
 907 | 
 908 | function initPlayer() {
 909 |     // Player
 910 |     /** @type {PlayerConnection|null} */
 911 |     let player = null;
 912 |     /** @type {HTMLVideoElement} */
 913 |     let videoEl = document.querySelector('#screen');
 914 |     /** @type {DeviceSettings} */
 915 |     let currentDevice = null;
 916 |     let playStream = (/** @type {DeviceSettings} */ d) => {
 917 |         currentDevice = d;
 918 |         player?.disconnect();
 919 |         videoEl.style.display = "none";
 920 |         if (currentDevice) {
 921 |             document.getElementById('connectingBox').style.display = "block";
 922 |             document.body.classList.add('player');
 923 |             let roomId = currentDevice.roomId;
 924 |             player = new PlayerConnection(signalingUrl, currentDevice.signalingKey, roomId, videoEl);
 925 |             player.authToken = currentDevice.token;
 926 |             if (globalThis.rtcFileSystemManager) {
 927 |                 globalThis.storageAccessors ??= {};
 928 |                 // defined in ../app/rtcfilesystem-client.js
 929 |                 player.dataChannels['fileServer'] = globalThis.rtcFileSystemManager.getRtcChannelSpec('RDP-' + roomId, 'files');
 930 |             }
 931 |             player.onstatechange = (state) => {
 932 |                 if (state == "connected") {
 933 |                     document.getElementById('connectingBox').style.display = "none";
 934 |                     videoEl.style.display = "block";
 935 |                 }
 936 |             };
 937 |             player.connect();
 938 |         }
 939 |     };
 940 |     let dragging = false;
 941 |     let dragTimer = null;
 942 |     let mousePos = { x: 0, y: 0 };
 943 |     let updateMousePos = (ev) => {
 944 |         let rect = videoEl.getBoundingClientRect();
 945 |         let vw = Math.min(rect.width, rect.height * videoEl.videoWidth / videoEl.videoHeight);
 946 |         let vh = Math.min(rect.height, rect.width * videoEl.videoHeight / videoEl.videoWidth);
 947 |         let x = (ev.clientX - rect.left - (rect.width - vw) / 2) / vw, y = (ev.clientY - rect.top - (rect.height - vh) / 2) / vh;
 948 |         mousePos = { x: x, y: y };
 949 |     };
 950 |     let sendMouse = (action, ev) => {
 951 |         if (player?.state == 'connected') {
 952 |             updateMousePos(ev);
 953 |             if (action == 'click' && cancelSelect) {
 954 |                 (async () => {
 955 |                     let stream = await player.sendRpcAsync('streamFromPoint', { x: mousePos.x, y: mousePos.y });
 956 |                     if (stream && cancelSelect) {
 957 |                         player.sendRpcAsync('play', { streamId: stream.id, redirect: true });
 958 |                     }
 959 |                     cancelSelect?.();
 960 |                 })();
 961 |                 return;
 962 |             }
 963 |             let x = mousePos.x, y = mousePos.y;
 964 |             if (action != 'up' && (x > 1 || x < 0 || y > 1 || y < 0)) action = 'move';
 965 |             if (action == 'click' && dragging) action = 'up';
 966 |             player.sendMouseEvent(action, x, y, ev.button);
 967 |             ev.preventDefault();
 968 |         }
 969 |     };
 970 |     videoEl.addEventListener('click', (ev) => sendMouse('click', ev));
 971 |     videoEl.addEventListener('auxclick', (ev) => sendMouse('click', ev));
 972 |     videoEl.addEventListener('mousemove', (ev) => updateMousePos(ev));
 973 |     videoEl.addEventListener('pointerdown', (ev) => {
 974 |         videoEl.setPointerCapture(ev.pointerId);
 975 |         dragTimer = setTimeout(() => {
 976 |             dragging = true;
 977 |             sendMouse('down', ev);
 978 |         }, 200);
 979 |     });
 980 |     videoEl.addEventListener('pointermove', (ev) => dragging && sendMouse('move', ev));
 981 |     videoEl.addEventListener('pointerup', (ev) => {
 982 |         videoEl.releasePointerCapture(ev.pointerId);
 983 |         clearTimeout(dragTimer);
 984 |         if (dragging) {
 985 |             dragging = false;
 986 |             sendMouse('up', ev);
 987 |             let cancelClick = ev => ev.stopPropagation();
 988 |             window.addEventListener('click', cancelClick, true);
 989 |             setTimeout(() => window.removeEventListener('click', cancelClick, true), 10);
 990 |         }
 991 |     });
 992 |     videoEl.addEventListener('contextmenu', (ev) => {
 993 |         ev.preventDefault();
 994 |         if (!dragging && ev.button === -1) {
 995 |             // Oculus Quest B button fires contextmenu event w/o pointerdown/up.
 996 |             sendMouse('click', { button: 2, clientX: ev.clientX, clientY: ev.clientY, preventDefault: () => { } });
 997 |         }
 998 |     });
 999 | 
1000 |     let modKeyState = {};
1001 |     let updateModKeyState = (key, state) => {
1002 |         let modkey = ['Shift', 'Control', 'Alt'].includes(key);
1003 |         if (modkey) {
1004 |             modKeyState[key] = state;
1005 |             let el = document.getElementById('toggle' + key + 'Button');
1006 |             state ? el?.classList.add('active') : el?.classList.remove('active');
1007 |         }
1008 |         return modkey;
1009 |     };
1010 |     document.addEventListener('keydown', (ev) => {
1011 |         let modkey = updateModKeyState(ev.key, true);
1012 |         player?.sendKeyEvent(modkey ? 'down' : 'press', ev.key, ev.code, modKeyState['Shift'], modKeyState['Control'], modKeyState['Alt']);
1013 |         if (player) {
1014 |             ev.preventDefault();
1015 |         }
1016 |     });
1017 |     document.addEventListener('keyup', (ev) => {
1018 |         if (updateModKeyState(ev.key, false)) {
1019 |             player?.sendKeyEvent('up', ev.key, ev.code, modKeyState['Shift'], modKeyState['Control'], modKeyState['Alt']);
1020 |             if (player) {
1021 |                 ev.preventDefault();
1022 |             }
1023 |         }
1024 |     });
1025 |     document.getElementById('playButton')?.addEventListener('click', (ev) => playStream(currentDevice));
1026 |     let fullscreenButtonEl = document.getElementById('fullscreenButton');
1027 |     fullscreenButtonEl?.addEventListener('click', (ev) =>
1028 |         document.fullscreenElement ?
1029 |             document.exitFullscreen() :
1030 |             document.getElementById('player').requestFullscreen());
1031 |     document.addEventListener('fullscreenchange', (ev) => {
1032 |         document.fullscreenElement ?
1033 |             fullscreenButtonEl.classList.add('active') :
1034 |             fullscreenButtonEl.classList.remove('active');
1035 |     });
1036 |     for (let key of ['Shift', 'Control', 'Alt']) {
1037 |         let el = document.getElementById('toggle' + key + 'Button');
1038 |         el?.addEventListener('click', (ev) => {
1039 |             let down = !el.classList.contains('active');
1040 |             updateModKeyState(key, down);
1041 |             player?.sendKeyEvent(down ? 'down' : 'up', key, key + 'Left', modKeyState['Shift'], modKeyState['Control'], modKeyState['Alt']);
1042 |         });
1043 |     }
1044 |     document.getElementById('closePlayerButton')?.addEventListener('click', (ev) => {
1045 |         player?.disconnect();
1046 |         document.body.classList.remove('player');
1047 |     });
1048 |     let muteEl = document.getElementById('muteButton');
1049 |     if (muteEl) {
1050 |         videoEl.muted ? muteEl.classList.add('active') : muteEl.classList.remove('active');
1051 |         muteEl.addEventListener('click', (ev) => {
1052 |             videoEl.muted = muteEl.classList.toggle('active');
1053 |         });
1054 |     }
1055 |     let cancelSelect = null;
1056 |     document.getElementById('selectWindowButton')?.addEventListener('click', (ev) => {
1057 |         if (cancelSelect) {
1058 |             cancelSelect();
1059 |         } else {
1060 |             document.getElementById('selectWindowButton').classList.add('active');
1061 |             let boxEl = document.createElement('div');
1062 |             boxEl.className = 'streamrect';
1063 |             videoEl.parentElement.append(boxEl);
1064 |             let selectWindowTimer = setInterval(async () => {
1065 |                 if (player?.state != 'connected') {
1066 |                     cancelSelect?.();
1067 |                     return;
1068 |                 }
1069 |                 let stream = await player.sendRpcAsync('streamFromPoint', { x: mousePos.x, y: mousePos.y });
1070 |                 if (!cancelSelect || !stream) {
1071 |                     return;
1072 |                 }
1073 |                 let rect = videoEl.getBoundingClientRect();
1074 |                 let vw = Math.min(rect.width, rect.height * videoEl.videoWidth / videoEl.videoHeight);
1075 |                 let vh = Math.min(rect.height, rect.width * videoEl.videoHeight / videoEl.videoWidth);
1076 |                 boxEl.style.width = (vw * stream.rect.width) + 'px';
1077 |                 boxEl.style.height = (vh * stream.rect.height) + 'px';
1078 |                 boxEl.style.left = (vw * stream.rect.x + rect.left + (rect.width - vw) / 2) + 'px';
1079 |                 boxEl.style.top = (vh * stream.rect.y + rect.top + (rect.height - vh) / 2) + 'px';
1080 |                 boxEl.textContent = stream.id;
1081 |             }, 500);
1082 |             cancelSelect = () => {
1083 |                 clearInterval(selectWindowTimer);
1084 |                 boxEl.parentElement.removeChild(boxEl);
1085 |                 cancelSelect = null;
1086 |                 document.getElementById('selectWindowButton').classList.remove('active');
1087 |             };
1088 |         }
1089 |     });
1090 |     return playStream;
1091 | }
1092 | 
1093 | function initPublisher(playStream) {
1094 |     /**
1095 |      * @template {keyof HTMLElementTagNameMap} T
1096 |      * @param {T} tag 
1097 |      * @param {string[] | string | Node[] | any} children 
1098 |      * @param {object | function} [attrs]
1099 |      * @returns {HTMLElementTagNameMap[T]}
1100 |      */
1101 |     let mkEl = (tag, children, attrs) => {
1102 |         let el = document.createElement(tag);
1103 |         children && el.append(...[children].flat(999));
1104 |         attrs instanceof Function ? attrs(el) : (attrs && Object.assign(el, attrs));
1105 |         return el;
1106 |     };
1107 | 
1108 |     let isElectronApp = globalThis.RDP != null;
1109 |     if (isElectronApp) {
1110 |         document.body.classList.add('standalone');
1111 |     }
1112 |     document.querySelector('#clearSettingsButton').addEventListener('click', (ev) => confirm('CLear all settings?') && Settings.clear());
1113 | 
1114 |     // Publisher
1115 |     /** @typedef {{cm: ConnectionManager, streamProvider?: StreamProvider&Record<string,any>, el: HTMLElement}} DeviceState */
1116 |     /** @type {(ds: DeviceState)=>any} */
1117 |     let initStreams = (_) => { };
1118 |     /** @type {(ds: DeviceState, listEl: HTMLElement, isCamera:boolean)=>any} */
1119 |     let addStream = null;
1120 |     /** @type {FileServer} */
1121 |     let fileServer = null;
1122 |     if (typeof FileServer != 'undefined') {
1123 |         console.log('Starting fileServer');
1124 |         fileServer = new FileServer(new FileSystemHandleArray()); // !! Global variable
1125 |         let targetEl = document.body;
1126 |         let listEl = document.getElementById('files');
1127 |         targetEl.addEventListener('dragover', (ev) => ev.preventDefault());
1128 |         targetEl.addEventListener('drop', (ev) => {
1129 |             ev.preventDefault();
1130 |             for (const item of ev.dataTransfer.items) {
1131 |                 if (item.kind != 'file') {
1132 |                     continue;
1133 |                 }
1134 |                 (async () => {
1135 |                     // @ts-ignore
1136 |                     const handle = await item.getAsFileSystemHandle();
1137 |                     if (await handle.queryPermission({ mode: "read" }) == 'granted') {
1138 |                         fileServer.fs.handle.addEntry(handle);
1139 |                         let el = mkEl('li', [
1140 |                             'File: ', mkEl('span', handle.name, { className: 'streamName', title: handle.kind }),
1141 |                             mkEl('button', 'x', { onclick: (_) => { fileServer.fs.handle.removeEntry(handle.name); el.parentElement.removeChild(el); }, title: 'Stop sharing' })
1142 |                         ]);
1143 |                         if (listEl.childElementCount == 0) {
1144 |                             let checkEl = mkEl('input', [], { type: 'checkbox', onchange: async (_) => checkEl.checked = await fileServer.fs.setWritable(checkEl.checked), title: 'Make files writable' });
1145 |                             listEl.append(mkEl('li', [mkEl('label', [checkEl, 'Writable']),]));
1146 |                         }
1147 |                         listEl.append(el);
1148 |                     }
1149 |                 })();
1150 |             }
1151 |         });
1152 |     }
1153 | 
1154 |     if (isElectronApp) {
1155 |         initStreams = async (d) => {
1156 |             let streamProvider = d.streamProvider = new ElectronStreamProvider(['screen']);
1157 |             streamProvider.pseudoStreams['_selector'] = new StreamSelectScreen(streamProvider);
1158 |             // streamProvider.pseudoStreams['_redirector'] = new StreamRedirector(streamProvider, { id: '_selector', name: 'selector' });
1159 |             // await streamProvider.startStream(d.cm, { id: '_redirector', name: 'redirector' }, true);
1160 |             await streamProvider.startStream(d.cm, { id: '_selector', name: 'stream selector' }, true);
1161 |         };
1162 |     } else {
1163 |         /** @type {WebSocket} */
1164 |         let inputProxySoc = null;
1165 |         let initStreamProvider = async (d) => {
1166 |             let streamProvider = d.streamProvider = new BrowserStreamProvider();
1167 |             streamProvider.sendInputEvent = (target, msg) => {
1168 |                 if (target && inputProxySoc?.readyState == 1) {
1169 |                     msg.target = target;
1170 |                     inputProxySoc.send(JSON.stringify(msg));
1171 |                 }
1172 |             };
1173 |             d.streamProvider.pseudoStreams['_selector'] = new StreamSelectScreen(d.streamProvider);
1174 |             await d.streamProvider.startStream(d.cm, { id: '_selector', name: 'stream selector' }, true);
1175 |         };
1176 |         addStream = async (d, listEl, camera) => {
1177 |             if (!d.streamProvider) { initStreamProvider(d); }
1178 |             let stream = await d.streamProvider.addMediaStream(camera);
1179 |             let el = mkEl('li', [
1180 |                 "(", mkEl('span', stream.name, { className: 'streamName', title: stream.id }), ")",
1181 |                 mkEl('button', 'x', { onclick: (_) => { d.streamProvider.remove(stream); el.parentElement.removeChild(el); } })
1182 |             ]);
1183 |             listEl.append(el);
1184 |         };
1185 | 
1186 |         let connectInputProxy = () => {
1187 |             inputProxySoc?.close();
1188 |             /** @type {HTMLInputElement} */
1189 |             let inputProxyUrlEl = document.querySelector("#inputProxyUrl");
1190 |             if (inputProxyUrlEl?.value) {
1191 |                 inputProxySoc = new WebSocket(inputProxyUrlEl.value);
1192 |             }
1193 |         };
1194 |         document.querySelector('#connectInputButton')?.addEventListener('click', (ev) => connectInputProxy());
1195 |     }
1196 | 
1197 |     /** @type {Record<string, DeviceState>} */
1198 |     let devices = {};
1199 |     let updateDeviceList = (/** @type {DeviceSettings[]} */ deviceSettings) => {
1200 |         let parentEl = document.getElementById('devices');
1201 |         let exstings = Object.keys(devices);
1202 |         let current = deviceSettings.map(d => d.roomId);
1203 |         for (let d of exstings) {
1204 |             if (!current.includes(d)) {
1205 |                 devices[d].cm.dispose();
1206 |                 devices[d].el.parentNode.removeChild(devices[d].el);
1207 |                 delete devices[d];
1208 |             }
1209 |         }
1210 |         for (let d of deviceSettings) {
1211 |             if (exstings.includes(d.roomId)) {
1212 |                 continue;
1213 |             }
1214 |             let name = d.name || d.userAgent.replace(/^Mozilla\/[\d\.]+\s*/, '').replace(/[\s\(\)]+/g, ' ');
1215 |             let cm = new ConnectionManager(d);
1216 |             let listEl = mkEl('ul', [], { className: 'streamlist' });
1217 |             let removeButtonEl = mkEl('button', 'x', {
1218 |                 onclick: (ev) => confirm(`Remove ${name} ?`) && Settings.removePeerDevice(d)
1219 |             });
1220 |             cm.onadded = (c) => {
1221 |                 if (fileServer) {
1222 |                     c.conn.dataChannels['fileServer'] = {};
1223 |                     c.conn.onauth = () => {
1224 |                         Object.assign(c.conn.dataChannels['fileServer'], fileServer.getRtcChannelSpec());
1225 |                     };
1226 |                 }
1227 |                 let el = mkEl('li');
1228 |                 listEl.appendChild(el);
1229 |                 c.conn.onstatechange = () => {
1230 |                     if (c.conn.state == 'disposed') {
1231 |                         el.parentNode?.removeChild(el);
1232 |                         return;
1233 |                     }
1234 |                     if (c.conn.state == 'disconnected' && !c.permanent) {
1235 |                         c.conn.dispose();
1236 |                         return;
1237 |                     }
1238 |                     el.innerText = '';
1239 |                     el.append(
1240 |                         mkEl('span', c.name, { className: 'streamName', title: c.conn.roomId }),
1241 |                         mkEl('span', c.conn.state, { className: 'connectionstate connectionstate_' + c.conn.state }),
1242 |                     );
1243 |                     if (c.conn.state == 'connected') {
1244 |                         el.append(mkEl('button', 'x', { onclick: (_) => c.conn.disconnect() }));
1245 |                     }
1246 |                 };
1247 |             };
1248 |             let titleEl = mkEl('span', name, {
1249 |                 title: d.userAgent,
1250 |                 ondblclick: (_) => {
1251 |                     let n = prompt("Change name", name);
1252 |                     if (n) {
1253 |                         titleEl.innerText = d.name = name = n;
1254 |                         Settings.addPeerDevice(d);
1255 |                     }
1256 |                 }
1257 |             });
1258 |             let el = mkEl('div', [mkEl('span', [titleEl, removeButtonEl]), listEl]);
1259 |             let ds = { el: el, cm: cm };
1260 |             if (!d.services?.includes('no-client') && addStream) {
1261 |                 let streamListEl = mkEl('ul', [], { className: 'streamlist' });
1262 |                 el.append(
1263 |                     streamListEl,
1264 |                     mkEl('button', 'Share Desktop', { onclick: (_) => addStream(ds, streamListEl, false) }),
1265 |                     mkEl('button', 'Share Camera', { onclick: (_) => addStream(ds, streamListEl, true) }),
1266 |                 );
1267 |             }
1268 |             if (d.services?.includes('screen') ?? true) {
1269 |                 // el.append(mkEl('button', 'Open Remote Desktop', { onclick: (_ev) => playStream(d) }));
1270 |                 el.append(mkEl('a', 'Open Remote Desktop', { target: '_blank', href: '#room:' + d.roomId }));
1271 |             }
1272 |             parentEl.append(el);
1273 |             devices[d.roomId] = ds;
1274 |             initStreams(ds);
1275 |         }
1276 |         if (deviceSettings.length == 0) {
1277 |             parentEl.classList.add('nodevices');
1278 |         } else {
1279 |             parentEl.classList.remove('nodevices');
1280 |         }
1281 |     };
1282 | 
1283 |     let onSettingUpdated = (settings) => {
1284 |         document.getElementById('pairing').style.display = settings[0] ? "none" : "block";
1285 |         document.getElementById('publishOrPlay').style.display = settings[0] ? "block" : "none";
1286 |         updateDeviceList(settings);
1287 |     };
1288 |     onSettingUpdated(Settings.getPeerDevices());
1289 |     Settings.onsettingsupdate = onSettingUpdated;
1290 | }
1291 | 
1292 | window.addEventListener('DOMContentLoaded', (ev) => {
1293 |     let playStream = initPlayer();
1294 |     let m = decodeURI(location.hash).match(/[&#]room:([^&]+)/);
1295 |     if (m) {
1296 |         let d = Settings.getPeerDevices().find(d => d.roomId == m[1]);
1297 |         if (d) {
1298 |             let closeEl = document.getElementById('closePlayerButton');
1299 |             if (closeEl) { closeEl.style.display = "none"; }
1300 |             playStream(d);
1301 |             return;
1302 |         }
1303 |     }
1304 |     initPublisher(playStream);
1305 |     initPairing();
1306 | }, { once: true });
1307 | 


--------------------------------------------------------------------------------
/design.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/binzume/webrtc-rdp/9904ba09c1bbdbf777cae74b82e3bedc89329adb/design.png


--------------------------------------------------------------------------------
/index.html:
--------------------------------------------------------------------------------
 1 | <html>
 2 | 
 3 | <head>
 4 |   <link rel="stylesheet" type="text/css" href="app/style.css" />
 5 |   <title>WebRTC Remote Desktop</title>
 6 |   <script src="https://cdn.jsdelivr.net/npm/@open-ayame/ayame-web-sdk@2020.3.0/dist/ayame.js"></script>
 7 |   <script src="app/rtcfilesystem-server.js?20220520"></script>
 8 |   <script src="app/webrtc-rdp.js?20220520"></script>
 9 | </head>
10 | 
11 | <body>
12 |   <h1 class="noplayer browser">WebRTC Remote Desktop</h1>
13 | 
14 |   <div id="content" class="noplayer">
15 |   <div id="pairing" class="block" style="display:none;">
16 |     <h2>Add new device</h2>
17 |     <button id="generatePin">Generate PIN</button> / <button id="inputPin">Input PIN</button>
18 |     <div id="pinDisplayBox" style="display:none;">
19 |       <div class="pin">
20 |         PIN: <span id="pin" class="selectable"></span>
21 |       </div>
22 |       <select id="modeSelect">
23 |         <option value="twoway" selected>two-way</option>
24 |         <option value="host">host</option>
25 |         <option value="guest">guest</option>
26 |       </select><br />
27 |       Waiting.... Visit this page on peer device, and input the PIN.
28 |     </div>
29 |     <form id="pinInputBox">
30 |       <div class="pin">
31 |         PIN:<input type="number" inputmode="numeric" maxlength="8" id="pinInput"
32 |           style="width:200pt;height:50pt;" /><button type='submit' style="width:80pt;height:50pt;">OK</button>
33 |       </div>
34 |       Visit this page on peer device, and generate PIN.
35 |     </form>
36 |   </div>
37 | 
38 |   <div id="publishOrPlay" style="display:none;text-align: center">
39 |     <div style="text-align:right;">
40 |       <button id="addDeviceButton">Add Device</button>
41 |       <button class="browser" id="clearSettingsButton">Reset Settings</button>
42 |     </div>
43 |     <div id="devices"></div>
44 |     <ul id="files" class="streamlist"></ul>
45 |     <div class="browser block">
46 |       Input proxy: <input id="inputProxyUrl" />
47 |       <button id="connectInputButton">Connect</button> <br />
48 |       If you want to cntrol mouse, set <a href="https://github.com/binzume/inputproxy" target="_blank">inputproxy</a>
49 |       url.
50 |     </div>
51 |   </div>
52 |   </div>
53 | 
54 |   <div id="player" style="width: 100%; height: 100vh; position: relative; overflow: hidden;">
55 |     <div id="player-menu">
56 |       <button id="closePlayerButton">Back</button>
57 |       <button id="playButton">Reset</button>
58 |       <button id="toggleShiftButton" class='onoff'>Shift</button>
59 |       <button id="toggleControlButton" class='onoff'>Ctrl</button>
60 |       <button id="toggleAltButton" class='onoff'>Alt</button>
61 |       <button id="muteButton" class='onoff'>Mute</button>
62 |       <button id="selectWindowButton" class='onoff'>Select Window</button>
63 |       <button id="fullscreenButton" class='onoff'>Fullscreen</button>
64 |     </div>
65 |     <video id="screen" autoplay playsinline muted></video>
66 |     <div id="connectingBox" style="margin: 50pt 0;font-size: 40pt;">
67 |       <div class="loading-spinner"></div>
68 |       Connecting...
69 |     </div>
70 |   </div>
71 | 
72 |   <div class="browser noplayer" style="margin: 30pt 0;text-align: center">
73 |     <div><a href="https://binzume.github.io/vr-workspace/#app:app-webrtc-rdp">WebXR Client</a></div>
74 |     <div><a href="https://github.com/binzume/webrtc-rdp/blob/master/README.md">README.md</a></div>
75 |   </div>
76 | 
77 | </body>
78 | 
79 | </html>


--------------------------------------------------------------------------------
/screenshot-xr.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/binzume/webrtc-rdp/9904ba09c1bbdbf777cae74b82e3bedc89329adb/screenshot-xr.png


--------------------------------------------------------------------------------
/webxr/aframe-rdp.js:
--------------------------------------------------------------------------------
  1 | // @ts-check
  2 | 'use strict';
  3 | 
  4 | class Settings {
  5 | 	static settingsKey = 'webrtc-rdp-settings';
  6 | 	static onsettingsupdate = null;
  7 | 
  8 | 	/**
  9 | 	 * @param {DeviceSettings} deviceInfo 
 10 | 	 */
 11 | 	static addPeerDevice(deviceInfo) {
 12 | 		let devices = this.getPeerDevices();
 13 | 		let idx = devices.findIndex(d => d.roomId == deviceInfo.roomId);
 14 | 		if (idx < 0) {
 15 | 			devices.push(deviceInfo);
 16 | 		} else {
 17 | 			devices[idx] = deviceInfo;
 18 | 		}
 19 | 		this._save(devices);
 20 | 	}
 21 | 
 22 | 	/**
 23 | 	 * @returns {DeviceSettings[]}
 24 | 	 */
 25 | 	static getPeerDevices() {
 26 | 		try {
 27 | 			let s = localStorage.getItem(this.settingsKey);
 28 | 			if (!s) { return []; }
 29 | 			let settings = JSON.parse(s);
 30 | 			return settings.version == 2 ? (settings.devices || []) : [settings];
 31 | 		} catch {
 32 | 			// ignore
 33 | 		}
 34 | 		return [];
 35 | 	}
 36 | 
 37 | 	/**
 38 | 	 * @param {DeviceSettings} deviceInfo 
 39 | 	 */
 40 | 	static findPeerDevice(roomId) {
 41 | 		return this.getPeerDevices().find(d => d.roomId == roomId);
 42 | 	}
 43 | 
 44 | 	/**
 45 | 	 * @param {DeviceSettings} deviceInfo 
 46 | 	 */
 47 | 	static removePeerDevice(deviceInfo) {
 48 | 		let devices = this.getPeerDevices();
 49 | 		let filtered = devices.filter(d => d.roomId != deviceInfo.roomId);
 50 | 		if (filtered.length != devices.length) {
 51 | 			this._save(filtered);
 52 | 		}
 53 | 	}
 54 | 
 55 | 	static clear() {
 56 | 		this._save([]);
 57 | 	}
 58 | 
 59 | 	static _save(devices) {
 60 | 		if (devices.length == 0) {
 61 | 			localStorage.removeItem(this.settingsKey);
 62 | 		} else if (devices.length == 1) {
 63 | 			// compat
 64 | 			devices[0].version = 1;
 65 | 			localStorage.setItem(this.settingsKey, JSON.stringify(devices[0]));
 66 | 		} else {
 67 | 			localStorage.setItem(this.settingsKey, JSON.stringify({ devices: devices, version: 2 }));
 68 | 		}
 69 | 		this.onsettingsupdate && this.onsettingsupdate(devices);
 70 | 	}
 71 | }
 72 | 
 73 | class BaseConnection {
 74 | 	/**
 75 | 	 * @param {string} signalingUrl 
 76 | 	 * @param {string|undefined} signalingKey 
 77 | 	 * @param {string} roomId 
 78 | 	 */
 79 | 	constructor(signalingUrl, signalingKey, roomId) {
 80 | 		this.signalingUrl = signalingUrl;
 81 | 		this.roomId = roomId;
 82 | 		this.conn = null;
 83 | 		/** @type {MediaStream|null} */
 84 | 		this.mediaStream = null;
 85 | 		this.stopTracksOnDisposed = true;
 86 | 		/** @type {Record<string, DataChannelInfo>} */
 87 | 		this.dataChannels = {};
 88 | 		this.onstatechange = null;
 89 | 		/** @type {'disconnected' | 'connecting' | 'waiting' | 'disposed' | 'connected'} */
 90 | 		this.state = 'disconnected';
 91 | 		this.options = Object.assign({}, Ayame.defaultOptions);
 92 | 		this.options.video = Object.assign({}, this.options.video);
 93 | 		this.options.audio = Object.assign({}, this.options.audio);
 94 | 		this.options.signalingKey = signalingKey;
 95 | 		this.reconnectWaitMs = -1;
 96 | 		this.connectTimeoutMs = -1;
 97 | 	}
 98 | 	async connect() {
 99 | 		if (this.conn || this.state == 'disposed') {
100 | 			throw 'invalid operation';
101 | 		}
102 | 		await this.setupConnection().connect(this.mediaStream, null);
103 | 	}
104 | 	setupConnection() {
105 | 		console.log("connecting..." + this.signalingUrl + " " + this.roomId);
106 | 		this.updateState('connecting');
107 | 		clearTimeout(this._connectTimer);
108 | 		if (this.connectTimeoutMs > 0) {
109 | 			this._connectTimer = setTimeout(() => this.disconnect(), this.connectTimeoutMs);
110 | 		}
111 | 
112 | 		let conn = this.conn = Ayame.connection(this.signalingUrl, this.roomId, this.options, false);
113 | 		conn.on('open', async (e) => {
114 | 			for (let c of Object.keys(this.dataChannels)) {
115 | 				this._handleDataChannel(await conn.createDataChannel(c));
116 | 			}
117 | 			this.updateState('waiting');
118 | 		});
119 | 		conn.on('connect', (e) => {
120 | 			clearTimeout(this._connectTimer);
121 | 			this.updateState('connected');
122 | 		});
123 | 		conn.on('datachannel', (channel) => {
124 | 			this._handleDataChannel(channel);
125 | 		});
126 | 		conn.on('disconnect', (e) => {
127 | 			this.conn = null;
128 | 			this.disconnect(e.reason);
129 | 		});
130 | 		return conn;
131 | 	}
132 | 	/**
133 | 	 * @param {string|null} reason 
134 | 	 */
135 | 	disconnect(reason = null) {
136 | 		console.log('disconnect', reason);
137 | 		clearTimeout(this._connectTimer);
138 | 		if (this.conn) {
139 | 			this.conn.on('disconnect', () => { });
140 | 			this.conn.disconnect();
141 | 			this.conn.stream = null;
142 | 			this.conn = null;
143 | 		}
144 | 		if (reason != 'dispose' && this.state != 'disconnected' && this.reconnectWaitMs >= 0) {
145 | 			this._connectTimer = setTimeout(() => this.connect(), this.reconnectWaitMs);
146 | 		}
147 | 		for (let c of Object.values(this.dataChannels)) {
148 | 			c.ch = null;
149 | 		}
150 | 		this.updateState('disconnected', reason);
151 | 	}
152 | 	dispose() {
153 | 		this.disconnect('dispose');
154 | 		this.updateState('disposed');
155 | 		this.stopTracksOnDisposed && this.mediaStream?.getTracks().forEach(t => t.stop());
156 | 		this.mediaStream = null;
157 | 	}
158 | 	/**
159 | 	 * @param {'disconnected' | 'connecting' | 'waiting' | 'disposed' | 'connected'} s
160 | 	 * @param {string|null} reason 
161 | 	 */
162 | 	updateState(s, reason = null) {
163 | 		if (s != this.state && this.state != 'disposed') {
164 | 			console.log(this.roomId, s);
165 | 			let oldState = this.state;
166 | 			this.state = s;
167 | 			this.onstatechange && this.onstatechange(s, oldState, reason);
168 | 		}
169 | 	}
170 | 	/**
171 | 	 * @param {RTCDataChannel|null} ch
172 | 	 */
173 | 	_handleDataChannel(ch) {
174 | 		if (!ch) return;
175 | 		let c = this.dataChannels[ch.label];
176 | 		if (c && !c.ch) {
177 | 			console.log('datachannel', ch.label);
178 | 			c.ch = ch;
179 | 			ch.onmessage = (ev) => c.onmessage?.(ch, ev);
180 | 			// NOTE: dataChannel.onclose = null in Ayame web sdk.
181 | 			ch.addEventListener('open', (ev) => c.onopen?.(ch, ev));
182 | 			ch.addEventListener('close', (ev) => c.onclose?.(ch, ev));
183 | 		}
184 | 	}
185 | 	getFingerprint(remote = false) {
186 | 		let pc = this.conn._pc;
187 | 		let m = pc && (remote ? pc.currentRemoteDescription : pc.currentLocalDescription).sdp.match(/a=fingerprint:\s*([\w-]+ [a-f0-9:]+)/i);
188 | 		return m && m[1];
189 | 	}
190 | 	async hmacSha256(password, fingerprint) {
191 | 		let enc = new TextEncoder();
192 | 		let key = await crypto.subtle.importKey('raw', enc.encode(password),
193 | 			{ name: 'HMAC', hash: { name: 'SHA-256' } }, false, ['sign']);
194 | 		let sign = await crypto.subtle.sign('HMAC', key, enc.encode(fingerprint));
195 | 		return btoa(String.fromCharCode(...new Uint8Array(sign)));
196 | 	}
197 | }
198 | 
199 | class PlayerConnection extends BaseConnection {
200 | 	/**
201 | 	 * @param {string} signalingUrl 
202 | 	 * @param {string} roomId 
203 | 	 * @param {HTMLVideoElement|null} videoEl
204 | 	 */
205 | 	constructor(signalingUrl, signalingKey, roomId, videoEl) {
206 | 		super(signalingUrl, signalingKey, roomId);
207 | 		if (videoEl) {
208 | 			this.options.video.direction = 'recvonly';
209 | 			this.options.audio.direction = 'recvonly';
210 | 		}
211 | 		this.videoEl = videoEl;
212 | 		this._rpcResultHandler = {};
213 | 		this.authToken = null;
214 | 		this.services = null;
215 | 		this.onauth = null;
216 | 		this.onstreaminfo = null;
217 | 		this.dataChannels['controlEvent'] = {
218 | 			onopen: async (ch, ev) => {
219 | 				if (window.crypto?.subtle) {
220 | 					let localFingerprint = this.getFingerprint(false);
221 | 					if (!localFingerprint) {
222 | 						console.log("Failed to get DTLS cert fingerprint");
223 | 						return;
224 | 					}
225 | 					console.log("local fingerprint:", localFingerprint);
226 | 					let hmac = this.authToken && await this.hmacSha256(this.authToken, localFingerprint);
227 | 					ch.send(JSON.stringify({
228 | 						type: "auth",
229 | 						requestServices: videoEl ? ['screen', 'file'] : ['file'],
230 | 						fingerprint: localFingerprint,
231 | 						hmac: hmac
232 | 					}));
233 | 				} else {
234 | 					ch.send(JSON.stringify({ type: "auth", token: this.authToken }));
235 | 				}
236 | 			},
237 | 			onmessage: (ch, ev) => {
238 | 				let msg = JSON.parse(ev.data);
239 | 				if (msg.type == 'redirect' && msg.roomId) {
240 | 					this.disconnect('redirect');
241 | 					this.roomId = msg.roomId;
242 | 					this.connect();
243 | 				} else if (msg.type == 'auth') {
244 | 					// player and player error
245 | 					this.disconnect();
246 | 				} else if (msg.type == 'authResult') {
247 | 					this.services = msg.services;
248 | 					this.onauth?.(msg.result);
249 | 				} else if (msg.type == 'streamInfo') {
250 | 					this.onstreaminfo?.(msg);
251 | 				} else if (msg.type == 'rpcResult') {
252 | 					this._rpcResultHandler[msg.reqId]?.(msg);
253 | 				}
254 | 			}
255 | 		};
256 | 	}
257 | 	setupConnection() {
258 | 		let conn = super.setupConnection();
259 | 		conn.on('addstream', (ev) => {
260 | 			this.mediaStream = ev.stream;
261 | 			if (this.videoEl) {
262 | 				this.videoEl.srcObject = ev.stream;
263 | 			}
264 | 		});
265 | 		return conn;
266 | 	}
267 | 	/**
268 | 	 * @param {string|null} reason 
269 | 	 */
270 | 	disconnect(reason = null) {
271 | 		if (this.videoEl && this.videoEl.srcObject == this.mediaStream) {
272 | 			this.videoEl.srcObject = null;
273 | 		}
274 | 		super.disconnect(reason);
275 | 	}
276 | 	sendRpcAsync(name, params, timeoutMs = 10000) {
277 | 		let reqId = Date.now(); // TODO: monotonic
278 | 		this.dataChannels['controlEvent'].ch?.send(JSON.stringify({ type: 'rpc', name: name, reqId: reqId, params: params }));
279 | 		return new Promise((resolve, reject) => {
280 | 			let timer = setTimeout(() => {
281 | 				delete this._rpcResultHandler[reqId];
282 | 				reject('timeout');
283 | 			}, timeoutMs);
284 | 			this._rpcResultHandler[reqId] = (res) => {
285 | 				clearTimeout(timer);
286 | 				delete this._rpcResultHandler[reqId];
287 | 				resolve(res.value);
288 | 			};
289 | 		});
290 | 	}
291 | 	sendMouseEvent(action, x, y, button) {
292 | 		this.dataChannels['controlEvent'].ch?.send(JSON.stringify({ type: 'mouse', action: action, x: x, y: y, button: button }));
293 | 	}
294 | 	sendKeyEvent(action, key, code, shift = false, ctrl = false, alt = false) {
295 | 		this.dataChannels['controlEvent'].ch?.send(JSON.stringify({ type: 'key', action: action, key: key, code: code, shift: shift, ctrl: ctrl, alt: alt }));
296 | 	}
297 | }
298 | 
299 | AFRAME.registerComponent('webrtc-rdp', {
300 | 	schema: {
301 | 		signalingUrl: { default: "wss://ayame-labo.shiguredo.app/signaling" },
302 | 		roomId: { default: "" },
303 | 		streamId: { default: "" },
304 | 		maxWidth: { default: 8 },
305 | 		maxHeight: { default: 6 },
306 | 		settingUrl: { default: "/webrtc-rdp/" },
307 | 		filesystem: { default: 'all', oneOf: ['none', 'connected', 'all'] },
308 | 		adaptiveResolution: { default: true },
309 | 	},
310 | 	/** @type {PlayerConnection} */
311 | 	playerConn: null,
312 | 	/** @type {HTMLVideoElement} */
313 | 	videoEl: null,
314 | 	width: 0,
315 | 	height: 0,
316 | 	/** @type {string|null} */
317 | 	tempRoomId: null,
318 | 	settingIndex: -1,
319 | 	timer: 0,
320 | 	init() {
321 | 		if (this.data.filesystem == 'all') {
322 | 			// defined in ../app/rtcfilesystem-client.js
323 | 			const roomIdPrefix = 'binzume@rdp-room-';
324 | 			globalThis.rtcFileSystemManager?.registerAll((key, id) => new PlayerConnection(this.data.signalingUrl, key, id, null), roomIdPrefix);
325 | 		}
326 | 		// @ts-ignore
327 | 		let screenEl = this.screenEl = this._byName("screen");
328 | 
329 | 		let dragging = false;
330 | 		let dragTimer = null;
331 | 		let mouseMoveTimer = null;
332 | 		screenEl.focus();
333 | 
334 | 		// EXPERIMENT
335 | 		let dragMode = false;
336 | 		let draggingStream = null;
337 | 		/** @type {THREE.Object3D} */
338 | 		let rectObj = null;
339 | 		let rectOffset = new THREE.Vector3();
340 | 		let prepareDrag = () => dragMode = true;
341 | 		let startDrag = async (/** @type {THREE.Raycaster} */ raycaster, intersection) => {
342 | 			let x = intersection.uv.x, y = 1 - intersection.uv.y, distance = intersection.distance;
343 | 			dragMode = true;
344 | 			draggingStream = await this.playerConn?.sendRpcAsync('streamFromPoint', { x: x, y: y });
345 | 			if (!draggingStream || !dragMode) {
346 | 				return;
347 | 			}
348 | 			let sw = screenEl.getAttribute('width') || 1, sh = screenEl.getAttribute('height') || 1;
349 | 			let rect = draggingStream.rect;
350 | 			rectOffset.set((rect.x + rect.width / 2 - x) * sw, -(rect.y + rect.height / 2 - y) * sh, 0);
351 | 
352 | 			if (!rectObj) {
353 | 				const material = new THREE.LineBasicMaterial({ color: 0x8888ff, depthTest: false });
354 | 				const points = [
355 | 					new THREE.Vector3(-0.5, -0.5, 0),
356 | 					new THREE.Vector3(-0.5, 0.5, 0),
357 | 					new THREE.Vector3(0.5, 0.5, 0),
358 | 					new THREE.Vector3(0.5, -0.5, 0),
359 | 				];
360 | 				rectObj = new THREE.LineLoop(new THREE.BufferGeometry().setFromPoints(points), material);
361 | 				rectObj.scale.set(rect.width * sw, rect.height * sh, 1);
362 | 				this.el.setObject3D('draggingrect', rectObj);
363 | 			}
364 | 			let update = () => {
365 | 				let v = raycaster.ray.origin.clone().addScaledVector(raycaster.ray.direction, distance);
366 | 				rectObj.position.copy(rectObj.parent.worldToLocal(v).add(rectOffset));
367 | 			};
368 | 			clearTimeout(dragTimer);
369 | 			dragTimer = setInterval(update, 50);
370 | 			update();
371 | 		};
372 | 		let stopDrag = async () => {
373 | 			dragMode = false;
374 | 			if (!rectObj) { return; }
375 | 			clearTimeout(dragTimer);
376 | 			let position = rectObj.getWorldPosition(new THREE.Vector3());
377 | 			let scale = rectObj.scale.clone();
378 | 			this.el.removeObject3D('draggingrect');
379 | 			rectObj.geometry.dispose();
380 | 			if (rectObj.material instanceof THREE.Material) {
381 | 				rectObj.material.dispose();
382 | 			}
383 | 			rectObj = null;
384 | 			let stream = draggingStream;
385 | 			if (!screenEl.is('cursor-hovered') && stream != null) {
386 | 				let vrapp = this.el.components.vrapp;
387 | 				let r = await this.playerConn.sendRpcAsync('play', { streamId: stream.id, redirect: vrapp == null });
388 | 				if (r && vrapp) {
389 | 					let app = await vrapp.appManager.start(vrapp.app.id, null, { disableWindowLocator: true });
390 | 					app.object3D.quaternion.copy(this.el.object3D.quaternion);
391 | 					app.setAttribute('position', app.object3D.parent.worldToLocal(position));
392 | 					app.addEventListener('loaded', (_) => {
393 | 						app.components['webrtc-rdp'].tempRoomId = r.roomId;
394 | 						app.setAttribute('webrtc-rdp', { roomId: this.data.roomId, streamId: stream.id, maxWidth: Math.max(scale.x, 1.5), maxHeight: Math.max(scale.y, 1.5) });
395 | 						app.components['webrtc-rdp']?.resize(scale.x, scale.y);
396 | 					}, { once: true });
397 | 				}
398 | 			}
399 | 		};
400 | 		// Grip
401 | 		this._ongripdown = (ev) => {
402 | 			let raycaster = ev.target.components.raycaster;
403 | 			let intersection = raycaster.intersectedEls[0] == screenEl && raycaster.getIntersection(screenEl);
404 | 			if (intersection) {
405 | 				startDrag(raycaster.raycaster, intersection);
406 | 			}
407 | 		};
408 | 		this._ongripup = (_) => stopDrag();
409 | 		this._onbuttondown = (ev) => {
410 | 			let raycaster = ev.target.components.raycaster;
411 | 			let intersection = raycaster.intersectedEls[0] == screenEl && raycaster.getIntersection(screenEl);
412 | 			if (intersection) {
413 | 				this.playerConn?.sendMouseEvent("click", intersection.uv.x, 1 - intersection.uv.y, ev.type == 'abuttondown' ? 1 : 2);
414 | 				screenEl.focus();
415 | 				ev.stopPropagation();
416 | 			}
417 | 		};
418 | 		for (let el of this.el.sceneEl.querySelectorAll('[laser-controls]')) {
419 | 			el.addEventListener('gripdown', this._ongripdown);
420 | 			el.addEventListener('gripup', this._ongripup);
421 | 			el.addEventListener('bbuttondown', this._onbuttondown);
422 | 			el.addEventListener('abuttondown', this._onbuttondown);
423 | 		}
424 | 		// Right ALT
425 | 		this._onkeydown = (ev) => ev.code == 'AltRight' && prepareDrag();
426 | 		this._onkeyup = (ev) => ev.code == 'AltRight' && stopDrag();
427 | 		window.addEventListener('keydown', this._onkeydown);
428 | 		window.addEventListener('keyup', this._onkeyup);
429 | 
430 | 		let mousePos = new THREE.Vector2();
431 | 		screenEl.addEventListener('mousedown', (ev) => {
432 | 			screenEl.focus();
433 | 			let intersection = ev.detail.intersection;
434 | 			if (dragMode) {
435 | 				startDrag(ev.detail.cursorEl.components.raycaster.raycaster, intersection);
436 | 				return;
437 | 			}
438 | 			if (intersection) {
439 | 				clearTimeout(dragTimer);
440 | 				mousePos.copy(intersection.uv);
441 | 				dragTimer = setTimeout(() => {
442 | 					dragging = true;
443 | 					this.playerConn?.sendMouseEvent("down", mousePos.x, 1 - mousePos.y, 0);
444 | 				}, 200);
445 | 			}
446 | 		});
447 | 		this.el.sceneEl.addEventListener('mouseup', (ev) => {
448 | 			clearTimeout(dragTimer);
449 | 			if (dragging) {
450 | 				let intersection = ev.detail.cursorEl?.components.raycaster?.getIntersection(screenEl);
451 | 				intersection && mousePos.copy(intersection.uv);
452 | 				this.playerConn?.sendMouseEvent("up", mousePos.x, 1 - mousePos.y, 0);
453 | 				let cancelClick = ev => ev.stopPropagation();
454 | 				window.addEventListener('click', cancelClick, true);
455 | 				setTimeout(() => window.removeEventListener('click', cancelClick, true), 0);
456 | 				dragging = false;
457 | 			}
458 | 			stopDrag();
459 | 		});
460 | 		screenEl.addEventListener('click', (ev) => {
461 | 			let intersection = ev.detail.intersection;
462 | 			intersection && this.playerConn?.sendMouseEvent("click", intersection.uv.x, 1 - intersection.uv.y, 0);
463 | 		});
464 | 		screenEl.addEventListener('materialtextureloaded', (ev) => {
465 | 			/** @type {THREE.Texture} */
466 | 			let map = ev.detail.texture;
467 | 			map.anisotropy = Math.min(16, this.el.sceneEl.renderer.capabilities.getMaxAnisotropy());
468 | 			map.magFilter = THREE.LinearFilter;
469 | 			map.minFilter = THREE.LinearFilter;
470 | 			map.needsUpdate = true;
471 | 		});
472 | 		screenEl.addEventListener('keydown', (ev) => {
473 | 			let modkey = ev.key == "Control" || ev.key == "Alt" || ev.key == "Shift";
474 | 			let k = ev.key;
475 | 			if (!modkey && !ev.shiftKey && k.length == 1 && ev.getModifierState("CapsLock")) {
476 | 				k = k.toLowerCase();
477 | 			}
478 | 			this.playerConn?.sendKeyEvent(modkey ? 'down' : 'press', k, ev.code, ev.shiftKey, ev.ctrlKey, ev.altKey);
479 | 			ev.preventDefault();
480 | 		});
481 | 		screenEl.addEventListener('keyup', (ev) => {
482 | 			let modkey = ev.key == "Control" || ev.key == "Alt" || ev.key == "Shift";
483 | 			if (modkey) {
484 | 				this.playerConn?.sendKeyEvent('up', ev.key, ev.code, ev.shiftKey, ev.ctrlKey, ev.altKey);
485 | 			}
486 | 			ev.preventDefault();
487 | 		});
488 | 		screenEl.addEventListener('mouseenter', (ev) => {
489 | 			let raycaster = ev.detail.cursorEl.components.raycaster;
490 | 			clearTimeout(mouseMoveTimer);
491 | 			mouseMoveTimer = setInterval(() => {
492 | 				let intersection = raycaster.getIntersection(screenEl);
493 | 				if (intersection && mousePos.distanceToSquared(intersection.uv) > 0 && !rectObj) {
494 | 					mousePos.copy(intersection.uv);
495 | 					this.playerConn?.sendMouseEvent("move", mousePos.x, 1 - mousePos.y, 0);
496 | 				}
497 | 			}, 100);
498 | 		});
499 | 		screenEl.addEventListener('mouseleave', (ev) => clearTimeout(mouseMoveTimer));
500 | 		screenEl.focus();
501 | 
502 | 		this._byName("connectButton").addEventListener('click', ev => {
503 | 			let settings = Settings.getPeerDevices()[this.settingIndex];
504 | 			if (settings) {
505 | 				this.el.setAttribute('webrtc-rdp', { roomId: settings.roomId });
506 | 			}
507 | 		});
508 | 		this._byName("addButton").addEventListener('click', ev => {
509 | 			this.el.sceneEl.exitVR();
510 | 			window.open(this.data.settingUrl, '_blank');
511 | 		});
512 | 		let selectSettings = (n) => {
513 | 			let d = Settings.getPeerDevices()[n];
514 | 			if (d) {
515 | 				this.settingIndex = n;
516 | 				this._byName("roomName").setAttribute('value', this._settingName(d));
517 | 			}
518 | 		};
519 | 		this._byName("roomNext").addEventListener('click', ev => selectSettings(this.settingIndex + 1));
520 | 		this._byName("roomPrev").addEventListener('click', ev => selectSettings(this.settingIndex - 1));
521 | 
522 | 		let showControls = visible => {
523 | 			visible = visible || (this.playerConn == null && this.data.roomId == '');
524 | 			this.el.querySelectorAll(".control")
525 | 				.forEach(el => el.setAttribute("visible", visible));
526 | 			if (this.el.components.xywindow) {
527 | 				this.el.components.xywindow.controls.setAttribute("visible", visible);
528 | 			}
529 | 		}
530 | 		showControls(false);
531 | 		this.el.addEventListener('mouseenter', ev => { ev.target != screenEl && setTimeout(() => showControls(true), 0) });
532 | 		this.el.addEventListener('mouseleave', ev => showControls(false));
533 | 		this.el.addEventListener('xyresize', ev => {
534 | 			let r = ev.detail.xyrect;
535 | 			if (r.width != this.width || r.height != this.height) {
536 | 				if (this.videoEl) {
537 | 					this.el.setAttribute('webrtc-rdp', { maxWidth: r.width, maxHeight: r.height });
538 | 					this.resize(this.videoEl.naturalWidth || this.videoEl.videoWidth, this.videoEl.naturalHeight || this.videoEl.videoHeight);
539 | 				}
540 | 			}
541 | 		});
542 | 
543 | 		this._byName('kbdButton').addEventListener('click', ev => {
544 | 			this.el.emit('xykeyboard-request', '');
545 | 		});
546 | 	},
547 | 	update(oldData) {
548 | 		if (oldData.roomId != this.data.roomId) {
549 | 			if (oldData.roomId) {
550 | 				this.tempRoomId = null;
551 | 			}
552 | 			this.data.roomId ? this.connect() : this.disconnect();
553 | 		}
554 | 	},
555 | 	connect() {
556 | 		this.disconnect();
557 | 
558 | 		let data = this.data;
559 | 		this._updateScreen(null);
560 | 		this._byName('statusMessage').setAttribute('value', 'Connecting...');
561 | 		let settings = Settings.findPeerDevice(data.roomId) || { userAgent: '' };
562 | 		let roomId = this.tempRoomId || data.roomId;
563 | 		if (this.el.components.xywindow) {
564 | 			this.el.setAttribute("xywindow", "title", this._settingName(settings));
565 | 		}
566 | 
567 | 		// video element
568 | 		let videoElId = "webrtc-rdp-" + new Date().getTime().toString(16) + Math.floor(Math.random() * 65536).toString(16);
569 | 
570 | 		let videoEl =/** @type {HTMLVideoElement}} */ (Object.assign(document.createElement("video"), {
571 | 			autoplay: true, controls: false, loop: false, id: videoElId, crossOrigin: "", volume: 0.5
572 | 		}));
573 | 		videoEl.addEventListener('loadeddata', ev => {
574 | 			if (videoEl != this.videoEl) { return; }
575 | 			this._updateScreen("#" + videoEl.id);
576 | 			this._byName('statusMessage').setAttribute('value', '');
577 | 			this.resize(videoEl.videoWidth, videoEl.videoHeight);
578 | 			this.el.dispatchEvent(new CustomEvent('webrtc-rdp-connected', { detail: { roomId: roomId, event: ev } }));
579 | 		});
580 | 		videoEl.addEventListener('ended', ev => {
581 | 			this.el.dispatchEvent(new CustomEvent('webrtc-rdp-ended', { detail: { roomId: roomId, event: ev } }));
582 | 		});
583 | 
584 | 		// replace
585 | 		if (this.videoEl) this.videoEl.parentNode.removeChild(this.videoEl);
586 | 		this.el.sceneEl.querySelector('a-assets').append(videoEl);
587 | 		this.videoEl = videoEl;
588 | 
589 | 		// connect
590 | 		let player = this.playerConn = new PlayerConnection(data.signalingUrl, settings.signalingKey, roomId, videoEl);
591 | 		player.authToken = settings.token;
592 | 		player.reconnectWaitMs = 3000 + Math.random() * 5000;
593 | 		if (this.data.filesystem == 'connected' && globalThis.rtcFileSystemManager) {
594 | 			// defined in ../app/rtcfilesystem-client.js
595 | 			player.dataChannels['fileServer'] = globalThis.rtcFileSystemManager.getRtcChannelSpec('RDP-' + settings.roomId, 'RDP-' + this._settingName(settings, 12));
596 | 		}
597 | 		player.onstatechange = (state) => {
598 | 			if (state == 'disconnected') {
599 | 				this._byName('statusMessage').setAttribute('value', 'Disconnected');
600 | 				this._updateScreen(null);
601 | 			}
602 | 		};
603 | 		player.onauth = (ok) => {
604 | 			if (!ok) {
605 | 				this.disconnect();
606 | 				this._byName('statusMessage').setAttribute('value', 'Access denied');
607 | 				return;
608 | 			}
609 | 			if (this.tempRoomId == null && this.data.streamId) {
610 | 				this.tempRoomId = ""; // avoid ridirect loop. TODO
611 | 				this.playerConn.sendRpcAsync('play', { streamId: this.data.streamId, redirect: true });
612 | 			}
613 | 			if (player.services && player.services['RDP'] === undefined) {
614 | 				this._byName('statusMessage').setAttribute('value', 'No desktop');
615 | 			}
616 | 		};
617 | 		player.onstreaminfo = (info) => {
618 | 			if (info.title) {
619 | 				this.el.setAttribute("xywindow", "title", this._settingName(settings) + ' - ' + info.title);
620 | 			}
621 | 		};
622 | 		player.connect();
623 | 		if (data.adaptiveResolution) {
624 | 			this.timer = setInterval(() => this._checkResolution(), 1000);
625 | 		}
626 | 	},
627 | 	_checkResolution() {
628 | 		let sceneEl = this.el.sceneEl;
629 | 		let renderer = sceneEl?.renderer;
630 | 		if (!sceneEl || !renderer) { return; }
631 | 		let vp = renderer.xr.isPresenting ?
632 | 			renderer.xr.getCamera().cameras[0].viewport :
633 | 			renderer.getViewport(new THREE.Vector4());
634 | 		let vw = this.videoEl.videoWidth;
635 | 		if (vw > 0) {
636 | 			let camera = renderer.xr.isPresenting ? renderer.xr.getCamera() : sceneEl.camera;
637 | 			let w = this.screenEl.getAttribute('width') * 0.2;
638 | 			let d = this.el.object3D.getWorldPosition(new THREE.Vector3()).distanceTo(camera.getWorldPosition(new THREE.Vector3()));
639 | 			let preferredWidth = vp.width * w / d * 0.75; // TODO
640 | 			if (vw * 1.5 < preferredWidth || vw / 1.5 > preferredWidth) {
641 | 				this.playerConn.sendRpcAsync('setResolution', { preferredWidth: preferredWidth });
642 | 			}
643 | 		}
644 | 	},
645 | 	disconnect() {
646 | 		this.playerConn?.dispose();
647 | 		this.playerConn = null;
648 | 		clearTimeout(this.timer);
649 | 		this.timer = 0;
650 | 	},
651 | 	resize(width, height) {
652 | 		console.log("media size: " + width + "x" + height);
653 | 		let w = this.data.maxWidth;
654 | 		let h = height / width * w;
655 | 		if (h > this.data.maxHeight) {
656 | 			h = this.data.maxHeight;
657 | 			w = width / height * h;
658 | 		}
659 | 		if (isNaN(h)) {
660 | 			h = 3;
661 | 			w = 10;
662 | 		}
663 | 
664 | 		this.width = w;
665 | 		this.height = h;
666 | 		this.screenEl.setAttribute("width", w);
667 | 		this.screenEl.setAttribute("height", h);
668 | 		setTimeout(() => {
669 | 			this.el.setAttribute("xyrect", { width: w, height: h });
670 | 		}, 0);
671 | 	},
672 | 	_settingName(d, limit = 32) {
673 | 		return d.name || d.userAgent.replace(/^Mozilla\/[\d\.]+\s*/, '').replace(/[\s\(\)]+/g, ' ').substring(0, limit) + '...';
674 | 	},
675 | 	_updateScreen(src) {
676 | 		this.screenEl.removeAttribute("material"); // to avoid texture leaks.
677 | 		this.screenEl.setAttribute('material', { shader: "flat", src: src, color: src ? '#fff' : '#000' });
678 | 	},
679 | 	_byName(name) {
680 | 		return /** @type {import("aframe").Entity} */ (this.el.querySelector("[name=" + name + "]"));
681 | 	},
682 | 	remove() {
683 | 		for (let el of this.el.sceneEl.querySelectorAll('[laser-controls]')) {
684 | 			el.removeEventListener('gripdown', this._ongripdown);
685 | 			el.removeEventListener('gripup', this._ongripup);
686 | 			el.removeEventListener('bbuttondown', this._onbuttondown);
687 | 			el.removeEventListener('abuttondown', this._onbuttondown);
688 | 		}
689 | 		window.removeEventListener('keydown', this._onkeydown);
690 | 		window.removeEventListener('keyup', this._onkeyup);
691 | 		this.disconnect();
692 | 		// @ts-ignore
693 | 		this.screenEl.removeAttribute("material"); // to avoid texture leaks.
694 | 		if (this.videoEl) this.videoEl.parentNode.removeChild(this.videoEl);
695 | 	},
696 | });
697 | 
698 | AFRAME.registerComponent('webrtc-rdp-app', {
699 | 	schema: {},
700 | 	init() {
701 | 		this.el.addEventListener('app-start', async (ev) => {
702 | 			if (ev.detail.restoreState) {
703 | 				this.el.setAttribute('webrtc-rdp', ev.detail.restoreState);
704 | 			}
705 | 			this.el.addEventListener('app-save-state', async (ev) => {
706 | 				ev.detail.setState(this.el.getAttribute('webrtc-rdp'));
707 | 			});
708 | 		}, { once: true });
709 | 	}
710 | });
711 | 


--------------------------------------------------------------------------------
/webxr/index.html:
--------------------------------------------------------------------------------
 1 | <!DOCTYPE html>
 2 | <html>
 3 | 
 4 | <head>
 5 |   <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
 6 |   <title>WebRTC VR RDP</title>
 7 |   <script id='script-aframe' src="https://aframe.io/releases/1.5.0/aframe.min.js"></script>
 8 |   <script id='script-xylayout' src="https://binzume.github.io/aframe-xylayout/dist/xylayout-all.min.js"></script>
 9 |   <script id='script-xyresize' src="https://binzume.github.io/aframe-xylayout/experimental/xyresize.js"></script>
10 |   <script id='ayane-web-sdk' src="https://cdn.jsdelivr.net/npm/@open-ayame/ayame-web-sdk@2020.3.0/dist/ayame.js"></script>
11 |   <script src="aframe-rdp.js"></script>
12 |   <script src="../app/rtcfilesystem-client.js"></script>
13 | </head>
14 | 
15 | <body style="background-color: black; color:white;">
16 |   <a-scene cursor="rayOrigin: mouse; fuse:false" raycaster="objects:.collidable" renderer="antialias: true;">
17 | 
18 |     <a-xywindow id="app" title="WebRTC RDP" webrtc-rdp="settingIndex:0" webrtc-rdp-app xyresize="" scale='0.2 0.2 0.2' width="8" height="1.2" position="0 1.5 -1.5" xycontainer="direction:column">
19 |       <a-video name="screen" tabindex="0" class="collidable nogesture" width="8" height="1.2" src="" material="color:#000"></a-video>
20 |       <a-entity xyrect="height:0">
21 |         <a-xylabel name="statusMessage" width="4" height="0.8" position="0 0.5 0.05" align="center" value="Select Host"></a-xylabel>
22 |         <a-xycontainer class="control" width="3.2" height="1.0" position="0 -0.6 0.05" direction="column" spacing="0.1" justify-items="center">
23 |           <a-entity class="collidable" xyitem="fixed:true" geometry="primitive:xy-rounded-rect;width:3.2;height:1.0" position="0 0.05 -0.02" material="color:#444; opacity:0.8"></a-entity>
24 |           <a-xycontainer class="control" width="3.0" height="0.4" direction="row" spacing="0.1" justify-items="center">
25 |             <a-xybutton name="roomPrev" width="0.4" height="0.4" label="&lt;"></a-xybutton>
26 |             <a-xylabel name="roomName" width="2.0" height="0.4" value="select host"></a-xylabel>
27 |             <a-xybutton name="roomNext" width="0.4" height="0.4" label="&gt;"></a-xybutton>
28 |           </a-xycontainer>
29 |           <a-xycontainer class="control" width="3.0" height="0.4" direction="row" spacing="0.1" justify-items="center">
30 |             <a-xybutton name="addButton" width="0.5" height="0.4" label="+"></a-xybutton>
31 |             <a-xybutton name="connectButton" width="1.4" height="0.4" label="Connect"></a-xybutton>
32 |             <a-xybutton name="kbdButton" width="0.5" height="0.4" label="KBD"></a-xybutton>
33 |           </a-xycontainer>
34 |         </a-xycontainer>
35 |       </a-entity>
36 |     </a-xywindow>
37 | 
38 |     <a-entity id="camera-rig" >
39 |       <a-camera position="0 1.5 0" look-controls="enabled:false"></a-camera>
40 |       <a-entity laser-controls="hand: right" raycaster="objects:.collidable"></a-entity>
41 |       <a-entity laser-controls="hand: left" raycaster="objects:.collidable"></a-entity>
42 |       <a-xykeyboard distance="0.7" scale='0.22 0.22 0.22' position="0 0.9 -1" rotation='-30 0 0'></a-xykeyboard>
43 |     </a-entity>
44 | 
45 |   </a-scene>
46 | 
47 |   <div style="position:absolute;top:10pt;right:10pt;width:60pt;background-color:white;border-radius:4pt;text-align:center;">
48 |     <a href="https://github.com/binzume/vr-workspace">GitHub</a>
49 |   </div>
50 | 
51 | </body>
52 | 
53 | </html>
54 | 


--------------------------------------------------------------------------------