├── webrtc-setup-uml.txt
├── LICENSE
├── flow-uml.txt
├── README.md
├── beta-api.html
└── index.html


/webrtc-setup-uml.txt:
--------------------------------------------------------------------------------
 1 | @startuml
 2 | 
 3 | participant "Browser" as Client
 4 | participant "OpenAI" as API
 5 | 
 6 | Client -> Client: Acquire local audio (getUserMedia())
 7 | Client -> Client: Create RTCPeerConnection
 8 | Client -> Client: Create DataChannel
 9 | Client -> API: addTrack(): attach local audio to RTCPeerConnection
10 | Client -> Client: setLocalDescription()
11 | Client -> API: HTTP POST ({localDescription.sdp, session}, API key)
12 | API -> Client: SDP Answer
13 | Client -> Client: setRemoteDescription(answer)
14 | Client -> API: WebRTC: local audio stream
15 | Client -> API: WebRTC: local video stream
16 | API -> Client: WebRTC: remote audio stream
17 | Client -> Client: ontrack() attaches remote audio stream to <audio>
18 | Client <-> API: DataChannel connected
19 | Client -> Client: DataChannel onopen event fires
20 | API --> Client: Welcome audio according to start instructions
21 | 
22 | @endumlml
23 | 


--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
 1 | MIT License
 2 | 
 3 | Copyright (c) 2025 webrtcHacks contributors
 4 | 
 5 | Permission is hereby granted, free of charge, to any person obtaining a copy
 6 | of this software and associated documentation files (the "Software"), to deal
 7 | in the Software without restriction, including without limitation the rights
 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 | 
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 | 
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 | 
23 | 


--------------------------------------------------------------------------------
/flow-uml.txt:
--------------------------------------------------------------------------------
 1 | @startuml
 2 | 
 3 | participant "index.html" as Client
 4 | participant "OpenAI" as API
 5 | 
 6 | Client -> Client: User clicks "Start Session"
 7 | == Session Start ==
 8 | Client -> Client: Acquire local audio (getUserMedia())
 9 | Client -> Client: Create RTCPeerConnection
10 | Client -> Client: Create DataChannel
11 | Client -> API: addTrack(): attach local audio to RTCPeerConnection
12 | Client -> Client: setLocalDescription()
13 | Client -> API: HTTP POST (localDescription.sdp, API key)
14 | API -> Client: SDP Answer
15 | Client -> Client: setRemoteDescription(answer)
16 | Client -> API: WebRTC: local audio stream
17 | API -> Client: WebRTC: remote audio stream
18 | Client -> Client: ontrack() attaches remote audio stream to <audio>
19 | Client <-> API: DataChannel connected
20 | Client -> Client: DataChannel onopen event fires
21 | Client -> API: DataChannel: "session.update" with session params:\n  instructions, voice, functions, temperature
22 | Client -> API: DataChannel: "response.create" with start instructions
23 | API --> Client: Welcome audio according to start instructions
24 | 
25 | == During Session ==
26 | Client --> API: User audio requests to agent
27 | API --> Client: Agent audio responses
28 | API -> Client: DataChannel messages:\n  audio buffer changes, transcripts, function calls
29 | Client -> API: DataChannel system message updates
30 | 
31 | == Session End ==
32 | Client --> API: User asks to end session
33 | API -> Client: "end_session" function call response
34 | Client -> API: DataChannel: "response.create" with end instructions
35 | API --> Client: Goodbye audio
36 | API -> Client: DataChannel: "output_audio_buffer.stopped"
37 | Client -> Client: Close RTCPeerConnection after audio done
38 | @endumlml


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # GPT Realtime WebRTC Demo
 2 | 
 3 | A lightweight, static, client‑side demo showing how to connect a browser directly to OpenAI's Realtime API over WebRTC for low‑latency, bi‑directional audio and video transmission. 
 4 | It is designed as an simple, educational reference. 
 5 | Vanilla JavaScript only inside a couple `<script>` tags inside a single HTML - no frameworks.
 6 | 
 7 | `index.html` runs the new gpt-realtime GA API.
 8 | 
 9 | `beta-api.html` uses the original realtime preview API.
10 | 
11 | ## Further Reading / Background
12 | For full context, protocol details, architectural breakdowns, and troubleshooting guidance, see these companion posts:
13 | 1. How OpenAI does WebRTC in the new GPT Realtime – https://webrtchacks.com/how-openai-does-webrtc-in-the-new-gpt-realtime/
14 | 2. The (Unofficial) Guide to the OpenAI Realtime WebRTC API – https://webrtchacks.com/the-unofficial-guide-to-openai-realtime-webrtc-api/
15 | 
16 | 
17 | ## Live Demo Pages
18 | Hosted versions of this page on GitHub Pages:
19 | - gpt-realtime: https://webrtchacks.github.io/gpt-realtime-webrtc
20 | - Realtime preview/Beta: https://webrtchacks.github.io/gpt-realtime-webrtc/beta-api.html (source: `beta-api.html`)
21 | 
22 | 
23 | ## What This Project Shows
24 | - Direct browser ↔ OpenAI Realtime API connection using WebRTC (no custom server in this repo)
25 | - Creation and negotiation of a peer connection using an SDP offer/answer flow
26 | - Outbound microphone capture and inbound synthesized audio playback
27 | - Video capture and transmission (on _gpt-realtime_)
28 | - Basic settings UI for pasting your own OpenAI API key and adjusting options
29 | - A beta page for exploring new API fields / events as they appear
30 | 
31 | Supporting reference text diagrams:
32 | - `webrtc-setup-uml.txt` – WebRTC setup sequence outline in the new gpt-realtime
33 | - `flow-uml.txt` – high-level interaction / event flow for beta-api.html 
34 | 
35 | ## Quick Start
36 | Option A: Use GitHub Pages (fastest)
37 | 1. Open the main demo: https://webrtchacks.github.io/gpt-realtime-webrtc
38 | 2. Click the Settings button (gear icon) in the UI.
39 | 3. Paste an OpenAI API key that has Realtime API access.
40 | 4. Allow microphone access when prompted.
41 | 5. Start interacting (speak) and listen for low-latency responses.
42 | 
43 | Option B: Run locally (still static)
44 | 1. Clone this repo.
45 | 2. Just open `index.html` (or `beta-api.html`) directly in a modern Chromium-based browser or Firefox. (If your browser blocks local mic capture via file:// you can serve the folder with a simple static server, e.g. `python -m http.server`.)
46 | 3. Follow steps 2–5 above.
47 | 
48 | ## IMPORTANT: API Key & Security Notes
49 | This is a pure client-side demo. There is no backend here performing token exchange.
50 | 
51 | You must open the Settings dialog and paste your own OpenAI API key for it to work. The key is stored locally (e.g. in `localStorage`) so treat this as an educational sandbox only.
52 | 
53 | ## License
54 | Released under the MIT License. See the `LICENSE` file for full text.
55 | 


--------------------------------------------------------------------------------
/beta-api.html:
--------------------------------------------------------------------------------
  1 | <!DOCTYPE html>
  2 | <html lang="en">
  3 | <head>
  4 |     <meta charset="UTF-8">
  5 |     <title>chatGPT Realtime Beta Chat</title>
  6 |     <style>
  7 |         body {
  8 |             background: #222;
  9 |             color: #f9f9f9;
 10 |             font-family: Arial, sans-serif;
 11 |         }
 12 | 
 13 |         /* Log container styling */
 14 |         #log {
 15 |             border: 1px solid #444;
 16 |             padding: 10px;
 17 |             margin-top: 10px;
 18 |             height: 300px;
 19 |             overflow-y: auto;
 20 |             background: #333;
 21 |             color: #f9f9f9;
 22 |             display: flex;
 23 |             flex-direction: column;
 24 |         }
 25 | 
 26 |         .log-message {
 27 |             margin-bottom: 5px;
 28 |         }
 29 | 
 30 |         /* Input area as part of the log container */
 31 |         #inputContainer {
 32 |             margin-top: auto; /* push input area to the bottom */
 33 |             display: flex;
 34 |             align-items: center;
 35 |         }
 36 | 
 37 |         #textInput {
 38 |             flex: 1;
 39 |             padding: 8px;
 40 |             margin-right: 4px;
 41 |             background: #444;
 42 |             color: #f9f9f9;
 43 |             border: 1px solid #555;
 44 |         }
 45 | 
 46 |         button {
 47 |             padding: 8px;
 48 |             background: #555;
 49 |             color: #f9f9f9;
 50 |             border: 1px solid #666;
 51 |         }
 52 | 
 53 |         /* Settings form styling */
 54 |         #settingsForm {
 55 |             display: none;
 56 |             margin-top: 10px;
 57 |             padding: 10px;
 58 |             border: 1px solid #444;
 59 |             background: #333;
 60 |         }
 61 | 
 62 |         #settingsForm input {
 63 |             width: 100%;
 64 |             padding: 8px;
 65 |             margin-bottom: 10px;
 66 |             background: #444;
 67 |             color: #f9f9f9;
 68 |             border: 1px solid #555;
 69 |             box-sizing: border-box;
 70 |         }
 71 | 
 72 |         #settingsForm button {
 73 |             padding: 8px;
 74 |             background: #555;
 75 |             color: #f9f9f9;
 76 |             border: 1px solid #666;
 77 |         }
 78 | 
 79 |         /* Remote audio control styling */
 80 |         .audio-container {
 81 |             margin: 10px 0;
 82 |             padding: 8px;
 83 |             background: #333;
 84 |             border: 1px solid #444;
 85 |             border-radius: 4px;
 86 |         }
 87 | 
 88 |         .volume-control {
 89 |             display: flex;
 90 |             align-items: center;
 91 |             gap: 8px;
 92 |         }
 93 | 
 94 |         #volumeSlider {
 95 |             width: 100px;
 96 |             accent-color: #555;
 97 |         }
 98 | 
 99 |         #muteButton {
100 |             padding: 4px 8px;
101 |             margin-right: 8px;
102 |             background: #444;
103 |             border: 1px solid #555;
104 |             border-radius: 4px;
105 |             cursor: pointer;
106 |         }
107 |     </style>
108 | </head>
109 | 
110 | <body>
111 | <h1>Chat with GPT-4 Realtime Beta with WebRTC</h1>
112 | <p>Click below start/end a session. You must configure your OpenAI API key by clicking on settings first. </p>
113 | 
114 | <!-- Buttons for session control and settings -->
115 | <button id="startButton">🟢 Start Session</button>
116 | <button id="endButton" hidden>🔴 End Session</button>
117 | <button id="settingsButton">⚙️ Settings</button>
118 | 
119 | <!-- Settings form -->
120 | <div id="settingsForm">
121 |     <label for="openaiApiKey">OpenAI API Key: </label>
122 |     <input id="openaiApiKey" type="text" placeholder="OpenAI API Key"
123 |            value="Your OPENAI_API_KEY from platform.openai.com">
124 |     <br>
125 |     <label for="temperature">Temperature: </label>
126 |     <input id="temperature" type="number" step="0.1" min="0" max="2" placeholder="Temperature" value="1.0"
127 |            style="width: 60px">
128 |     <br>
129 |     <label for="voice">Select Voice: </label>
130 |     <select id="voice">
131 |         <option value="alloy" selected>Alloy</option>
132 |         <option value="echo">Echo</option>
133 |         <option value="shimmer">Shimmer</option>
134 |         <option value="ash">Ash</option>
135 |         <option value="ballad">Ballad</option>
136 |         <option value="coral">Coral</option>
137 |         <option value="sage">Sage</option>
138 |         <option value="verse">Verse</option>
139 |     </select>
140 |     <br>
141 |     <label for="sessionInstructions">Session instructions: </label>
142 |     <input id="sessionInstructions" type="text" placeholder="Session instructions" value="You are a friendly assistant">
143 |     <br>
144 |     <label for="startInstructions">Start instructions: </label>
145 |     <input id="startInstructions" type="text" placeholder="Start instructions"
146 |            value="Greet the user and ask how you can help">
147 |     <br>
148 |     <label for="endInstructions">End instructions: </label>
149 |     <input id="endInstructions" type="text" placeholder="End instructions"
150 |            value="Give a quick good-bye. Sometimes remind the user to press the button to start a new session.">
151 |     <button onclick="resetSettings()"> 🔄 Reset Settings</button>
152 |     <span id="autoSaveStatus"></span>
153 | </div>
154 | 
155 | <!-- Audio element for remote audio -->
156 | <div class="audio-container">
157 |     <audio id="remoteAudio" autoplay></audio>
158 |     <div class="volume-control">
159 |         <label for="volumeSlider">Assistant Audio Control: </label>
160 |         <button id="muteButton">🔊</button>
161 |         <input type="range" id="volumeSlider" min="0" max="1" step="0.1" value="1">
162 |     </div>
163 | </div>
164 | 
165 | <!-- Log container with an input field at the bottom -->
166 | <div id="log">
167 |     <div id="inputContainer">
168 |         <label for="textInput"></label>
169 |         <input id="textInput" type="text" placeholder="Type your message here">
170 |         <button id="sendButton">Send</button>
171 |     </div>
172 | </div>
173 | 
174 | <!--
175 |   1) Core WebRTC & Realtime logic:
176 |      - Global variables
177 |      - fetchEphemeralKey
178 |      - startSession
179 |      - endSession
180 |      - handleMessage
181 |      ...
182 | -->
183 | <script id="rtc-logic">
184 |     // Globals
185 |     let pc;                      // RTCPeerConnection
186 |     let track;                   // Local audio track
187 |     let dc;                      // Data channel
188 |     const assistantResults = {}; // Track interim/final transcripts
189 |     const userMessages = {};     // Track user messages per item ID
190 | 
191 |     // Expose to console for debugging & fun
192 |     window.pc = pc;
193 |     window.track = track;
194 |     window.dc = dc;
195 | 
196 |     // Model & function definitions
197 |     const model = "gpt-4o-mini-realtime-preview";
198 |     const gptFunctions = [
199 |         {
200 |             type: "function",
201 |             name: "end_session",
202 |             description: "the user would like to stop interacting with the Agent",
203 |             parameters: {},
204 |         },
205 |     ];
206 | 
207 | 
208 |     /**
209 |      * Send initial session instructions and start message.
210 |      * This is called when the data channel is opened.
211 |      * @returns {void}
212 |      */
213 |     function sessionStartMessages() {
214 |         const sessionInstruct = localStorage.getItem("sessionInstructions") || defaultSessionInstructions;
215 |         const startInstruct = localStorage.getItem("startInstructions") || defaultStartInstructions;
216 |         const temperature = parseFloat(localStorage.getItem("temperature")) || defaultTemperature;
217 | 
218 |         // Update the session
219 |         const systemMessage = {
220 |             type: "session.update",
221 |             session: {
222 |                 instructions: sessionInstruct,
223 |                 voice: voiceEl.value,
224 |                 tools: gptFunctions,
225 |                 tool_choice: "auto",
226 |                 input_audio_transcription: {model: "whisper-1"},
227 |                 temperature: temperature,
228 |             }
229 |         };
230 |         dc.send(JSON.stringify(systemMessage));
231 | 
232 |         // Start instructions
233 |         const startMessage = {
234 |             type: "response.create",
235 |             response: {
236 |                 modalities: ["text", "audio"],
237 |                 instructions: startInstruct,
238 |                 max_output_tokens: 100
239 |             }
240 |         };
241 |         dc.send(JSON.stringify(startMessage));
242 |         appendOrUpdateLog("Session started.", "system-message");
243 |     }
244 | 
245 |     /**
246 |      * Handle incoming DataChannel messages from the GPT server.
247 |      */
248 |     function handleMessage(event) {
249 |         const message = JSON.parse(event.data);
250 |         const itemId = message.item_id;
251 | 
252 |         switch (message.type) {
253 |             case "session.created":
254 |                 const expiresAt = new Date(message?.session?.expires_at * 1000);
255 |                 console.log(`Session created and will expire at ${expiresAt}`, message);
256 |                 toggleSessionButtons(true);
257 |                 break;
258 | 
259 |             case "input_audio_buffer.speech_started":
260 |                 userMessages[itemId] = {message: ""};
261 |                 appendOrUpdateLog(`User: ...`, "user-message", itemId);
262 |                 break;
263 | 
264 |             case "conversation.item.input_audio_transcription.completed":
265 |                 const content = message.transcript;
266 |                 if (content && userMessages[itemId]) {
267 |                     userMessages[itemId].message = content;
268 |                     appendOrUpdateLog(`User: ${content}`, "user-message", itemId);
269 |                 }
270 |                 break;
271 | 
272 |             case "response.audio_transcript.delta":
273 |                 if (itemId) {
274 |                     assistantResults[itemId] = assistantResults[itemId] || "";
275 |                     assistantResults[itemId] += message.delta;
276 |                     appendOrUpdateLog(`Assistant (interim): ${assistantResults[itemId]}`, "interim-result", itemId);
277 |                 }
278 |                 break;
279 | 
280 |             case "response.audio_transcript.done":
281 |                 if (itemId) {
282 |                     assistantResults[itemId] = message.transcript;
283 |                     appendOrUpdateLog(`Assistant: ${message.transcript}`, "agent-message", itemId);
284 |                 }
285 |                 break;
286 | 
287 |             case "response.function_call_arguments.done":
288 |                 const {name} = message;
289 |                 if (name === "end_session") {
290 |                     console.log("Ending session based on user request");
291 |                     endSession();
292 |                 }
293 |                 break;
294 | 
295 |             default:
296 |                 // For debugging
297 |                 console.info(`Unhandled message from server: ${message.type}`, message);
298 |                 break;
299 |         }
300 |     }
301 | 
302 |     /**
303 |      * Start a new WebRTC session with the OpenAI Realtime API.
304 |      */
305 |     async function startSession() {
306 |         console.log("start_session");
307 |         startButtonEl.disabled = true;
308 | 
309 |         // API Key required
310 |         const apiKey = keyEl.value;
311 |         if (!apiKey || apiKey === "Your OPENAI_API_KEY from platform.openai.com") {
312 |             console.error("No OpenAI API Key provided");
313 |             appendOrUpdateLog("Error: No OpenAI API Key provided.", "system-message");
314 |             toggleSessionButtons(false);
315 |             return;
316 |         }
317 | 
318 |         // Capture the local mic
319 |         // technically optional for the API, but required in this example
320 |         let stream;
321 |         try {
322 |             stream = await navigator.mediaDevices.getUserMedia({audio: true});
323 |             if (!stream) {
324 |                 console.error("Failed to get local stream");
325 |                 return;
326 |             }
327 |             [track] = stream.getAudioTracks();
328 |         } catch (err) {
329 |             console.error("Error accessing mic:", err);
330 |             appendOrUpdateLog("Mic access error. Check permissions.", "system-message");
331 |             toggleSessionButtons(false);
332 |             return;
333 |         }
334 | 
335 |         // Start the WebRTC session
336 |         try {
337 |             // Create PeerConnection
338 |             pc = new RTCPeerConnection();
339 | 
340 |             // On receiving remote track
341 |             pc.ontrack = (e) => remoteAudioEl.srcObject = e.streams[0];
342 |             // Add the local audio track and reference to its source stream
343 |             pc.addTrack(track, stream);
344 | 
345 |             // Create data channel
346 |             dc = pc.createDataChannel("oai");
347 | 
348 |             // Send session instructions upon opening the data channel
349 |             dc.addEventListener("open", () => sessionStartMessages());
350 | 
351 |             // Handle incoming messages from the server
352 |             dc.addEventListener("message", handleMessage);
353 | 
354 |             // implicit setLocalDescription style
355 |             await pc.setLocalDescription();
356 | 
357 |             // Create answer
358 |             const baseUrl = "https://api.openai.com/v1/realtime";
359 |             const response = await fetch(`${baseUrl}?model=${model}`, {
360 |                 method: "POST",
361 |                 body: pc.localDescription.sdp,
362 |                 headers: {
363 |                     Authorization: `Bearer ${apiKey}`,
364 |                     "Content-Type": "application/sdp"
365 |                 },
366 |             });
367 |             if (!response.ok) {
368 |                 console.error("Failed to fetch SDP answer:", await response.text());
369 |             }
370 |             const answer = {type: "answer", sdp: await response.text()};
371 |             await pc.setRemoteDescription(answer);
372 | 
373 |             // Wait for connection to be established before proceeding
374 |             await new Promise((resolve, reject) => {
375 |                 const timeout = setTimeout(() => reject(`Connection timeout. Current state: ${pc.connectionState}`), 10_000);
376 |                 pc.addEventListener("connectionstatechange", () => {
377 |                     if (pc.connectionState === "connected") {
378 |                         clearTimeout(timeout);
379 |                         console.log("Peer connection established!");
380 |                         resolve();
381 |                     }
382 |                 });
383 |             });
384 | 
385 |             // toggleSessionButtons(true);
386 |         } catch (err) {
387 |             console.error("Error starting session:", err);
388 |             appendOrUpdateLog("Error starting session. Please try again.", "system-message");
389 |             if (pc?.connectionState !== "closed") {
390 |                 pc.close();
391 |             }
392 |             toggleSessionButtons(false);
393 |         }
394 |     }
395 | 
396 |     /**
397 |      * End the current session, optionally sending instructions for a closing message.
398 |      * @param {string} instructions - Instructions for the closing message.
399 |      */
400 |     async function endSession(instructions = endInstructionsEl.value || defaultEndInstructions) {
401 |         console.log("Ending session...");
402 | 
403 |         if (dc?.readyState === "open") {
404 |             // Close after the final message
405 |             dc.addEventListener("message", (event) => {
406 |                 const message = JSON.parse(event.data);
407 |                 if (message.type === "output_audio_buffer.stopped") {
408 |                     pc.close();
409 | 
410 |                     console.log("Session ended.");
411 |                     appendOrUpdateLog("Session ended.", "system-message");
412 |                     toggleSessionButtons(false);
413 |                 }
414 |             });
415 | 
416 |             const message = {
417 |                 type: "response.create",
418 |                 response: {
419 |                     modalities: ["text", "audio"],
420 |                     instructions: instructions,
421 |                     max_output_tokens: 200
422 |                 }
423 |             };
424 |             dc.send(JSON.stringify(message));
425 |         }
426 | 
427 |         // Turn off mic
428 |         if (track.status !== "ended") {
429 |             track.stop();
430 |         }
431 | 
432 |         endButtonEl.disabled = true;
433 | 
434 |     }
435 | </script>
436 | 
437 | <!--
438 |   2) UI logic & event wiring:
439 |      - references to DOM elements
440 |      - load/save settings
441 |      - button event handlers
442 |      ...
443 | -->
444 | <script id="ui-logic">
445 |     // Helper to get an element by ID
446 |     const $ = (id) => document.getElementById(id);
447 | 
448 |     // DOM element references
449 |     const keyEl = $("openaiApiKey");
450 |     const sessionInstructionsEl = $("sessionInstructions");
451 |     const startInstructionsEl = $("startInstructions");
452 |     const endInstructionsEl = $("endInstructions");
453 |     const temperatureEl = $("temperature");
454 |     const voiceEl = $("voice");
455 |     const startButtonEl = $("startButton");
456 |     const endButtonEl = $("endButton");
457 |     const autoSaveStatusEl = $("autoSaveStatus");
458 |     const settingsButtonEl = $("settingsButton");
459 |     const settingsFormEl = $("settingsForm");
460 |     const remoteAudioEl = $('remoteAudio');
461 |     const muteButton = $('muteButton');
462 |     const volumeSlider = $('volumeSlider');
463 |     const logEl = $("log");
464 |     const inputContainerEl = $("inputContainer");
465 |     const textInputEl = $("textInput");
466 |     const sendButtonEl = $("sendButton");
467 | 
468 |     // Default values
469 |     const defaultKey = "Your OPENAI_API_KEY from platform.openai.com";
470 |     const defaultSessionInstructions = "You are a friendly assistant";
471 |     const defaultStartInstructions = "Greet the user and ask how you can help";
472 |     const defaultEndInstructions = "Give a quick good-bye. Sometimes remind the user to press the button to start a new session.";
473 |     const defaultTemperature = 1.0;
474 |     const defaultVoice = "alloy";
475 | 
476 | 
477 |     // Volume controls
478 |     volumeSlider.addEventListener('input', (e) => remoteAudioEl.volume = e.target.value);
479 | 
480 |     // Mute button
481 |     let isMuted = false;
482 |     muteButton.addEventListener('click', () => {
483 |         isMuted = !isMuted;
484 |         remoteAudioEl.muted = isMuted;
485 |         muteButton.textContent = isMuted ? '🔇' : '🔊';
486 |     });
487 | 
488 | 
489 |     /**
490 |      * Toggle the session control buttons based on the session state.
491 |      * @param {boolean} isSessionActive - Whether a session is currently active.
492 |      */
493 |     function toggleSessionButtons(isSessionActive) {
494 |         [startButtonEl, endButtonEl].forEach(button => {
495 |             button.hidden = (button === startButtonEl) ? isSessionActive : !isSessionActive;
496 |             button.disabled = (button === startButtonEl) ? isSessionActive : false;
497 |         });
498 |     }
499 | 
500 |     /**
501 |      * Insert a new log entry or update an existing one (by messageId).
502 |      * @param {string} message - The message to log.
503 |      * @param {string} className - Optional CSS class for styling
504 |      * @param {string} messageId - Optional ID to update an existing log entry.
505 |      */
506 |     function appendOrUpdateLog(message, className = "", messageId = null) {
507 |         let logEntry;
508 |         if (messageId && $(messageId)) {
509 |             logEntry = $(messageId);
510 |             logEntry.textContent = message;
511 |         } else {
512 |             logEntry = document.createElement("div");
513 |             logEntry.textContent = message;
514 |             logEntry.classList.add("log-message");
515 |             if (className) logEntry.classList.add(className);
516 |             if (messageId) logEntry.id = messageId;
517 |             logEl.insertBefore(logEntry, inputContainerEl);
518 |         }
519 |         logEl.scrollTop = logEl.scrollHeight;
520 |     }
521 | 
522 |     /**
523 |      * Load settings from localStorage into UI fields.
524 |      * If not set, use default values.
525 |      * @returns {void}
526 |      */
527 |     function loadSettings() {
528 |         keyEl.value = localStorage.getItem("openaiApiKey") || defaultKey;
529 |         sessionInstructionsEl.value = localStorage.getItem("sessionInstructions") || defaultSessionInstructions;
530 |         startInstructionsEl.value = localStorage.getItem("startInstructions") || defaultStartInstructions;
531 |         endInstructionsEl.value = localStorage.getItem("endInstructions") || defaultEndInstructions;
532 |         temperatureEl.value = localStorage.getItem("temperature") || defaultTemperature;
533 |         voiceEl.value = localStorage.getItem("voice") || defaultVoice;
534 |     }
535 | 
536 |     /**
537 |      * Autosave changed settings to localStorage (debounced by 2s).
538 |      * @returns {void}
539 |      */
540 |     function autoSaveSettings() {
541 |         const settings = {
542 |             openaiApiKey: keyEl.value,
543 |             sessionInstructions: sessionInstructionsEl.value,
544 |             startInstructions: startInstructionsEl.value,
545 |             temperature: temperatureEl.value,
546 |             voice: voiceEl.value
547 |         };
548 |         for (const [k, v] of Object.entries(settings)) {
549 |             localStorage.setItem(k, v);
550 |         }
551 |         sendNewSettings();
552 |         autoSaveStatusEl.textContent =
553 |             "Settings autosaved at " + new Date().toLocaleTimeString();
554 |     }
555 | 
556 |     /**
557 |      * Reset settings and re-save them to localStorage.
558 |      * @returns {void}
559 |      */
560 |     function resetSettings() {
561 |         const settings = {
562 |             sessionInstructions: sessionInstructionsEl.value,
563 |             startInstructions: startInstructionsEl.value,
564 |             temperature: temperatureEl.value,
565 |             voice: voiceEl.value
566 |         };
567 |         for (const [k, v] of Object.entries(settings)) {
568 |             localStorage.setItem(k, v);
569 |         }
570 |         loadSettings();
571 |         sendNewSettings();
572 |         console.log("Settings reset!");
573 |     }
574 | 
575 |     /**
576 |      * Send updated session instructions to the GPT server (if data channel is open).
577 |      * @returns {void}
578 |      */
579 |     function sendNewSettings() {
580 |         if (dc && dc.readyState === "open") {
581 |             const message = {
582 |                 type: "session.update",
583 |                 session: {
584 |                     instructions: sessionInstructionsEl.value,
585 |                     tools: gptFunctions,
586 |                     tool_choice: "auto",
587 |                     temperature: parseFloat(temperatureEl.value),
588 |                     // voice: voiceEl.value  // can't change voice mid-session
589 |                 }
590 |             };
591 |             dc.send(JSON.stringify(message));
592 |             console.log("Sent updated session settings:", message);
593 |         } else {
594 |             console.log("Data channel not open. Please wait until session starts.");
595 |         }
596 |     }
597 | 
598 |     // Debounce the settings autosave
599 |     [keyEl, sessionInstructionsEl, startInstructionsEl, endInstructionsEl, temperatureEl, voiceEl]
600 |         .forEach((input) => {
601 |             input.addEventListener("change", () => {
602 |                 clearTimeout(window.autoSaveTimeout);
603 |                 window.autoSaveTimeout = setTimeout(autoSaveSettings, 2000);
604 |             });
605 |         });
606 | 
607 |     /**
608 |      * Send the user's typed message to the GPT server over the data channel.
609 |      * @returns {void}
610 |      */
611 |     function sendText() {
612 |         const text = textInputEl.value.trim();
613 |         if (!text) return;
614 |         if (dc && dc.readyState === "open") {
615 |             const message = {
616 |                 type: "response.create",
617 |                 response: {
618 |                     modalities: ["text", "audio"],
619 |                     instructions: text,
620 |                     temperature: parseFloat(temperatureEl.value),
621 |                     max_output_tokens: 500
622 |                 }
623 |             };
624 |             dc.send(JSON.stringify(message));
625 |             appendOrUpdateLog("User (text): " + text, "user-message");
626 |             textInputEl.value = "";
627 |         } else {
628 |             appendOrUpdateLog("Data channel not open. Please wait until session starts.", "system-message");
629 |         }
630 |     }
631 | 
632 |     // Button event handlers for starting / ending sessions
633 |     startButtonEl.addEventListener("click", async () => {
634 |         if (pc?.connectionState === "connected") {
635 |             console.log("Session already started");
636 |             appendOrUpdateLog("Session already started.", "system-message");
637 |             return;
638 |         }
639 |         toggleSessionButtons(true);
640 |         await startSession();
641 |     });
642 | 
643 |     // End session button - cancel any in-progress response and close the session
644 |     endButtonEl.addEventListener("click", async () => {
645 |         if (pc?.connectionState === "closed") {
646 |             console.log(`No session to end. Connection state: ${pc.connectionState}`);
647 |             appendOrUpdateLog("No session to end.", "system-message");
648 |             return;
649 |         }
650 |         toggleSessionButtons(false);
651 |         // cancel any in-progress response - not needed for speech input due to turn detection
652 |         dc.send(JSON.stringify({type: "response.cancel"}));
653 |         await endSession();
654 |     });
655 | 
656 |     // Show/hide settings form
657 |     settingsButtonEl.addEventListener("click", () => {
658 |         settingsFormEl.style.display =
659 |             (settingsFormEl.style.display === "none") ? "block" : "none";
660 |     });
661 | 
662 |     // Send on click or Enter press
663 |     sendButtonEl.addEventListener("click", sendText);
664 |     textInputEl.addEventListener("keydown", (ev) => {
665 |         if (ev.key === "Enter") {
666 |             sendText();
667 |         }
668 |     });
669 | 
670 |     loadSettings();
671 | </script>
672 | 
673 | </body>
674 | </html>
675 | 


--------------------------------------------------------------------------------
/index.html:
--------------------------------------------------------------------------------
  1 | <!DOCTYPE html>
  2 | <html lang="en">
  3 | <head>
  4 |     <meta charset="UTF-8">
  5 |     <title>gpt-realtime WebRTC Chat</title>
  6 |     <style>
  7 |         body {
  8 |             background: #222;
  9 |             color: #f9f9f9;
 10 |             font-family: Arial, sans-serif;
 11 |         }
 12 | 
 13 |         /* Make links bold and white instead of the default blue */
 14 |         a, a:visited {
 15 |             color: #ffffff;       /* white */
 16 |             font-weight: 700;     /* bold */
 17 |             text-decoration: none;
 18 |         }
 19 |         a:hover, a:focus {
 20 |             color: #ffffff;       /* keep white on hover/focus */
 21 |             text-decoration: underline; /* optional visual cue */
 22 |         }
 23 | 
 24 |         /* Log container styling */
 25 |         #log {
 26 |             border: 1px solid #444;
 27 |             padding: 10px;
 28 |             margin-top: 10px;
 29 |             height: 300px;
 30 |             overflow-y: auto;
 31 |             background: #333;
 32 |             color: #f9f9f9;
 33 |             display: flex;
 34 |             flex-direction: column;
 35 |         }
 36 | 
 37 |         .log-message {
 38 |             margin-bottom: 5px;
 39 |         }
 40 | 
 41 |         /* Input area as part of the log container */
 42 |         #inputContainer {
 43 |             margin-top: auto; /* push input area to the bottom */
 44 |             display: flex;
 45 |             align-items: center;
 46 |         }
 47 | 
 48 |         #textInput {
 49 |             flex: 1;
 50 |             padding: 8px;
 51 |             margin-right: 4px;
 52 |             background: #444;
 53 |             color: #f9f9f9;
 54 |             border: 1px solid #555;
 55 |         }
 56 | 
 57 |         button {
 58 |             padding: 8px;
 59 |             background: #555;
 60 |             color: #f9f9f9;
 61 |             border: 1px solid #666;
 62 |         }
 63 | 
 64 |         /* Settings form styling */
 65 |         #settingsForm {
 66 |             display: none;
 67 |             margin-top: 10px;
 68 |             padding: 10px;
 69 |             border: 1px solid #444;
 70 |             background: #333;
 71 |         }
 72 | 
 73 |         #settingsForm input {
 74 |             width: 100%;
 75 |             padding: 8px;
 76 |             margin-bottom: 10px;
 77 |             background: #444;
 78 |             color: #f9f9f9;
 79 |             border: 1px solid #555;
 80 |             box-sizing: border-box;
 81 |         }
 82 | 
 83 |         #settingsForm button {
 84 |             padding: 8px;
 85 |             background: #555;
 86 |             color: #f9f9f9;
 87 |             border: 1px solid #666;
 88 |         }
 89 | 
 90 |         /* Remote audio control styling */
 91 |         .audio-container {
 92 |             margin: 10px 0;
 93 |             padding: 8px;
 94 |             background: #333;
 95 |             border: 1px solid #444;
 96 |             border-radius: 4px;
 97 |         }
 98 | 
 99 |         .volume-control {
100 |             display: flex;
101 |             align-items: center;
102 |             gap: 8px;
103 |             flex-wrap: wrap;
104 |         }
105 | 
106 |         #volumeSlider {
107 |             width: 100px;
108 |             accent-color: #555;
109 |         }
110 | 
111 |         #muteButton, #cameraButton {
112 |             padding: 4px 8px;
113 |             margin-right: 4px;
114 |             background: #444;
115 |             border: 1px solid #555;
116 |             border-radius: 4px;
117 |             cursor: pointer;
118 |         }
119 |         #cameraButton.video-off {
120 |             opacity: 0.6;
121 |         }
122 |     </style>
123 | </head>
124 | 
125 | <body>
126 | <h1>Chat with GPT-4 Realtime with WebRTC</h1>
127 | <p>Click below start/end a session. Audio and video capture are enabled by default unless you turn them off.
128 |     Browser permissions are required for microphone and camera access.
129 |     You must configure your OpenAI API key by clicking on settings first.
130 | </p>
131 | </p>
132 |     Visit <a href="./beta-api.html">/beta-api</a> to run the original preview version of the realtime API.
133 | 
134 | </p>
135 | 
136 | <!-- Buttons for session control and settings -->
137 | <button id="startButton">🟢 Start Session</button>
138 | <button id="endButton" hidden>🔴 End Session</button>
139 | <button id="settingsButton">⚙️ Settings</button>
140 | 
141 | <!-- Settings form -->
142 | <div id="settingsForm">
143 |     <label for="openaiApiKey">OpenAI API Key: </label>
144 |     <input id="openaiApiKey" type="text" placeholder="OpenAI API Key"
145 |            value="Your OPENAI_API_KEY from platform.openai.com">
146 |     <br>
147 |     <label for="voice">Select Voice: </label>
148 |     <select id="voice">
149 |         <option value="alloy">Alloy</option>
150 |         <option value="ash">Ash</option>
151 |         <option value="ballad">Ballad</option>
152 |         <option value="birch">Birch</option>
153 |         <option value="cedar" selected>Cedar</option>
154 |         <option value="coral">Coral</option>
155 |         <option value="echo">Echo</option>
156 |         <option value="marin">Marin</option>
157 |         <option value="sage">Sage</option>
158 |         <option value="shimmer">Shimmer</option>
159 |         <option value="verse">Verse</option>
160 |     </select>
161 |     <br>
162 |     <label for="sessionInstructions">Session instructions: </label>
163 |     <input id="sessionInstructions" type="text" placeholder="Session instructions" value="You are a friendly assistant">
164 |     <br>
165 |     <label for="startInstructions">Start instructions: </label>
166 |     <input id="startInstructions" type="text" placeholder="Start instructions"
167 |            value="Greet the user and ask how you can help">
168 |     <br>
169 |     <label for="endInstructions">End instructions: </label>
170 |     <input id="endInstructions" type="text" placeholder="End instructions"
171 |            value="Give a quick good-bye. Sometimes remind the user to press the button to start a new session.">
172 |     <button onclick="resetSettings()"> 🔄 Reset Settings</button>
173 |     <span id="autoSaveStatus"></span>
174 | </div>
175 | 
176 | <!-- Audio element for remote audio -->
177 | <div class="audio-container">
178 |     <audio id="remoteAudio" autoplay></audio>
179 |     <div class="volume-control">
180 |         <span id="mediaControlLabel">Assistant Media Control:</span>
181 |         <label for="volumeSlider" style="margin-left:8px;">Volume</label>
182 |         <input type="range" id="volumeSlider" min="0" max="1" step="0.1" value="1">
183 |         <button id="muteButton" title="Toggle assistant audio playback">🔊</button>
184 |         <button id="cameraButton" title="Toggle camera on/off">📹</button>
185 | 
186 |     </div>
187 | </div>
188 | 
189 | <!-- Log container with an input field at the bottom -->
190 | <div id="log">
191 |     <div id="inputContainer">
192 |         <label for="textInput"></label>
193 |         <input id="textInput" type="text" placeholder="Type your message here">
194 |         <button id="sendButton">Send</button>
195 |     </div>
196 | </div>
197 | 
198 | <!--
199 |   1) Core WebRTC & Realtime logic:
200 |      - Global variables
201 |      - fetchEphemeralKey
202 |      - startSession
203 |      - endSession
204 |      - handleMessage
205 |      ...
206 | -->
207 | <script id="rtc-logic">
208 |     // Globals
209 |     let pc;                      // RTCPeerConnection
210 |     let audioTrack;              // Local audio track
211 |     let videoTrack;              // Local video track
212 |     let dc;                      // Data channel
213 |     const assistantResults = {}; // Track interim/final transcripts
214 |     const userMessages = {};     // Track user messages per item ID
215 |     let assistantSpeaking = false; // Track if assistant is currently speaking
216 | 
217 |     // Expose to console for debugging & fun
218 |     window.pc = pc;
219 |     window.dc = dc;
220 | 
221 |     // Model & function definitions
222 |     const model = "gpt-realtime"   // beta: "gpt-4o-mini-realtime-preview";
223 |     const gptFunctions = [
224 |         {
225 |             type: "function",
226 |             name: "end_session",
227 |             description: "the user would like to stop interacting with the Agent",
228 |             parameters: {},
229 |         },
230 |     ];
231 | 
232 | 
233 |     /**
234 |      * Send initial session instructions and start message.
235 |      * This is called when the data channel is opened.
236 |      * @returns {void}
237 |      */
238 |     function sessionStartMessages() {
239 |         const sessionInstruct = localStorage.getItem("sessionInstructions") || defaultSessionInstructions;
240 |         const startInstruct = localStorage.getItem("startInstructions") || defaultStartInstructions;
241 | 
242 |         // Start instructions
243 |         const startMessage = {
244 |             type: "response.create",
245 |             response: {
246 |                 instructions: startInstruct,
247 |                 max_output_tokens: 100
248 |             }
249 |         };
250 |         dc.send(JSON.stringify(startMessage));
251 |         appendOrUpdateLog("Session started.", "system-message");
252 |     }
253 | 
254 |     /**
255 |      * Handle incoming DataChannel messages from the GPT server.
256 |      */
257 |     function handleMessage(event) {
258 |         const message = JSON.parse(event.data);
259 |         const itemId = message.item_id;
260 | 
261 |         switch (message.type) {
262 |             case "session.created":
263 |                 const expiresAt = new Date(message?.session?.expires_at * 1000);
264 |                 console.log(`Session created and will expire at ${expiresAt}`, message);
265 |                 toggleSessionButtons(true);
266 |                 break;
267 | 
268 |             case "input_audio_buffer.speech_started":
269 |                 userMessages[itemId] = {message: ""};
270 |                 appendOrUpdateLog(`User: ...`, "user-message", itemId);
271 |                 break;
272 | 
273 |             case "conversation.item.input_audio_transcription.completed":
274 |                 const content = message.transcript;
275 |                 if (content && userMessages[itemId]) {
276 |                     userMessages[itemId].message = content;
277 |                     appendOrUpdateLog(`User: ${content}`, "user-message", itemId);
278 |                 }
279 |                 break;
280 | 
281 |             // NEW: detect when teh video stream was viewed
282 |             case "conversation.item.added":
283 |                 const contentText = message?.item?.content?.[0]?.type;
284 |                 if(contentText === "input_image")
285 |                     appendOrUpdateLog(`📸 image added`, "system-message", itemId);
286 |                 else
287 |                     console.info(`Unhandled ${message.type} from server: `, message);
288 |                 break;
289 | 
290 | 
291 |             // NEW: response.output_text.delta (was response.text.delta)
292 |             case "response.output_text.delta": // changed from "response.text.delta"
293 |                 if (itemId) {
294 |                     assistantResults[itemId] = assistantResults[itemId] || "";
295 |                     assistantResults[itemId] += message.delta;
296 |                     appendOrUpdateLog(`Assistant (interim): ${assistantResults[itemId]}`, "interim-result", itemId);
297 |                 }
298 |                 break;
299 | 
300 |             case "response.output_audio_transcript.delta": // previously "response.audio_transcript.delta"
301 |                 if (itemId) {
302 |                     assistantResults[itemId] = assistantResults[itemId] || "";
303 |                     assistantResults[itemId] += message.delta;
304 |                     appendOrUpdateLog(`Assistant (interim): ${assistantResults[itemId]}`, "interim-result", itemId);
305 |                 }
306 |                 break;
307 | 
308 |             case "response.output_audio_transcript.done": // changed from "response.audio_transcript.done"
309 |                 if (itemId) {
310 |                     assistantResults[itemId] = message.transcript;
311 |                     appendOrUpdateLog(`Assistant: ${message.transcript}`, "agent-message", itemId);
312 |                 }
313 |                 break;
314 | 
315 |             case "response.function_call_arguments.done":
316 |                 const {name} = message;
317 |                 if (name === "end_session") {
318 |                     console.log("Ending session based on user request");
319 |                     endSession();
320 |                 }
321 |                 break;
322 | 
323 |             case "output_audio_buffer.started":
324 |                 assistantSpeaking = true;
325 |                 console.log("🗣️ Assistant started speaking");
326 |                 break;
327 | 
328 |             case "output_audio_buffer.stopped":
329 |                 assistantSpeaking = false;
330 |                 console.log("🤫 Assistant ended speaking");
331 |                 break;
332 | 
333 |             default:
334 |                 // For debugging
335 |                 console.info(`Unhandled message from server: ${message.type}`, message);
336 |                 break;
337 |         }
338 |     }
339 | 
340 |     /**
341 |      * Start a new WebRTC session with the OpenAI Realtime API.
342 |      */
343 |     async function startSession() {
344 |         console.log("start_session");
345 |         startButtonEl.disabled = true;
346 | 
347 |         // API Key required
348 |         const apiKey = keyEl.value;
349 |         if (!apiKey || apiKey === "Your OPENAI_API_KEY from platform.openai.com") {
350 |             console.error("No OpenAI API Key provided");
351 |             appendOrUpdateLog("Error: No OpenAI API Key provided.", "system-message");
352 |             toggleSessionButtons(false);
353 |             return;
354 |         }
355 | 
356 |         // Capture the local mic
357 |         // technically optional for the API, but required in this example
358 |         let stream;
359 |         try {
360 |             // Updated constraints: prefer 1920x1080 at 1 fps (low fps to reduce bandwidth)
361 |             const videoConstraints = {
362 |                 width: { ideal: 1920 },
363 |                 height: { ideal: 1080 },
364 |                 frameRate: { ideal: 1}
365 |             };
366 |             stream = await navigator.mediaDevices.getUserMedia({audio: true, video: videoConstraints});
367 |             if (!stream) {
368 |                 console.error("Failed to get local stream");
369 |                 return;
370 |             }
371 |             [audioTrack] = stream.getAudioTracks();
372 |             [videoTrack] = stream.getVideoTracks();
373 |             window.tracks = [audioTrack, videoTrack];
374 |         } catch (err) {
375 |             console.error("Error accessing mic:", err);
376 |             appendOrUpdateLog("Mic access error. Check permissions.", "system-message");
377 |             toggleSessionButtons(false);
378 |             return;
379 |         }
380 | 
381 |         // Start the WebRTC session
382 |         try {
383 |             // Create PeerConnection
384 |             pc = new RTCPeerConnection();
385 | 
386 |             // On receiving remote audio track
387 |             pc.ontrack = (e) => remoteAudioEl.srcObject = e.streams[0];
388 | 
389 |             // Add the local tracks and reference them to their source stream
390 |             if(audioTrack)
391 |                 pc.addTrack(audioTrack, stream);
392 |             if(videoTrack)
393 |                 pc.addTrack(videoTrack, stream);
394 | 
395 | 
396 |             // Create data channel
397 |             dc = pc.createDataChannel("oai");
398 | 
399 |             // Send session instructions upon opening the data channel
400 |             dc.addEventListener("open", () => sessionStartMessages());
401 | 
402 |             // Handle incoming messages from the server
403 |             dc.addEventListener("message", handleMessage);
404 | 
405 |             // implicit setLocalDescription style
406 |             await pc.setLocalDescription();
407 | 
408 |             const newSession = {
409 |                 type: "realtime",
410 |                 model: model,
411 |                 instructions: sessionInstructionsEl.value,
412 |                 audio: {
413 |                     input: {
414 |                         transcription: {
415 |                             model: "whisper-1",     // whisper-1, gpt-4o-transcribe-latest, gpt-4o-mini-transcribe, or gpt-4o-transcribe
416 |                         }
417 |                     },
418 |                     output: {
419 |                         voice: voiceEl.value,
420 |                     }
421 |                 },
422 |                 // tool_choice: "auto",      // was "function_call" in beta
423 |                 tools: gptFunctions,
424 |                 tracing: "auto"
425 |             };
426 | 
427 |             const fd = new FormData();
428 |             fd.set("sdp", pc.localDescription.sdp);
429 |             fd.set("session", JSON.stringify(newSession));
430 | 
431 | 
432 |             // Create answer
433 |             const baseUrl = "https://api.openai.com/v1/realtime/calls";
434 |             const response = await fetch(`${baseUrl}`, { //?model=${model}`, {
435 |                 method: "POST",
436 |                 headers: {Authorization: `Bearer ${apiKey}`},
437 |                 body: fd
438 |             });
439 |             if (!response.ok) {
440 |                 console.error("Failed to fetch SDP answer:", await response.text());
441 |             }
442 | 
443 |             // Extract call ID from the Location header - could be useful for debugging
444 |             const location = response.headers.get("Location");
445 |             const callId = location?.split("/").pop();
446 |             console.log("call created:", callId);
447 | 
448 |             const answerSdp = await response.text();
449 |             // console.log("SDP answer:", answerSdp);
450 | 
451 |             await pc.setRemoteDescription({type: "answer", sdp: answerSdp});
452 | 
453 |             // Wait for connection to be established before proceeding
454 |             await new Promise((resolve, reject) => {
455 |                 const timeout = setTimeout(() => reject(`Connection timeout. Current state: ${pc.connectionState}`), 10_000);
456 |                 pc.addEventListener("connectionstatechange", () => {
457 |                     if (pc.connectionState === "connected") {
458 |                         clearTimeout(timeout);
459 |                         console.log("Peer connection established!");
460 |                         resolve();
461 |                     }
462 |                 });
463 |             });
464 | 
465 |             // toggleSessionButtons(true);
466 |         } catch (err) {
467 |             console.error("Error starting session:", err);
468 |             appendOrUpdateLog("Error starting session. Please try again.", "system-message");
469 |             if (pc?.connectionState !== "closed") {
470 |                 pc.close();
471 |             }
472 |             toggleSessionButtons(false);
473 |         }
474 |     }
475 | 
476 |     /**
477 |      * End the current session, optionally sending instructions for a closing message.
478 |      * @param {string} instructions - Instructions for the closing message.
479 |      */
480 |     async function endSession(instructions = endInstructionsEl.value || defaultEndInstructions) {
481 |         console.log("Ending session...");
482 | 
483 |         if (dc?.readyState === "open") {
484 |             // Close after the final message
485 |             dc.addEventListener("message", (event) => {
486 |                 const message = JSON.parse(event.data);
487 |                 if (message.type === "output_audio_buffer.stopped") {
488 |                     pc.close();
489 | 
490 |                     console.log("Session ended.");
491 |                     appendOrUpdateLog("Session ended.", "system-message");
492 |                     toggleSessionButtons(false);
493 |                 }
494 |             });
495 | 
496 |             const message = {
497 |                 type: "response.create",
498 |                 response: {
499 |                     instructions: instructions,
500 |                     max_output_tokens: 200
501 |                 }
502 |             };
503 |             dc.send(JSON.stringify(message));
504 |         }
505 | 
506 |         // Turn off mic & camera
507 |         if (audioTrack && audioTrack.status !== "ended") {
508 |             audioTrack.stop();
509 |             audioTrack = null;
510 |         }
511 |         if(videoTrack && videoTrack.status !== "ended") {
512 |             videoTrack.stop();
513 |             videoTrack = null;
514 |         }
515 | 
516 |         endButtonEl.disabled = true;
517 | 
518 |     }
519 | </script>
520 | 
521 | <!--
522 |   2) UI logic & event wiring:
523 |      - references to DOM elements
524 |      - load/save settings
525 |      - button event handlers
526 |      ...
527 | -->
528 | <script id="ui-logic">
529 |     // Helper to get an element by ID
530 |     const $ = (id) => document.getElementById(id);
531 | 
532 |     // DOM element references
533 |     const keyEl = $("openaiApiKey");
534 |     const sessionInstructionsEl = $("sessionInstructions");
535 |     const startInstructionsEl = $("startInstructions");
536 |     const endInstructionsEl = $("endInstructions");
537 |     const voiceEl = $("voice");
538 |     const startButtonEl = $("startButton");
539 |     const endButtonEl = $("endButton");
540 |     const autoSaveStatusEl = $("autoSaveStatus");
541 |     const settingsButtonEl = $("settingsButton");
542 |     const settingsFormEl = $("settingsForm");
543 |     const remoteAudioEl = $('remoteAudio');
544 |     const muteButton = $('muteButton');
545 |     const cameraButton = $('cameraButton');
546 |     const volumeSlider = $('volumeSlider');
547 |     const logEl = $("log");
548 |     const inputContainerEl = $("inputContainer");
549 |     const textInputEl = $("textInput");
550 |     const sendButtonEl = $("sendButton");
551 | 
552 |     // Default values
553 |     const defaultKey = "Your OPENAI_API_KEY from platform.openai.com";
554 |     const defaultSessionInstructions = `You are a natural, concise voice AI assistant that speaks clearly and conversationally.
555 |     Your primary language is English.
556 |     Listen for user speech or text; wait for a brief pause before responding; avoid interrupting active user speech.
557 |     If input is unclear, ask a short clarifying question before proceeding; summarize only when explicitly asked.
558 |     Support multimodal inputs (voice, text, images) and respond with helpful, real-time friendly guidance.`;
559 |     const defaultStartInstructions = "Greet the user and ask how you can help";
560 |     const defaultEndInstructions = "Give a quick good-bye. Sometimes remind the user to press the button to start a new session.";
561 |     const defaultVoice = "cedar";
562 | 
563 | 
564 |     // Volume controls
565 |     volumeSlider.title = "Adjust assistant audio volume";
566 |     volumeSlider.addEventListener('input', (e) => remoteAudioEl.volume = e.target.value);
567 | 
568 |     // Mute (assistant playback) button
569 |     let isMuted = false;
570 |     muteButton.addEventListener('click', () => {
571 |         isMuted = !isMuted;
572 |         remoteAudioEl.muted = isMuted;
573 |         muteButton.textContent = isMuted ? '🔇' : '🔊';
574 |         muteButton.title = isMuted ? 'Unmute assistant audio' : 'Mute assistant audio';
575 |     });
576 | 
577 |     // Camera (local video track) toggle
578 |     let cameraOn = true;
579 |     cameraButton.addEventListener('click', () => {
580 |         cameraOn = !cameraOn;
581 |         cameraButton.textContent = cameraOn ? '📹' : '🚫📹';
582 |         cameraButton.title = cameraOn ? 'Turn camera off' : 'Turn camera on';
583 |         cameraButton.classList.toggle('video-off', !cameraOn);
584 |         if (videoTrack)
585 |             videoTrack.enabled = cameraOn; // do not fully stop so it can be resumed quickly
586 | 
587 |     });
588 | 
589 |     /**
590 |      * Toggle the session control buttons based on the session state.
591 |      * @param {boolean} isSessionActive - Whether a session is currently active.
592 |      */
593 |     function toggleSessionButtons(isSessionActive) {
594 |         [startButtonEl, endButtonEl].forEach(button => {
595 |             button.hidden = (button === startButtonEl) ? isSessionActive : !isSessionActive;
596 |             button.disabled = (button === startButtonEl) ? isSessionActive : false;
597 |         });
598 |     }
599 | 
600 |     /**
601 |      * Insert a new log entry or update an existing one (by messageId).
602 |      * @param {string} message - The message to log.
603 |      * @param {string} className - Optional CSS class for styling
604 |      * @param {string} messageId - Optional ID to update an existing log entry.
605 |      */
606 |     function appendOrUpdateLog(message, className = "", messageId = null) {
607 |         let logEntry;
608 |         if (messageId && $(messageId)) {
609 |             logEntry = $(messageId);
610 |             logEntry.textContent = message;
611 |         } else {
612 |             logEntry = document.createElement("div");
613 |             logEntry.textContent = message;
614 |             logEntry.classList.add("log-message");
615 |             if (className) logEntry.classList.add(className);
616 |             if (messageId) logEntry.id = messageId;
617 |             logEl.insertBefore(logEntry, inputContainerEl);
618 |         }
619 |         logEl.scrollTop = logEl.scrollHeight;
620 |     }
621 | 
622 |     /**
623 |      * Load settings from localStorage into UI fields.
624 |      * If not set, use default values.
625 |      * @returns {void}
626 |      */
627 |     function loadSettings() {
628 |         keyEl.value = localStorage.getItem("openaiApiKey") || defaultKey;
629 |         sessionInstructionsEl.value = localStorage.getItem("sessionInstructions") || defaultSessionInstructions;
630 |         startInstructionsEl.value = localStorage.getItem("startInstructions") || defaultStartInstructions;
631 |         endInstructionsEl.value = localStorage.getItem("endInstructions") || defaultEndInstructions;
632 |         voiceEl.value = localStorage.getItem("voice") || defaultVoice;
633 |     }
634 | 
635 |     /**
636 |      * Autosave changed settings to localStorage (debounced by 2s).
637 |      * @returns {void}
638 |      */
639 |     function autoSaveSettings() {
640 |         const settings = {
641 |             openaiApiKey: keyEl.value,
642 |             sessionInstructions: sessionInstructionsEl.value,
643 |             startInstructions: startInstructionsEl.value,
644 |             voice: voiceEl.value
645 |         };
646 |         for (const [k, v] of Object.entries(settings)) {
647 |             localStorage.setItem(k, v);
648 |         }
649 |         sendNewSettings();
650 |         autoSaveStatusEl.textContent =
651 |             "Settings autosaved at " + new Date().toLocaleTimeString();
652 |     }
653 | 
654 |     /**
655 |      * Reset settings and re-save them to localStorage.
656 |      * @returns {void}
657 |      */
658 |     function resetSettings() {
659 |         const settings = {
660 |             sessionInstructions: defaultSessionInstructions,
661 |             startInstructions: defaultStartInstructions,
662 |             endInstructions: defaultEndInstructions,
663 |             voice: defaultVoice
664 |         };
665 |         for (const [k, v] of Object.entries(settings)) {
666 |             localStorage.setItem(k, v);
667 |         }
668 |         loadSettings();
669 |         sendNewSettings();
670 |         console.log("Settings reset!");
671 |     }
672 | 
673 |     /**
674 |      * Send updated session instructions to the GPT server (if data channel is open).
675 |      * @returns {void}
676 |      */
677 |     function sendNewSettings() {
678 |         if (dc && dc.readyState === "open") {
679 |             const message = {
680 |                 type: "session.update",
681 |                 session: {
682 |                     instructions: sessionInstructionsEl.value,
683 |                     tools: gptFunctions,
684 |                     tool_choice: "auto",
685 |                 }
686 |             };
687 |             dc.send(JSON.stringify(message));
688 |             console.log("Sent updated session settings:", message);
689 |         } else {
690 |             console.log("Data channel not open. Please wait until session starts.");
691 |         }
692 |     }
693 | 
694 |     // Debounce the settings autosave
695 |     [keyEl, sessionInstructionsEl, startInstructionsEl, endInstructionsEl, voiceEl]
696 |         .forEach((input) => {
697 |             input.addEventListener("change", () => {
698 |                 clearTimeout(window.autoSaveTimeout);
699 |                 window.autoSaveTimeout = setTimeout(autoSaveSettings, 2000);
700 |             });
701 |         });
702 | 
703 |     /**
704 |      * Send the user's typed message to the GPT server over the data channel.
705 |      * @returns {void}
706 |      */
707 |     function sendText() {
708 |         const text = textInputEl.value.trim();
709 |         if (!text) return;
710 |         if (dc && dc.readyState === "open") {
711 |             const message = {
712 |                 type: "response.create",
713 |                 response: {
714 |                     input: [{
715 |                         type: "message",
716 |                         role: "user",
717 |                         content: [{
718 |                             type: "input_text",
719 |                             text: text
720 |                         }],
721 |                     }],
722 |                     max_output_tokens: 700
723 |                 }
724 |             };
725 |             dc.send(JSON.stringify(message));
726 |             appendOrUpdateLog("User (text): " + text, "user-message");
727 |             textInputEl.value = "";
728 |         } else {
729 |             appendOrUpdateLog("Data channel not open. Please wait until session starts.", "system-message");
730 |         }
731 |     }
732 | 
733 |     // Button event handlers for starting / ending sessions
734 |     startButtonEl.addEventListener("click", async () => {
735 |         if (pc?.connectionState === "connected") {
736 |             console.log("Session already started");
737 |             appendOrUpdateLog("Session already started.", "system-message");
738 |             return;
739 |         }
740 |         toggleSessionButtons(true);
741 |         await startSession();
742 |     });
743 | 
744 |     // End session button - cancel any in-progress response and close the session
745 |     endButtonEl.addEventListener("click", async () => {
746 |         if (pc?.connectionState === "closed") {
747 |             console.log(`No session to end. Connection state: ${pc.connectionState}`);
748 |             appendOrUpdateLog("No session to end.", "system-message");
749 |             return;
750 |         }
751 |         toggleSessionButtons(false);
752 |         // cancel any in-progress response - not needed for speech input due to turn detection
753 | 
754 |         if(assistantSpeaking)
755 |             dc.send(JSON.stringify({type: "response.cancel"}));
756 |         await endSession();
757 |     });
758 | 
759 |     // Show/hide settings form
760 |     settingsButtonEl.addEventListener("click", () => {
761 |         settingsFormEl.style.display =
762 |             (settingsFormEl.style.display === "none") ? "block" : "none";
763 |     });
764 | 
765 |     // Send on click or Enter press
766 |     sendButtonEl.addEventListener("click", sendText);
767 |     textInputEl.addEventListener("keydown", (ev) => {
768 |         if (ev.key === "Enter") {
769 |             sendText();
770 |         }
771 |     });
772 | 
773 |     loadSettings();
774 | </script>
775 | 
776 | </body>
777 | </html>
778 | 


--------------------------------------------------------------------------------