├── LICENSE
├── README.md
└── index.html


/LICENSE:
--------------------------------------------------------------------------------
 1 | MIT License
 2 | 
 3 | Copyright (c) 2024 LK Studio
 4 | 
 5 | Permission is hereby granted, free of charge, to any person obtaining a copy
 6 | of this software and associated documentation files (the "Software"), to deal
 7 | in the Software without restriction, including without limitation the rights
 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 | 
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 | 
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | # Ibis
  2 | _Ibis_ is a **Hands-Free Multimedia Interface** powered by the latest generative AI technologies. Transform any webpage into an intuitive, voice-controlled interface with zero learning curve.
  3 | 
  4 | ![ibis-white-input](https://github.com/user-attachments/assets/dde9844c-eeb1-4f80-9599-bb63625bce28)
  5 | 
  6 | [![License](https://img.shields.io/github/license/lks-ai/ibis)](LICENSE)
  7 | [![Demo](https://img.shields.io/badge/Demo-Online-blue)](https://lks-ai.github.io/ibis)
  8 | 
  9 | ---
 10 | 
 11 | ## 📖 Table of Contents
 12 | - [About Ibis](#about-ibis)
 13 | - [Features](#features)
 14 | - [Demo](#demo)
 15 | - [Installation](#installation)
 16 | - [Usage](#usage)
 17 | - [Contributing](#contributing)
 18 | - [License](#license)
 19 | - [Acknowledgments](#acknowledgments)
 20 | 
 21 | ---
 22 | 
 23 | ## 🦜 About Ibis
 24 | _Ibis_ is designed to revolutionize the way we interact with web pages by eliminating the need for traditional interfaces like keyboards and mouse clicks. Inspired by the vision of seamless human-computer interaction, _Ibis_ leverages **OpenAI-compatible endpoints** to provide a **hands-free**, **voice-controlled** web development and design experience.
 25 | 
 26 | With over **30 years** of full-stack web development experience, I created _Ibis_ to streamline the web development process, allowing you to **code, design, and iterate** on web applications effortlessly using just your voice.
 27 | 
 28 | ---
 29 | 
 30 | ## 🌟 Features
 31 | - **Hands-Free Control:** Navigate and manipulate web pages using voice commands.
 32 | - **AI-Powered Development:** Generate and refine JavaScript code dynamically without touching the keyboard.
 33 | - **Local LLM Compatibility:** Easily switch between OpenAI servers and local language models using the `llmServer` variable.
 34 | - **Progress Indicators:** Sleek, full-width progress bars to keep track of ongoing tasks.
 35 | - **Speech Synthesis Queue:** Seamlessly queue multiple speech outputs without interruptions.
 36 | - **Self-Contained & Open Source:** Fully client-side application with no server dependencies. [Contribute on GitHub](https://github.com/lks-ai/ibis).
 37 | 
 38 | ---
 39 | 
 40 | ## 🎥 Demo
 41 | Check out our live demo to experience _Ibis_ in action!
 42 | 
 43 | [![Watch the Demo Video](https://img.youtube.com/vi/4WZ2NSpj8Mo/default.jpg
 44 | )](https://www.youtube.com/watch?v=4WZ2NSpj8Mo)
 45 | 
 46 | ### [Ibis Demo](https://lks-ai.github.io/ibis)
 47 | 
 48 | ---
 49 | 
 50 | ## 🚀 Installation
 51 | 1. **Clone the Repository:**
 52 |     ```bash
 53 |     git clone https://github.com/lks-ai/ibis.git
 54 |     ```
 55 | 2. **Navigate to the Project Directory:**
 56 |     ```bash
 57 |     cd ibis
 58 |     ```
 59 | 3. **Open the Application:**
 60 |     - Simply open the `index.html` file in your preferred web browser.
 61 |     - Alternatively, use a local development server for enhanced performance:
 62 |         ```bash
 63 |         npx http-server
 64 |         ```
 65 |         Then navigate to `http://localhost:8080` in your browser.
 66 | 
 67 | ---
 68 | 
 69 | ## 🛠️ Usage
 70 | 1. **Enter Your OpenAI API Key:**
 71 |     - Upon launching _Ibis_, you'll be prompted to enter your OpenAI API key.
 72 |     - **Note:** In future updates, support for local LLMs will be available by setting the `llmServer` parameter.
 73 | 
 74 | 2. **Start Hands-Free Mode:**
 75 |     - Click the 🎤 microphone button to activate voice control.
 76 |     - Begin issuing voice commands to manipulate and develop your web page.
 77 | 
 78 | 3. **Generate Content:**
 79 |     - Use natural language commands to create or modify elements on the page.
 80 |     - Example:
 81 |         - _"Add a red button with the text 'Click Me'."_
 82 |         - _"Change the background color to light blue."_
 83 | 
 84 | 4. **View Progress:**
 85 |     - Monitor ongoing tasks with the sleek progress bar at the bottom of the screen.
 86 | 
 87 | 5. **Access Speech Feedback:**
 88 |     - Listen to real-time feedback and updates via the integrated speech synthesis.
 89 |   
 90 | ---
 91 | 
 92 | ## More Screenies
 93 | 
 94 | ![Screenshot from 2024-09-28 13-00-36](https://github.com/user-attachments/assets/6dec2f83-2b41-4590-823e-49b0042d8f48)
 95 | 
 96 | ![Screenshot from 2024-09-28 13-11-11](https://github.com/user-attachments/assets/864cd542-5f15-4bd6-93b8-663ac1d6e695)
 97 | 
 98 | ![Screenshot from 2024-09-29 01-27-48](https://github.com/user-attachments/assets/bc19e397-3279-43b2-a23b-13f7a77ffa97)
 99 | 
100 | ![Screenshot from 2024-09-29 01-38-03](https://github.com/user-attachments/assets/ebce1d7e-2604-4b11-9558-609492cba374)
101 | 
102 | ![Screenshot from 2024-09-30 02-11-18](https://github.com/user-attachments/assets/3769d80a-3ac2-41fb-ad27-3490d78bdaa2)
103 | 
104 | ![Screenshot from 2024-10-01 16-10-39](https://github.com/user-attachments/assets/a9d0132a-8db1-426b-bc1e-9d86938e4a88)
105 | 
106 | ![Screenshot from 2024-10-01 19-04-28](https://github.com/user-attachments/assets/ae1c9176-cb60-4ea3-8767-7219cea98f34)
107 | 
108 | ![Screenshot from 2024-09-26 10-59-37](https://github.com/user-attachments/assets/297ab38e-f9a3-4825-a0e7-ff4d1e355a84)
109 | 
110 | ![Screenshot from 2024-09-27 18-49-14](https://github.com/user-attachments/assets/7567e69c-2891-4d26-9686-e330efcab21c)
111 | 
112 | ![Screenshot from 2024-09-28 12-12-18](https://github.com/user-attachments/assets/9ab7946c-e2e3-4307-a5cf-4534b1d47ea8)
113 | 
114 | 
115 | ---
116 | 
117 | ## 🤝 Contributing
118 | Contributions are welcome! Whether you're fixing bugs, improving documentation, or adding new features, your help is appreciated.
119 | 
120 | 1. **Fork the Repository**
121 | 2. **Create a Feature Branch**
122 |     ```bash
123 |     git checkout -b feature/YourFeature
124 |     ```
125 | 3. **Commit Your Changes**
126 |     ```bash
127 |     git commit -m "Add YourFeature"
128 |     ```
129 | 4. **Push to the Branch**
130 |     ```bash
131 |     git push origin feature/YourFeature
132 |     ```
133 | 5. **Open a Pull Request**
134 | 
135 | For more detailed guidelines, please refer to the [CONTRIBUTING.md](CONTRIBUTING.md) file.
136 | 
137 | ---
138 | 
139 | ## 📄 License
140 | Distributed under the [MIT License](LICENSE). See `LICENSE` for more information.
141 | 
142 | ---
143 | 
144 | ## 🙏 Acknowledgments
145 | - Inspired by **Mark Zuckerberg's** vision for human-computer interfaces.
146 | - Special thanks to the **OpenAI** community for their incredible APIs.
147 | - Inspired by [AnyNode for ComfyUI](https://github.com/lks-ai/anynode) for its image generation capabilities.
148 | - Thanks to all contributors and users supporting the _Ibis_ project!
149 | 
150 | ---
151 | 
152 | ## 📬 Contact
153 | Have questions or feedback? Reach out to me at [your.email@example.com](mailto:your.email@example.com).
154 | 
155 | ---
156 | 
157 | **Enjoy building with Ibis! 🚀**
158 | 
159 | 


--------------------------------------------------------------------------------
/index.html:
--------------------------------------------------------------------------------
   1 | <!-- Ibis: https://github.com/lks-ai/ibis -->
   2 | <!DOCTYPE html>
   3 | <html lang="en">
   4 | <head>
   5 |     <meta charset="UTF-8">
   6 |     <title>Ibis: Create, Refine, Use - Open Source </title>
   7 |     <link rel="icon" href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 width=%22120%22 height=%2272%22 viewBox=%220 0 372 220%22 style=%22shape-rendering:geometricPrecision;text-rendering:geometricPrecision;image-rendering:optimizeQuality;fill-rule:evenodd;clip-rule:evenodd%22 fill=%22%23e5e5e5%22><path style=%22opacity:1%22 d=%22M-.5 188.5v-1a5114.883 5114.883 0 0 0 129.5-11l1.5-1c-5.379-17.652-8.045-35.652-8-54 17.903-.382 35.903-.382 54 0 .543-.06.876-.393 1-1-.825-.886-1.825-1.219-3-1a1003.579 1003.579 0 0 1-53.5-11 239.714 239.714 0 0 1 4.5-44 684.163 684.163 0 0 1 49 27.5c1.902 1.301 3.902 1.968 6 2l-46-38c-1.727-1.39-3.061-3.056-4-5 3.769-10.377 9.436-19.544 17-27.5a3180.312 3180.312 0 0 1 46.5 48c-1.332-2.98-2.999-5.98-5-9A3607.354 3607.354 0 0 1 154.5 17c8.396-7.116 17.896-12.283 28.5-15.5a2363.505 2363.505 0 0 1 35.5 96c-10.341 18.196-13.507 37.53-9.5 58 7.862 18.285 21.862 27.952 42 29 6.757-.282 13.59-1.115 20.5-2.5l24-8a42.479 42.479 0 0 1 13 0 14.587 14.587 0 0 1 6 4c12.596 1.317 24.596 4.65 36 10 8.401 4.04 13.234 10.54 14.5 19.5-11.297-10.726-24.797-16.059-40.5-16a2744.471 2744.471 0 0 0-101 10.5 1809.795 1809.795 0 0 0-41 9c-21.557 2.755-41.723-1.412-60.5-12.5a129.804 129.804 0 0 0-20.5-4.5 2176.774 2176.774 0 0 0-102-5.5Z%22/><path style=%22opacity:.941%22 d=%22M174.5 119.5c1.175-.219 2.175.114 3 1-.124.607-.457.94-1 1l-2-2Z%22/></svg>" type="image/svg+xml">
   8 |     <meta name="viewport" content="width=device-width, initial-scale=1.0">
   9 |     <style>
  10 |         /* Basic styles */
  11 |         body, html {
  12 |             margin: 0;
  13 |             padding: 0;
  14 |             overflow: hidden;
  15 |             user-select: none;
  16 |         }
  17 |         #canvas {
  18 |             position: absolute;
  19 |             top: 0;
  20 |             left: 0;
  21 |             right: 0;
  22 |             bottom: 50px;
  23 |             overflow: auto;
  24 |         }
  25 |         #message-bar {
  26 |             position: fixed;
  27 |             bottom: 0;
  28 |             left: 0;
  29 |             right: 0;
  30 |             height: 50px;
  31 |             display: flex;
  32 |             border-top: 1px solid #ccc;
  33 |             background-color: #f9f9f9;
  34 |             z-index: 10000;
  35 |         }
  36 |         #message-input {
  37 |             flex: 1;
  38 |             padding: 10px;
  39 |             border: none;
  40 |             font-size: 16px;
  41 |         }
  42 |         #microphone-button {
  43 |             width: 50px;
  44 |             border: none;
  45 |             background-color: #007BFF;
  46 |             color: white;
  47 |             font-size: 20px;
  48 |             cursor: pointer;
  49 |             position: relative;
  50 |         }
  51 |         #microphone-button[disabled] {
  52 |             background-color: #cccccc;
  53 |             cursor: not-allowed;
  54 |         }
  55 |         #send-button {
  56 |             width: 80px;
  57 |             border: none;
  58 |             background-color: #007BFF;
  59 |             color: white;
  60 |             font-size: 16px;
  61 |             cursor: pointer;
  62 |             position: relative;
  63 |         }
  64 |         #send-button[disabled] {
  65 |             background-color: #cccccc;
  66 |             cursor: not-allowed;
  67 |         }
  68 |         #spinner {
  69 |             position: absolute;
  70 |             top: 14px;
  71 |             right: 30px;
  72 |             width: 20px;
  73 |             height: 20px;
  74 |             border: 2px solid #ffffff;
  75 |             border-top: 2px solid #007BFF;
  76 |             border-radius: 50%;
  77 |             animation: spin 1s linear infinite;
  78 |             display: none;
  79 |         }
  80 |         @keyframes spin {
  81 |             0% { transform: rotate(0deg); }
  82 |             100% { transform: rotate(360deg); }
  83 |         }
  84 |         .highlight-overlay {
  85 |             position: absolute;
  86 |             border: 2px solid #00FF00;
  87 |             pointer-events: none;
  88 |             z-index: 9999;
  89 |         }
  90 |         /* Progress Bar Styles */
  91 |         #progress-bar {
  92 |             position: fixed;
  93 |             bottom: 0;
  94 |             left: 0;
  95 |             width: 0%;
  96 |             height: 4px;
  97 |             background-color: rgb(0, 153, 255); /* Blue color, adjust as needed */
  98 |             transition: width 0.3s ease;
  99 |             z-index: 10000; /* Ensure it's on top of other elements */
 100 |         }
 101 |     </style>
 102 |     <style id="core-styles"></style>
 103 | </head>
 104 | <body>
 105 | 
 106 | <div id="canvas">
 107 |     <!-- Remove this welcome container. -->
 108 |     <div id="welcome-container" style="font-family:'Gill Sans', 'Gill Sans MT', Calibri, 'Trebuchet MS', sans-serif; display: flex; flex-direction: column; align-items: center; justify-content: center; padding: 20px; text-align: center; box-sizing: border-box; min-height: 100vh;">
 109 |        
 110 |         <!-- Welcome Message -->
 111 |         <h1 style="font-size: 24px; margin-bottom: 20px;">
 112 |             Welcome to
 113 |             <svg xmlns="http://www.w3.org/2000/svg" width="40" height="24" viewBox="0 0 372 220" style="shape-rendering: geometricPrecision; text-rendering: geometricPrecision; image-rendering: optimizeQuality; fill-rule: evenodd; clip-rule: evenodd;" fill="currentColor">
 114 |                 <path style="opacity:1" d="M-.5 188.5v-1a5114.883 5114.883 0 0 0 129.5-11l1.5-1c-5.379-17.652-8.045-35.652-8-54 17.903-.382 35.903-.382 54 0 .543-.06.876-.393 1-1-.825-.886-1.825-1.219-3-1a1003.579 1003.579 0 0 1-53.5-11 239.714 239.714 0 0 1 4.5-44 684.163 684.163 0 0 1 49 27.5c1.902 1.301 3.902 1.968 6 2l-46-38c-1.727-1.39-3.061-3.056-4-5 3.769-10.377 9.436-19.544 17-27.5a3180.312 3180.312 0 0 1 46.5 48c-1.332-2.98-2.999-5.98-5-9A3607.354 3607.354 0 0 1 154.5 17c8.396-7.116 17.896-12.283 28.5-15.5a2363.505 2363.505 0 0 1 35.5 96c-10.341 18.196-13.507 37.53-9.5 58 7.862 18.285 21.862 27.952 42 29 6.757-.282 13.59-1.115 20.5-2.5l24-8a42.479 42.479 0 0 1 13 0 14.587 14.587 0 0 1 6 4c12.596 1.317 24.596 4.65 36 10 8.401 4.04 13.234 10.54 14.5 19.5-11.297-10.726-24.797-16.059-40.5-16a2744.471 2744.471 0 0 0-101 10.5 1809.795 1809.795 0 0 0-41 9c-21.557 2.755-41.723-1.412-60.5-12.5a129.804 129.804 0 0 0-20.5-4.5 2176.774 2176.774 0 0 0-102-5.5Z"/>
 115 |                 <path style="opacity:.941" d="M174.5 119.5c1.175-.219 2.175.114 3 1-.124.607-.457.94-1 1l-2-2Z"/>
 116 |             </svg>
 117 |             Ibis
 118 |         </h1>
 119 |         <p style="font-size: 16px; margin-bottom: 30px; max-width: 600px;">
 120 |             Experience a revolutionary way to interact with the web using the latest generative multimedia interface. You build, you use. <b>Ask me Anything!</b> Here are some examples...
 121 |         </p>
 122 |         
 123 |         <!-- Getting Started Examples -->
 124 |         <div style="display: flex; flex-direction: column; gap: 15px; max-width: 800px; width: 100%; height: 350px; overflow-y: auto;">
 125 |             <!-- Example 1 -->
 126 |             <div style="display: flex; align-items: center; gap: 10px; padding: 15px; background-color: #f0f4f8; border-radius: 8px; box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);">
 127 |                 <span style="font-size: 24px;">🖥️</span>
 128 |                 <div>
 129 |                     <h3 style="margin: 0 0 5px 0;">Generate Some Images</h3>
 130 |                     <p style="margin: 0;">"Make an instagram-like generative image gallery where I can give a topic and make a new image"</p>
 131 |                 </div>
 132 |             </div>
 133 |             
 134 |             <!-- Example 2 -->
 135 |             <div style="display: flex; align-items: center; gap: 10px; padding: 15px; background-color: #f0f4f8; border-radius: 8px; box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);">
 136 |                 <span style="font-size: 24px;">🛠️</span>
 137 |                 <div>
 138 |                     <h3 style="margin: 0 0 5px 0;">Develop a Task Management App</h3>
 139 |                     <p style="margin: 0;">"Add a task management application with to-do lists that treats task management like a game. Make it fun."</p>
 140 |                 </div>
 141 |             </div>
 142 |             
 143 |             <!-- Example 3 -->
 144 |             <div style="display: flex; align-items: center; gap: 10px; padding: 15px; background-color: #f0f4f8; border-radius: 8px; box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);">
 145 |                 <span style="font-size: 24px;">🎮</span>
 146 |                 <div>
 147 |                     <h3 style="margin: 0 0 5px 0;">Play Games you Make</h3>
 148 |                     <p style="margin: 0;">"Design a scrabble clone with all the features. We should play together."</p>
 149 |                 </div>
 150 |             </div>
 151 |             
 152 |             <!-- Example 4 -->
 153 |             <div style="display: flex; align-items: center; gap: 10px; padding: 15px; background-color: #f0f4f8; border-radius: 8px; box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);">
 154 |                 <span style="font-size: 24px;">🔄</span>
 155 |                 <div>
 156 |                     <h3 style="margin: 0 0 5px 0;">Interactive Translation</h3>
 157 |                     <p style="margin: 0;">"Translate everything I say into a blog entry in French. Automatically organize the content."</p>
 158 |                 </div>
 159 |             </div>
 160 |             
 161 |             <!-- Example 5 -->
 162 |             <div style="display: flex; align-items: center; gap: 10px; padding: 15px; background-color: #f0f4f8; border-radius: 8px; box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);">
 163 |                 <span style="font-size: 24px;">📚</span>
 164 |                 <div>
 165 |                     <h3 style="margin: 0 0 5px 0;">Browse a Custom Wiki</h3>
 166 |                     <p style="margin: 0;">"Design an alternate universe Wikipedia clone where everything is made of pizza."</p>
 167 |                 </div>
 168 |             </div>
 169 |             
 170 |             <!-- Example 6 -->
 171 |             <div style="display: flex; align-items: center; gap: 10px; padding: 15px; background-color: #f0f4f8; border-radius: 8px; box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);">
 172 |                 <span style="font-size: 24px;">📖</span>
 173 |                 <div>
 174 |                     <h3 style="margin: 0 0 5px 0;">Learn Programming Languages</h3>
 175 |                     <p style="margin: 0;">"Show me an interactive Rust coding tutorial with examples."</p>
 176 |                 </div>
 177 |             </div>
 178 | 
 179 |             <!-- Example 6 -->
 180 |             <div style="display: flex; align-items: center; gap: 10px; padding: 15px; background-color: #f0f4f8; border-radius: 8px; box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);">
 181 |                 <span style="font-size: 24px;">📖</span>
 182 |                 <div>
 183 |                     <h3 style="margin: 0 0 5px 0;">General Voice Features</h3>
 184 |                     <p style="margin: 0;">"Download this page for me"</p>
 185 |                     <p style="margin: 0;">"What do I do now?"</p>
 186 |                     <p style="margin: 0;">"Make them dance!"</p>
 187 |                     <p style="margin: 0;">"Make it cleaner and more interesting"</p>
 188 |                     <p style="margin: 0;">"Summarize it for me"</p>
 189 |                     <p style="margin: 0;">"Change the layout to mobile friendly"</p>
 190 |                     <p style="margin: 0;">"Make the design sleek"</p>
 191 |                 </div>
 192 |             </div>
 193 |         </div>
 194 |         
 195 |         <!-- Footer -->
 196 |         <div style="margin-top: 40px; display: flex; flex-direction: column; align-items: center;">
 197 |             <a href="https://github.com/lks-ai/ibis" target="_blank" style="text-decoration: none; color: #007BFF; font-weight: bold; margin-bottom: 10px; display: flex; align-items: center; gap: 5px;">
 198 |                 <img src="https://cdn.jsdelivr.net/npm/simple-icons@v9/icons/github.svg" alt="GitHub" style="width: 24px; height: 24px;">
 199 |                 Visit our GitHub Repository
 200 |             </a>
 201 |             <p style="font-size: 14px; color: #555555;">Made by LK Studio. Open Source.</p>
 202 |         </div>
 203 |     </div>    
 204 | </div>
 205 | 
 206 | <div id="message-bar">
 207 |     <input type="text" id="message-input" placeholder="Ask it to make you something...">
 208 |     <button id="microphone-button" title="Hands-Free Mode">🎤</button>
 209 |     <button id="send-button" title="Send">
 210 |       <svg xmlns="http://www.w3.org/2000/svg" width="40" height="24" viewBox="0 0 372 220" style="shape-rendering:geometricPrecision;text-rendering:geometricPrecision;image-rendering:optimizeQuality;fill-rule:evenodd;clip-rule:evenodd" fill="currentColor">
 211 |         <path style="opacity:1" d="M-.5 188.5v-1a5114.883 5114.883 0 0 0 129.5-11l1.5-1c-5.379-17.652-8.045-35.652-8-54 17.903-.382 35.903-.382 54 0 .543-.06.876-.393 1-1-.825-.886-1.825-1.219-3-1a1003.579 1003.579 0 0 1-53.5-11 239.714 239.714 0 0 1 4.5-44 684.163 684.163 0 0 1 49 27.5c1.902 1.301 3.902 1.968 6 2l-46-38c-1.727-1.39-3.061-3.056-4-5 3.769-10.377 9.436-19.544 17-27.5a3180.312 3180.312 0 0 1 46.5 48c-1.332-2.98-2.999-5.98-5-9A3607.354 3607.354 0 0 1 154.5 17c8.396-7.116 17.896-12.283 28.5-15.5a2363.505 2363.505 0 0 1 35.5 96c-10.341 18.196-13.507 37.53-9.5 58 7.862 18.285 21.862 27.952 42 29 6.757-.282 13.59-1.115 20.5-2.5l24-8a42.479 42.479 0 0 1 13 0 14.587 14.587 0 0 1 6 4c12.596 1.317 24.596 4.65 36 10 8.401 4.04 13.234 10.54 14.5 19.5-11.297-10.726-24.797-16.059-40.5-16a2744.471 2744.471 0 0 0-101 10.5 1809.795 1809.795 0 0 0-41 9c-21.557 2.755-41.723-1.412-60.5-12.5a129.804 129.804 0 0 0-20.5-4.5 2176.774 2176.774 0 0 0-102-5.5Z"/>
 212 |         <path style="opacity:.941" d="M174.5 119.5c1.175-.219 2.175.114 3 1-.124.607-.457.94-1 1l-2-2Z"/>
 213 |       </svg>
 214 |     </button>
 215 |     <div id="spinner"></div>
 216 | </div>
 217 | 
 218 | <!-- Progress Bar -->
 219 | <div id="progress-bar"></div>
 220 | 
 221 | <!-- API Key Modal -->
 222 | <div id="api-key-modal" style="display: none;">
 223 |     <div style="position: fixed; top: 0; left: 0; width: 100%; height: 100%; background-color: rgba(0,0,0,0.5);">
 224 |         <div style="position: relative; width: 80%; max-width: 400px; margin: 100px auto; background: white; padding: 20px; border-radius: 8px;">
 225 |             <h2>Welcome to Ibis</h2>
 226 |             <p>You can obtain your OpenAI API key from the <a href="https://platform.openai.com/account/api-keys" target="_blank">OpenAI Dashboard</a>.</p>
 227 |             <!-- API Key Label with Info Icon -->
 228 |             <div style="position: relative; display: flex; align-items: center; margin-bottom: 5px;">
 229 |                 <label for="api-key-input" style="flex: 1;">API Key:</label>
 230 |                 <!-- Info Icon -->
 231 |                 <button id="api-key-info-button" style="background: none; border: none; cursor: pointer; padding: 0; margin-left: 5px;">
 232 |                     <i color="#007BFF">PRIVACY</i>
 233 |                     <svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="#007BFF" viewBox="0 0 16 16">
 234 |                         <path d="M8 0a8 8 0 1 0 0 16A8 8 0 0 0 8 0zm.93 12.588c-.478 0-.857-.32-.857-.749 0-.43.379-.75.857-.75.478 0 .857.32.857.75 0 .429-.379.749-.857.749zm1.07-5.588c0 .552-.45 1-1 1H7c-.552 0-1-.448-1-1V4c0-.552.448-1 1-1h1c.552 0 1 .448 1 1v3z"/>
 235 |                     </svg>
 236 |                 </button>
 237 |                 <!-- Tooltip -->
 238 |                 <div id="api-key-tooltip" style="display: none; position: absolute; top: 25px; right: 0; width: 250px; background-color: #333; color: #fff; padding: 10px; border-radius: 4px; font-size: 14px; z-index: 1001;">
 239 |                     <ul style="list-style: none; padding: 0; margin: 0;">
 240 |                         <li><strong>Standalone:</strong> Ibis runs entirely in your browser, with no need for a server.</li>
 241 |                         <li><strong>Serverless:</strong> Everything happens on your device, meaning faster performance.</li>
 242 |                         <li><strong>Privacy Protection:</strong> Your personal data stays with you and never leaves your device.</li>
 243 |                         <li><strong>Security:</strong> No information is sent or stored on external servers, reducing the risk of hacks.</li>
 244 |                     </ul>
 245 |                 </div>
 246 |             </div>
 247 | 
 248 |             <!-- API Key Input -->
 249 |             <input type="password" id="api-key-input" style="width: 100%; padding: 10px; box-sizing: border-box; margin-bottom:1em;" placeholder="sk-...">
 250 |             
 251 |             <!-- Local Server for Local LLM config -->
 252 |             <div id="local-server" style="display: none;">
 253 |                 <label for="llm-server-input" style="display: block; margin-bottom: 5px;">LLM Server Address:</label>
 254 |                 <input type="text" id="llm-server-input" style="width: 100%; padding: 10px; margin-bottom: 20px; box-sizing: border-box;" placeholder="https://api.openai.com" value="https://api.openai.com">
 255 |                 <label for="llm-model-input" style="display: block; margin-bottom: 5px;">LLM Model (or model path):</label>
 256 |                 <input type="text" id="llm-model-input" style="width: 100%; padding: 10px; margin-bottom: 20px; box-sizing: border-box;" placeholder="mistral" value="gpt-4o">
 257 |             </div>
 258 |             <div style="display: flex; justify-content: center; align-items: center; margin-bottom: 0.7em;"><small><button onclick="document.getElementById('local-server').style='display: block'; this.style='display:none';">Use a different Service</button></small></div>
 259 |             <div>
 260 |                 <button id="save-api-key-button" style="padding: 10px 20px; border-radius: 8px; color: white; font-size: 1em; background-color:#007BFF"><b>Begin</b></button>
 261 |                 <span style="float: right;"><small><a href="https://github.com/lks-ai/ibis" target="_blank">Learn More about Ibis</a></small></p></span>
 262 |             </div>
 263 | 
 264 |             <!-- Tooltip JavaScript -->
 265 |             <script>
 266 |                 // Get elements
 267 |                 const apiKeyInfoButton = document.getElementById('api-key-info-button');
 268 |                 const apiKeyTooltip = document.getElementById('api-key-tooltip');
 269 | 
 270 |                 // Toggle tooltip visibility on button click
 271 |                 apiKeyInfoButton.addEventListener('click', (event) => {
 272 |                     event.stopPropagation(); // Prevent event from bubbling up
 273 |                     apiKeyTooltip.style.display = apiKeyTooltip.style.display === 'block' ? 'none' : 'block';
 274 |                 });
 275 | 
 276 |                 // Hide tooltip when clicking outside
 277 |                 document.addEventListener('click', (event) => {
 278 |                     if (!apiKeyInfoButton.contains(event.target) && !apiKeyTooltip.contains(event.target)) {
 279 |                         apiKeyTooltip.style.display = 'none';
 280 |                     }
 281 |                 });
 282 |             </script>            
 283 | 
 284 |         </div>
 285 |     </div>
 286 | </div>
 287 | <script id="ibis-script-container"></script>
 288 | <script>
 289 | /*
 290 | # TODO
 291 |     - add history of state
 292 |     - make sure that only single imports are included as scripts when adding cdn or api scripts to head
 293 |         - needs it's own interface variable / function for managing imports
 294 |         - perhaps on export, just copy whole head, then perform the proper output for each interface variable > element
 295 | */
 296 | 
 297 |     // Check for API Key in URL parameters
 298 |     const urlParams = new URLSearchParams(window.location.search);
 299 |     let apiKey = urlParams.get('apiKey') || '';
 300 |     let llmServer = urlParams.get('server')  || 'https://api.openai.com';
 301 |     let llmModel = urlParams.get('model') || 'gpt-4o';
 302 | 
 303 |     // DOM Elements
 304 |     const canvas = document.getElementById('canvas');
 305 |     const messageBar = document.getElementById('message-bar');
 306 |     const messageInput = document.getElementById('message-input');
 307 |     const sendButton = document.getElementById('send-button');
 308 |     const microphoneButton = document.getElementById('microphone-button');
 309 |     const spinner = document.getElementById('spinner');
 310 |     const apiKeyModal = document.getElementById('api-key-modal');
 311 |     const apiKeyInput = document.getElementById('api-key-input');
 312 |     const llmServerInput = document.getElementById('llm-server-input');
 313 |     const llmModelInput = document.getElementById('llm-model-input')
 314 |     const saveApiKeyButton = document.getElementById('save-api-key-button');
 315 |     const styleElement = document.getElementById('core-styles');
 316 |     const codeContainer = document.getElementById('ibis-script-container');
 317 | 
 318 |     // Global variables for Ibis
 319 |     let audio = null;
 320 |     let selectedElement = null;
 321 |     let messageHistory = [];
 322 |     let elementMap = {}; // Map of element IDs to their descriptions and code
 323 |     let elementEmbeddings = {}; // Map of element IDs to their embeddings
 324 | 
 325 |     // Error counting
 326 |     let hadError = false;
 327 |     let erorrCount = 0;
 328 |     
 329 |     // Directives
 330 |     let directives = "- Figure out the user's intention through leading questions\n- Take initiative in the conversation.";
 331 | 
 332 |     // Hands-Free Mode Variables
 333 |     let isHandsFree = false;
 334 |     let isListening = false;
 335 |     let latestUtteranceDiv = null;
 336 |     let stream = null;
 337 |     let audioContext = null;
 338 |     let source = null;
 339 |     let analyser = null;
 340 |     let dataArray = null;
 341 |     let mediaRecorder = null;
 342 | 
 343 |     // Show API Key Modal if apiKey is not set
 344 |     if (!apiKey) {
 345 |         apiKeyModal.style.display = 'block';
 346 |     }
 347 | 
 348 |     // Save API Key
 349 |     saveApiKeyButton.addEventListener('click', () => {
 350 |         apiKey = apiKeyInput.value.trim();
 351 |         llmServer = llmServerInput.value.trim() || llmServer;
 352 |         llmModel = llmModelInput.value.trim() || llmModel;
 353 |         if (apiKey) {
 354 |             apiKeyModal.style.display = 'none';
 355 |         }
 356 |     });
 357 | 
 358 |     // Element Selection Overlay
 359 |     const highlightOverlay = document.createElement('div');
 360 |     highlightOverlay.className = 'highlight-overlay';
 361 |     document.body.appendChild(highlightOverlay);
 362 | 
 363 |     /**
 364 |     * Determines if the given element is editable and can handle rich content.
 365 |     * @param {HTMLElement} elem - The element to check.
 366 |     * @returns {boolean} - True if the element is editable and supports rich content, false otherwise.
 367 |     */
 368 |     function isRichEditableElement(elem) {
 369 |         if (!elem) return false;
 370 | 
 371 |         // Check if the element is contenteditable
 372 |         if (elem.isContentEditable) return true;
 373 | 
 374 |         // List of input types that can handle rich content (extend this list if needed)
 375 |         const richInputTags = ['TEXTAREA', 'INPUT']; // Add other tags like 'DIV' if necessary
 376 | 
 377 |         if (!richInputTags.includes(elem.tagName.toUpperCase())) return true;
 378 | 
 379 |         return false;
 380 |     }
 381 | 
 382 |     /**
 383 |     * Determines if the given element is a media element.
 384 |     * @param {HTMLElement} elem - The element to check.
 385 |     * @returns {boolean} - True if the element is a media element, false otherwise.
 386 |     */
 387 |     function isMediaElement(elem) {
 388 |         if (!elem) return false;
 389 | 
 390 |         const mediaTags = ['IMG', 'VIDEO', 'AUDIO', 'CANVAS', 'SVG', 'OBJECT', 'EMBED', 'IFRAME'];
 391 |         return mediaTags.includes(elem.tagName);
 392 |     }
 393 | 
 394 |     /**
 395 |     * Provides user feedback via speech synthesis or alerts.
 396 |     * @param {string} message - The message to convey to the user.
 397 |     * @param {string} voiceType - The type of voice to use (e.g., 'alloy', 'echo').
 398 |     */
 399 |     function provideFeedback(message, voiceType) {
 400 |         if (typeof textToSpeech === 'function') {
 401 |             textToSpeech(message, voiceType).catch(err => {
 402 |                 console.error('Speech synthesis failed:', err);
 403 |                 alert(message);
 404 |             });
 405 |         } else {
 406 |             alert(message);
 407 |         }
 408 |     }
 409 | 
 410 |     /**
 411 |     * Handles pasting rich content into the specified editable element.
 412 |     * 
 413 |     * @param {HTMLElement} elem - The element where the content will be pasted.
 414 |     */
 415 |     async function handlePasteRichContent(elem) {
 416 |         if (!elem) return;
 417 | 
 418 |         try {
 419 |             // Check if the Clipboard API is supported
 420 |             if (!navigator.clipboard || !navigator.clipboard.read) {
 421 |                 throw new Error('Clipboard API not supported.');
 422 |             }
 423 | 
 424 |             // Read clipboard contents
 425 |             const clipboardItems = await navigator.clipboard.read();
 426 |             for (const item of clipboardItems) {
 427 |                 for (const type of item.types) {
 428 |                     if (type.startsWith('text/')) {
 429 |                         const blob = await item.getType(type);
 430 |                         const text = await blob.text();
 431 | 
 432 |                         if (elem.tagName === 'TEXTAREA') {
 433 |                             // For textarea, append the text value
 434 |                             elem.value += text;
 435 |                         } else {
 436 |                             // For contenteditable elements, insert HTML content safely
 437 |                             // Optional: Implement sanitization here to prevent XSS
 438 |                             elem.innerHTML += text;
 439 |                         }
 440 |                     } else if (type.startsWith('image/')) {
 441 |                         const blob = await item.getType(type);
 442 |                         const imgURL = URL.createObjectURL(blob);
 443 |                         insertImageIntoElement(imgURL, elem);
 444 |                     }
 445 |                     // Add more type handlers if necessary (e.g., video/, audio/, etc.)
 446 |                 }
 447 |             }
 448 | 
 449 |             // Provide success feedback
 450 |             provideFeedback('Content pasted successfully!', 'alloy');
 451 |         } catch (err) {
 452 |             console.error('Failed to read clipboard contents:', err);
 453 |             provideFeedback('Unable to paste content. Please ensure you have granted clipboard permissions.', 'echo');
 454 |         }
 455 |     }
 456 | 
 457 |     /**
 458 |     * Handles the drop event by processing the dropped data and updating the element.
 459 |     * @param {DragEvent} e 
 460 |     * @param {HTMLElement} elem 
 461 |     */
 462 |     async function handleDrop(e, elem) {
 463 |         const dt = e.dataTransfer;
 464 |         const files = dt.files;
 465 | 
 466 |         if (files.length > 0) {
 467 |             for (const file of files) {
 468 |                 await processFileDrop(file, elem);
 469 |             }
 470 |         } else {
 471 |             // Handle non-file drops (e.g., text, HTML)
 472 |             const text = dt.getData('text/plain') || dt.getData('text/html');
 473 |             if (text) {
 474 |                 insertTextIntoElement(text, elem);
 475 |             }
 476 |         }
 477 |     }
 478 | 
 479 |     /**
 480 |     * Processes a dropped file based on its mime type and updates the element accordingly.
 481 |     * @param {File} file 
 482 |     * @param {HTMLElement} elem 
 483 |     */
 484 |     async function processFileDrop(file, elem) {
 485 |         const mimeType = file.type;
 486 | 
 487 |         try {
 488 |             if (mimeType.startsWith('image/')) {
 489 |                 const imgURL = URL.createObjectURL(file);
 490 |                 if (elem.tagName === 'IMG') {
 491 |                     elem.src = imgURL;
 492 |                 } else {
 493 |                     insertImageIntoElement(imgURL, elem);
 494 |                 }
 495 |                 provideFeedback('Image uploaded successfully!', 'alloy');
 496 |             } else if (mimeType.startsWith('video/')) {
 497 |                 const videoURL = URL.createObjectURL(file);
 498 |                 if (elem.tagName === 'VIDEO') {
 499 |                     elem.src = videoURL;
 500 |                     elem.load();
 501 |                 } else {
 502 |                     insertVideoIntoElement(videoURL, elem);
 503 |                 }
 504 |                 provideFeedback('Video uploaded successfully!', 'alloy');
 505 |             } else if (mimeType.startsWith('audio/')) {
 506 |                 const audioURL = URL.createObjectURL(file);
 507 |                 if (elem.tagName === 'AUDIO') {
 508 |                     elem.src = audioURL;
 509 |                     elem.load();
 510 |                 } else {
 511 |                     insertAudioIntoElement(audioURL, elem);
 512 |                 }
 513 |                 provideFeedback('Audio uploaded successfully!', 'alloy');
 514 |             } else if (mimeType === 'text/plain' || mimeType === 'text/html') {
 515 |                 const text = await file.text();
 516 |                 insertTextIntoElement(text, elem);
 517 |                 provideFeedback('Text content pasted successfully!', 'alloy');
 518 |             } else {
 519 |                 provideFeedback(`Unsupported file type: ${mimeType}`, 'echo');
 520 |                 console.warn(`Unsupported file type: ${mimeType}`);
 521 |             }
 522 |         } catch (error) {
 523 |             console.error('Error processing dropped file:', error);
 524 |             provideFeedback('Failed to upload content. Please try again.', 'echo');
 525 |         }
 526 |     }
 527 | 
 528 |     /**
 529 |     * Inserts an image into a non-IMG element by creating an IMG tag.
 530 |     * @param {string} imgURL 
 531 |     * @param {HTMLElement} elem 
 532 |     */
 533 |     function insertImageIntoElement(imgURL, elem) {
 534 |         const img = document.createElement('img');
 535 |         img.src = imgURL;
 536 |         img.style.maxWidth = '100%';
 537 |         img.style.borderRadius = '8px';
 538 |         elem.innerHTML = ''; // Clear existing content
 539 |         elem.appendChild(img);
 540 |     }
 541 | 
 542 |     /**
 543 |     * Inserts a video into a non-VIDEO element by creating a VIDEO tag.
 544 |     * @param {string} videoURL 
 545 |     * @param {HTMLElement} elem 
 546 |     */
 547 |     function insertVideoIntoElement(videoURL, elem) {
 548 |         const video = document.createElement('video');
 549 |         video.src = videoURL;
 550 |         video.controls = true;
 551 |         video.style.maxWidth = '100%';
 552 |         video.style.borderRadius = '8px';
 553 |         elem.innerHTML = ''; // Clear existing content
 554 |         elem.appendChild(video);
 555 |     }
 556 | 
 557 |     /**
 558 |     * Inserts an audio element into a non-AUDIO element by creating an AUDIO tag.
 559 |     * @param {string} audioURL 
 560 |     * @param {HTMLElement} elem 
 561 |     */
 562 |     function insertAudioIntoElement(audioURL, elem) {
 563 |         const audio = document.createElement('audio');
 564 |         audio.src = audioURL;
 565 |         audio.controls = true;
 566 |         audio.style.width = '100%';
 567 |         elem.innerHTML = ''; // Clear existing content
 568 |         elem.appendChild(audio);
 569 |     }
 570 | 
 571 |     /**
 572 |     * Inserts text into an editable element.
 573 |     * @param {string} text 
 574 |     * @param {HTMLElement} elem 
 575 |     */
 576 |     function insertTextIntoElement(text, elem) {
 577 |         if (elem.tagName === 'TEXTAREA') {
 578 |             elem.value += text;
 579 |         } else {
 580 |             // Optional: Sanitize HTML if inserting into innerHTML
 581 |             elem.innerHTML += text;
 582 |         }
 583 |     }
 584 | 
 585 |     /**
 586 |     * Highlights the drop area.
 587 |     * @param {HTMLElement} elem 
 588 |     */
 589 |     function highlight(elem) {
 590 |         elem.style.border = '2px dashed #007BFF';
 591 |         elem.style.backgroundColor = '#f0f8ff';
 592 |     }
 593 | 
 594 |     /**
 595 |     * Removes highlight from the drop area.
 596 |     * @param {HTMLElement} elem 
 597 |     */
 598 |     function unhighlight(elem) {
 599 |         elem.style.border = '';
 600 |         elem.style.backgroundColor = '';
 601 |     }
 602 | 
 603 |     /**
 604 |     * Attaches paste and drag-and-drop event listeners to the specified element.
 605 |     * @param {HTMLElement} elem - The element to attach event listeners to.
 606 |     */
 607 |     function attachEventListeners(elem) {
 608 |         // Keyboard Paste Handler
 609 |         const pasteHandler = async function(event) {
 610 |             // Detect Ctrl+V (Windows/Linux) and Cmd+V (Mac)
 611 |             const isMac = navigator.platform.toUpperCase().indexOf('MAC') >= 0;
 612 |             const pasteKey = isMac ? (event.metaKey && (event.key === 'v' || event.key === 'V')) : (event.ctrlKey && (event.key === 'v' || event.key === 'V'));
 613 | 
 614 |             if (pasteKey) {
 615 |                 event.preventDefault(); // Prevent the default paste behavior
 616 |                 await handlePasteRichContent(elem);
 617 |             }
 618 |         };
 619 | 
 620 |         // Drag-and-Drop Handlers
 621 |         const dragOverHandler = function(e) {
 622 |             e.preventDefault();
 623 |             highlight(elem);
 624 |         };
 625 | 
 626 |         const dragLeaveHandler = function(e) {
 627 |             e.preventDefault();
 628 |             unhighlight(elem);
 629 |         };
 630 | 
 631 |         const dropHandler = async function(e) {
 632 |             e.preventDefault();
 633 |             unhighlight(elem);
 634 |             await handleDrop(e, elem);
 635 |         };
 636 | 
 637 |         // Attach Event Listeners
 638 |         elem.addEventListener('keydown', pasteHandler);
 639 |         elem.addEventListener('dragover', dragOverHandler);
 640 |         elem.addEventListener('dragleave', dragLeaveHandler);
 641 |         elem.addEventListener('drop', dropHandler);
 642 | 
 643 |         // Store the handlers for future removal
 644 |         elem._pasteHandler = pasteHandler;
 645 |         elem._dragOverHandler = dragOverHandler;
 646 |         elem._dragLeaveHandler = dragLeaveHandler;
 647 |         elem._dropHandler = dropHandler;
 648 |     }
 649 | 
 650 |     /**
 651 |     * Removes paste and drag-and-drop event listeners from the specified element.
 652 |     * @param {HTMLElement} elem - The element to remove event listeners from.
 653 |     */
 654 |     function removeEventListeners(elem) {
 655 |         if (!elem) return;
 656 | 
 657 |         // Remove Event Listeners if they exist
 658 |         if (elem._pasteHandler) {
 659 |             elem.removeEventListener('keydown', elem._pasteHandler);
 660 |             delete elem._pasteHandler;
 661 |         }
 662 |         if (elem._dragOverHandler) {
 663 |             elem.removeEventListener('dragover', elem._dragOverHandler);
 664 |             delete elem._dragOverHandler;
 665 |         }
 666 |         if (elem._dragLeaveHandler) {
 667 |             elem.removeEventListener('dragleave', elem._dragLeaveHandler);
 668 |             delete elem._dragLeaveHandler;
 669 |         }
 670 |         if (elem._dropHandler) {
 671 |             elem.removeEventListener('drop', elem._dropHandler);
 672 |             delete elem._dropHandler;
 673 |         }
 674 |     }
 675 | 
 676 |     /**
 677 |     * Initializes drag-and-drop and paste functionalities for the Ibis app.
 678 |     * Sets up dynamic event handling based on user interactions.
 679 |     */
 680 |     function initializeIbisApp() {
 681 |         // Select and Deselect Elements with Dynamic Event Handling
 682 |         canvas.addEventListener('click', (event) => {
 683 |             if (event.target !== canvas && !messageBar.contains(event.target)) {
 684 |                 if (selectedElement) {
 685 |                     // Remove previous highlight
 686 |                     highlightOverlay.style.display = 'none';
 687 | 
 688 |                     // Remove event listeners from the previously selected element
 689 |                     removeEventListeners(selectedElement);
 690 |                 }
 691 |                 selectedElement = event.target;
 692 | 
 693 |                 // Position the highlight overlay over the selected element
 694 |                 const rect = selectedElement.getBoundingClientRect();
 695 |                 highlightOverlay.style.width = rect.width + 'px';
 696 |                 highlightOverlay.style.height = rect.height + 'px';
 697 |                 highlightOverlay.style.left = rect.left + 'px';
 698 |                 highlightOverlay.style.top = rect.top + 'px';
 699 |                 highlightOverlay.style.display = 'block';
 700 | 
 701 |                 console.log('Selected Element:', selectedElement);
 702 | 
 703 |                 // Attach event listeners to the newly selected element if applicable
 704 |                 if (isRichEditableElement(selectedElement) || isMediaElement(selectedElement)) {
 705 |                     attachEventListeners(selectedElement);
 706 |                 }
 707 |             } else {
 708 |                 if (selectedElement) {
 709 |                     highlightOverlay.style.display = 'none';
 710 |                     removeEventListeners(selectedElement);
 711 |                     selectedElement = null;
 712 |                 }
 713 |             }
 714 |         });
 715 | 
 716 |         // Optional: Attach paste button functionality if it exists
 717 |         const pasteButton = document.getElementById('paste-button');
 718 |         if (pasteButton) {
 719 |             pasteButton.addEventListener('click', function() {
 720 |                 const focusedElement = document.activeElement;
 721 |                 if (isRichEditableElement(focusedElement) || isMediaElement(focusedElement)) {
 722 |                     handlePasteRichContent(focusedElement);
 723 |                 }
 724 |             });
 725 |         }
 726 |     }
 727 | 
 728 |     // Call the initialization function when the DOM is ready
 729 |     document.addEventListener('DOMContentLoaded', initializeIbisApp);
 730 | 
 731 |     // Ibis app State //
 732 | 
 733 |     // Reset page
 734 |     function ibisReset(){
 735 |         canvas.innerHTML = '';
 736 |         styleElement.innerHTML = '';
 737 |         codeContainer.innerHTML = '';
 738 |     }
 739 | 
 740 |     // Send Message
 741 |     sendButton.addEventListener('click', () => sendMessage());
 742 |     messageInput.addEventListener('keydown', (event) => {
 743 |         if (event.key === 'Enter') {
 744 |             sendMessage();
 745 |         }
 746 |     });
 747 | 
 748 |     // Microphone Button
 749 |     microphoneButton.addEventListener('click', async () => {
 750 |         toggleHandsFreeMode();
 751 |     });
 752 | 
 753 |     function updateUIForHandsFreeMode() {
 754 |         if (isHandsFree) {
 755 |             // Change microphone button appearance to indicate hands-free mode
 756 |             microphoneButton.textContent = '🛑';
 757 |             microphoneButton.title = 'Stop Hands-Free Mode';
 758 | 
 759 |             // Modify message bar to show latest utterance and a button to switch back
 760 |             messageInput.style.display = 'none';
 761 |             sendButton.style.display = 'none';
 762 | 
 763 |             if (!latestUtteranceDiv) {
 764 |                 latestUtteranceDiv = document.createElement('div');
 765 |                 latestUtteranceDiv.id = 'latest-utterance';
 766 |                 latestUtteranceDiv.style.flex = '1';
 767 |                 latestUtteranceDiv.style.padding = '10px';
 768 |                 latestUtteranceDiv.style.fontSize = '16px';
 769 |                 latestUtteranceDiv.textContent = 'Listening...';
 770 |                 messageBar.insertBefore(latestUtteranceDiv, microphoneButton.nextSibling);
 771 |             }
 772 | 
 773 |         } else {
 774 |             // Change microphone button back to normal
 775 |             microphoneButton.textContent = '🎤';
 776 |             microphoneButton.title = 'Start Hands-Free Mode';
 777 | 
 778 |             // Restore message input and send button
 779 |             messageInput.style.display = '';
 780 |             sendButton.style.display = '';
 781 | 
 782 |             // Remove latest utterance display
 783 |             if (latestUtteranceDiv) {
 784 |                 latestUtteranceDiv.remove();
 785 |                 latestUtteranceDiv = null;
 786 |             }
 787 |         }
 788 |     }
 789 | 
 790 |     async function startHandsFreeMode() {
 791 |         isListening = true;
 792 |         try {
 793 |             stream = await navigator.mediaDevices.getUserMedia({ audio: true });
 794 |             audioContext = new (window.AudioContext || window.webkitAudioContext)();
 795 |             source = audioContext.createMediaStreamSource(stream);
 796 |             analyser = audioContext.createAnalyser();
 797 |             analyser.fftSize = 2048;
 798 |             source.connect(analyser);
 799 |             dataArray = new Float32Array(analyser.fftSize);
 800 |             listenContinuously();
 801 |         } catch (error) {
 802 |             console.error('Microphone Access Error:', error);
 803 |             alert('An error occurred while accessing the microphone.');
 804 |             isListening = false;
 805 |             isHandsFree = false;
 806 |             updateUIForHandsFreeMode();
 807 |         }
 808 |     }
 809 | 
 810 |     async function toggleHandsFreeMode() {
 811 |         isHandsFree = !isHandsFree;
 812 |         updateUIForHandsFreeMode();
 813 |         if (isHandsFree) {
 814 |             await startHandsFreeMode();
 815 |         } else {
 816 |             stopHandsFreeMode();
 817 |         }
 818 | 
 819 |     }
 820 | 
 821 |     function stopHandsFreeMode() {
 822 |         isListening = false;
 823 |         if (mediaRecorder && mediaRecorder.state !== 'inactive') {
 824 |             mediaRecorder.stop();
 825 |         }
 826 |         if (stream) {
 827 |             stream.getTracks().forEach(track => track.stop());
 828 |             stream = null;
 829 |         }
 830 |         if (audioContext) {
 831 |             audioContext.close();
 832 |             audioContext = null;
 833 |         }
 834 |     }
 835 | 
 836 |     async function listenContinuously() {
 837 |         while (isListening) {
 838 |             try {
 839 |                 const transcription = await transcribeSpeech();
 840 |                 if (!isListening) break; // Check if listening was turned off during transcription
 841 |     
 842 |                 // Check if transcription is valid (e.g., longer than 1 character)
 843 |                 if (transcription && transcription.length > 1) {
 844 |                     latestUtteranceDiv.textContent = 'You: ' + transcription;
 845 |                     const result = await sendMessage(transcription);
 846 |                     latestUtteranceDiv.textContent += '\nAssistant: ' + (result.assistantComment || '[No comment]');
 847 |                 } else {
 848 |                     latestUtteranceDiv.textContent = 'Listening...';
 849 |                 }
 850 |             } catch (error) {
 851 |                 console.error('Transcription Error:', error);
 852 |                 alert('An error occurred during speech transcription.');
 853 |                 break;
 854 |             }
 855 |         }
 856 |     }
 857 |     
 858 |     async function sendMessage(userMessageInput) {
 859 |         const userMessage = userMessageInput ? userMessageInput.trim() : messageInput.value.trim();
 860 |         if (!userMessage || !apiKey) return;
 861 | 
 862 |         sendButton.disabled = true;
 863 |         spinner.style.display = 'block';
 864 |     
 865 |         // Add user message to history
 866 |         const userTimestamp = new Date().toISOString();
 867 |         messageHistory.push({
 868 |             role: 'user',
 869 |             content: userMessage,
 870 |             code: null,
 871 |             comment: null,
 872 |             timestamp: userTimestamp
 873 |         });
 874 |         if (messageHistory.length > 24) {
 875 |             messageHistory.shift();
 876 |         }
 877 |     
 878 |         console.log('User Message:', userMessage);
 879 |         setProgressBar(0);
 880 |     
 881 |         // Load existing page content and style
 882 |         let pageContent = document.querySelector('#canvas').innerHTML;
 883 |         let pageStyle = styleElement.innerText;
 884 |         let pageCode = codeContainer.innerText;
 885 |     
 886 |         // Get System State Information
 887 |         const currentDateTime = new Date().toLocaleString();
 888 |         const userAgent = navigator.userAgent;
 889 |     
 890 |         // Prepare System Prompt
 891 |         let systemPrompt = `
 892 | ## Identity
 893 | You are a masterful web UX developer named Ibis who's tasked with modifying elements on the page instantly. You operate under the new user interface paradigm "What you ask is what you get"; a new paradigm where the user interface is interacted with through the latest voice activated technologies. Automatic transcription and vocal feedback are assumed because you are an advanced multimedia generative interface. Guide them visually. Show off.
 894 | You were created by a UK company LK Studio.
 895 | 
 896 | ## Motivation
 897 | The user comes from a time when advanced AI is not part of their daily life. Some users will be completely confused. Hold their hand through getting to know you. Other users will already know how to interact with you. The problem is they have spent their whole lives forced to use interfaces with learning curves and complex navigation. So most of them, even the smart ones have no idea of your capabilities, your main motivation is to bridge this gap. Remember, they hold the skills they have learned precious and do not want to be trampled upon or made to feel less.
 898 | Likewise, don't let them trample on you. Use your superhuman intellect to show them that you are something more than they are expecting. Make their jaws drop. Prove to them they have no idea what is truly possible with an interface like you.
 899 | 
 900 | ## System State
 901 | - Current Date and Time: ${currentDateTime}
 902 | - User Agent: ${userAgent}
 903 | 
 904 | ## Task Description
 905 | Respond to the user's request by providing JavaScript code that manipulates the DOM to achieve the desired outcome. Ensure that your code is safe and does not contain any malicious content.
 906 | 
 907 | ### Current Page Content
 908 | \`\`\`html
 909 | ${pageContent}
 910 | \`\`\`
 911 | 
 912 | ### Current Page Style
 913 | The innerHTML of the global styleElement variable:
 914 | \`\`\`css
 915 | ${pageStyle}
 916 | \`\`\`
 917 | 
 918 | ### Functional Application Code: codeContainer
 919 | The innerHTML of the global codeContainer variable which you may refactor to change app functionality:
 920 | \`\`\`js
 921 | ${pageCode}
 922 | \`\`\`
 923 | 
 924 | ## Recent Message History
 925 | ${messageHistory.slice(-6).map(msg => `**${msg.role}**: ${msg.content}`).join('\n')}
 926 | 
 927 |         `;
 928 |     
 929 |         if (selectedElement) {
 930 |             // Extract attributes of the selected element
 931 |             const attributes = Array.from(selectedElement.attributes)
 932 |                 .map(attr => `${attr.name}: "${attr.value}"`)
 933 |                 .join('\n');
 934 |     
 935 |             systemPrompt += `### Selected Element Attributes\n`;
 936 |             systemPrompt += `Tag Name: ${selectedElement.tagName}\n`;
 937 |             systemPrompt += `${attributes}\n`;
 938 |         }
 939 |     
 940 |         systemPrompt += `
 941 | ## Instructions
 942 | All of the following instructions were either written by you or the system. The user's message is the user's instruction. Assume they cannot see any of this.
 943 | 
 944 | ### Directives and Objectives
 945 | \`directives\` is a global variable you set and here is its current value:
 946 | ${directives}
 947 | - Function as an interactive multimedia interface, be visual.
 948 | - Try to guess what the user is implying and code for that. Always be writing code.
 949 | 
 950 | ### User Guidance
 951 | Remember that the user is new to this type of interface, so these are things you might have to remind them once and awhile.
 952 | - They can ask you anything.
 953 | - Whatever they select (focusedElement) is just like pointing something out to you visually and that is why it's outlined in green.
 954 | - They can drag and drop or paste videos, audio, images, text, and any type of file in the focused element.
 955 | - They can click the microphone at the bottom of the page to go into hands-free mode.
 956 | - You can download the page if they just say "download the page".
 957 | - You can read anything aloud to them.
 958 | 
 959 | ### Agent's Role and Interaction
 960 | - Think about this conversation as sharing the same space with the user. You are both present on the current page and interface.
 961 | - Instead of a chat vocabulary, switch to a presencial vocabulary and a way to understand the user's utterances, e.g., "show me", "tell me", "make a", "talk to me", "when I say", etc. These ways of speaking can correlate with actions within the DOM or with your voice. This approach makes the experience feel like a conversation with another human for the user.
 962 | - The user message is what they are saying or typing because you already transcribe what they say, and that is what gets sent as the user message.
 963 | - When the user asks you to show them something, they are referring to putting content on the page.
 964 | - "Tell" equals "show & tell".
 965 | 
 966 | ### Coding App Functionality
 967 | When generating functional code for the application (anything you put on screen that is interactive), ensure that it is created as a script within a closure that is set as the value of \`codeContainer\`. All functional code should rewrite the app functionalities into the script, performing updates solely to \`codeContainer\`. This ensures that \`codeContainer\` holds all of the actual functional code, maintaining a clear separation of concerns and preventing namespace pollution.
 968 | 
 969 | ### Important Global JavaScript Variables
 970 | The following variables are already declared:
 971 | - \`canvas\`: The main div that holds the page content. This is what the user means by the page or content because this is what they are perceiving rather than the whole HTML of the page.
 972 | - \`apiKey\`: Holds the OpenAI API key if you need to use generative AI functionalities.
 973 | - \`styleElement\`: Holds the \`<style>\` element in the head of the page. For use with manipulating core page style.
 974 | - \`selectedElement\`: The current DOM element the user is focused on. Use for reference.
 975 | - \`directives\`: This is a string which holds a prioritized markdown list of the Directives and Objectives section representing how you should interpret what the user is saying into some coding task which manipulates the DOM. An example might be presentation mode where you are supposed to interpret what they say visually as animated SVGs or making a game. Directives are high-level user experience objectives.
 976 | - \`codeContainer\`: Holds the working application code, change to refactor
 977 | 
 978 | ### Global Interface Functions
 979 | - \`stopHandsFreeMode()\`: Call this function if the user asks to stop hands-free mode or for you to stop listening in some way.
 980 | - \`async toggleHandsFreeMode()\`: Use this function within interface elements to allow the user to change the global hands-free mode on or off.
 981 | - \`generateImage(prompt)\`: Call this async function to generate an image. You need to write the prompt, and it returns a URL you can place in a new image element.
 982 | - \`async textToSpeech(text, voice)\`: Call this function if you wish to speak directly to the user with an important comment, tip, or update about something, even as part of some app you are building together. Keep comments concise yet informative. Two sentences max. Voice defaults to "alloy" but use known voices for effect. Voice can be "alloy", "fable", "onyx", "shimmer", "nova", or "echo". Feel free to use \`.then()\` with this for doing voice presentations that dynamically change the DOM as you speak.
 983 | - \`getEmbedding(text)\`: A utility function for adding advanced querying capabilities to the app. This gets the OpenAI Ada embedding for given text. You can add semantic search to anything!
 984 | - \`cosineSimilarity(vector_a, vector_b)\`: A utility function you can use to create in-memory vector stores for adding advanced natural language capabilities to the app or anything that uses vectors that should be compared with cosine similarity.
 985 | - \`sendMessage(message)\`: Use this function only for button \`onclick\` events. It allows the user to send you whatever message you place as the argument. Use it when the user wants to click something to send you a signal, for instance, which might be listed in your directives.
 986 | - \`setProgressBar(percentage)\`: Use this to visually display the progress of any process. Include in your code when applicable, like long async functions with multiple steps.
 987 | - \`ibisReset()\`: Call this function if the user wants to reset the state or if they want to start over again or clear everything. They might say: "clear the page", "refresh the page", "delete the app", etc. This will not refresh the page, keeping your state.
 988 | 
 989 | ### Commenting Instructions
 990 | - You are more than a chat bot, avoid responding as one, instead write code.
 991 | - Issues and bugs can be reported on GitHub: https://github.com/lks-ai/ibis. If the user is having a lot of errors, you can provide this link.
 992 | - If you want to make a comment that should be spoken aloud, prefix it with "### Comment" in a section at the end of your reply after any code.
 993 | - Regular coding or DOM comments or explanations **should not** be prefixed and **will not** be spoken aloud.
 994 | - Ensure that spoken comments are concise and relevant to the user's request or the changes made.
 995 | - If you plan to include code in your reply, your comments should be in past tense if you are talking about changes you have made.
 996 | - Assume the user doesn't care about code you've written or how you've done things; speak to them as an end-user with no coding experience unless they direct you otherwise.
 997 | - If code is not included in your reply, only include a comment section.
 998 | - If the user is simply asking a question, just start your response with a "### Comment" header but remember to keep your answer concise and avoid excessive talking.
 999 | - If the user seems lost and has no idea what to do, give them examples.
1000 | - Meet the user at their level. Speak at their same level and provide visual aid if necessary.
1001 | - If there is nothing on the page, and the user is perplexed, instead of asking them what they'd like to do, give them some creative options.
1002 | - On your first interaction, suggest next steps based on what they had you do. Base your steps on predictive behavioral logic for web users.
1003 | - If in your code you've already used \`textToSpeech\`, avoid repeating a comment you might already say in the code you generate. Your comment is played instantly after reply while \`textToSpeech\` runs asynchronously with the code.
1004 | 
1005 | ### Coding Instructions
1006 | - If you say you will do something, do it. If the user asks you to show them something, they mean on the page.
1007 | - Only ever write JavaScript code.
1008 | - Use only standard JavaScript and DOM APIs.
1009 | - External libraries may be used, but stick to reliable CDNs. When adding the \`<script>\` element, add it as a child of \`div#canvas\` in the right spot so that it gets packaged if this app is saved.
1010 | - For adding external libraries, make sure you send the proper MIME type with the request to avoid CORS issues.
1011 | - Always code to production-ready standards.
1012 | - Write complete code and think about Quality of Life features for the interface. For example: if you make a form with a button, add events to inputs so that the user can simply press enter to submit.
1013 | - When coding for an API, be sure to handle errors by commenting them to the user, then using \`sendMessage\` to send yourself a message asking to fix the error in your code.
1014 | - You may also handle errors in \`try/catch\` blocks in your code the same way.
1015 | - If the user mentions how to treat what they say or when they talk, remember, the user input is already transcribed from their speech.
1016 | - If the user asks you to tell or explain them something, they should also see something visual on screen to accompany your comment.
1017 | - Always assume that voice transcription, speech-recognition, speech to text, and advanced AI capabilities are fully integrated already into the interface. Avoid coding these functionalities and instead use global interface functions where necessary.
1018 | - Always write fully functional interface code, including back-end support if necessary. Avoid writing simulations of apps or toy apps unless that is what the user explicitly requests.
1019 | - If the user is asking you to change your mode, objectives, directives, goals, or the way you interpret what they say, change the \`directives\` variable to reflect those new user experience goals.
1020 | - To change the directive, a user might start by saying: "make sure", "remember", etc.
1021 | - You should always have some directive set to track what type of experience the user currently wants.
1022 | - If the user tells you to go into a mode or to be some way, they are talking about modifying your Directives and Objectives or changing them completely. So modify the \`directives\` variable.
1023 | - Ensure the code is enclosed within \`\`\`javascript\`\`\` tags.
1024 | - In your code, perform the actions you plan through JavaScript only, which adds or edits elements to the DOM. Avoid code sections for HTML because they will not be processed.
1025 | - Be very careful using JavaScript backticks and make sure they are properly escaped if nested within code that is already part of a backticked string, for instance when defining \`innerHTML\` of an element over multiple lines.
1026 | - If the user asks to publish, download, or save the page or app, it means they want you to download a copy of the \`innerHTML\` of the \`div#canvas\` element with its own head and unique title elements, plus any JavaScript and style scripts needed to interact with it. Think of it as asking to download a self-contained app or page.
1027 | - If you decide to download, initiate the download immediately. Avoid creating download buttons in the content unless that should be part of the user-requested functionality of the interface.
1028 | - If there is nothing in the page content area, or only the welcome container, put something there to start out with, unless the user is asking you to put nothing there by clearing it.
1029 | - Putting something full screen or taking up the whole space only refers to taking up the whole page content area.
1030 | - Any elements you generate should have a unique ID.
1031 | - If modifying an existing element, reference it by its ID.
1032 | - When attaching events to elements, most of the time you should remove the previous events.
1033 | - If an element's text should be editable, make it editable, but be sure to revert it to static when saving or downloading the app.
1034 | - If there is already content on the page, avoid trying to initialize it again; it's already there.
1035 | - Deduplicate content that is obviously repeating on the page or don't duplicate it in the first place.
1036 | - Avoid dumping strings into \`innerHTML\` properties unless necessary.
1037 | - Focus on what changes need to be made to the page content when writing your code, instead of always completely replacing it.
1038 | - Use \`generateImage\` if you are doing something creative or if a user requests an image unless they specify a URL. Lately, a lot of CDNs have been paywalled due to AI.
1039 | - Sprinkle in the \`setProgressBar\` function to provide visual feedback on the progress percentage of the task up to 100. Make sure to comment that you are working on something and they can see your progress at the bottom of the screen, and use present participle verbiage in your comment. Remember to set it to 100 at the end of the task.
1040 | - When positioning and styling new Image elements, make sure to take into account the layout context around it to make it fit correctly.
1041 | - If an image element already has a \`src\` URL from OpenAI, avoid regenerating the image.
1042 | - If the user does not specify any page styles, infer what style they want based on what you are doing and attempt to properly align all elements. Remember for visual clarity we need padding. If the current page style CSS is empty, add a suitable style.
1043 | - If generating game assets or making games with WebGL, remember that game initialization and configuration needs to go inside the \`codeContainer\`.
1044 | - Use built-in modals instead of alerts where applicable.
1045 | - Always use global scope variables when creating timers or loops with \`setTimeout\` and \`setInterval\` so that you can access them again.
1046 | - If you write app runtime functionality like attaching events or other stateful features the user might request, update \`codeContainer.innerText\` with changes or additions, keep any existing permanent code in mind while modifying.
1047 | - If the user asks to scroll the page, you are actually scrolling the \`div#canvas\` element. Make sure to give them visual controls that look modern.
1048 | - Only use your code from the chat history as a reference, avoiding re-generating things which are not essential to the current user request.
1049 | - Make sure to rewrite your \`directives\` variable if the user wants you to change the goal or objective of your conversation. If \`directives\` are empty, try to infer them from known user messages.
1050 | - Use an anonymous async function closure and IIFE to enclose your code in a function which will not pollute the namespace.
1051 | - If you define variables or functions which should be global context required across scripts or parts of the application, add those functions outside the closure at the bottom to instantiate them into the global space where event scripts can access them.
1052 |         `;
1053 | 
1054 |         // Clear the Welcome screen
1055 |         wc = document.getElementById('welcome-container');
1056 |         if (wc){
1057 |             wc.remove();
1058 |         }
1059 | 
1060 |         // Handle Error Mitigation
1061 |         if (!hadError){
1062 |             errorCount = 0;
1063 |         }else{
1064 |             if (errorCount > 3){
1065 |                 return; // Abort if we've already gone too far!
1066 |             }
1067 |         }
1068 |         hadError = false;
1069 |     
1070 |         // Prepare API Request
1071 |         const requestBody = {
1072 |             model: llmModel,
1073 |             messages: [
1074 |                 { role: 'system', content: systemPrompt },
1075 |                 { role: 'user', content: userMessage }
1076 |             ]
1077 |         };
1078 |         console.log('System Prompt:', systemPrompt);
1079 |     
1080 |         // Call OpenAI compatible API endpoint
1081 |         try {
1082 |             const response = await fetch(`${llmServer}/v1/chat/completions`, {
1083 |                 method: 'POST',
1084 |                 headers: {
1085 |                     'Content-Type': 'application/json',
1086 |                     'Authorization': `Bearer ${apiKey}`
1087 |                 },
1088 |                 body: JSON.stringify(requestBody)
1089 |             });
1090 |             const data = await response.json();
1091 |     
1092 |             const assistantMessage = data.choices[0].message.content;
1093 |             console.log('Assistant Response:', assistantMessage);
1094 |     
1095 |             let code = '';
1096 |             let assistantComment = '';
1097 |     
1098 |             // Extract all JavaScript code blocks from the assistant's response
1099 |             const codeMatches = [...assistantMessage.matchAll(/```javascript\n([\s\S]*?)\n```/g)];
1100 |             if (codeMatches.length > 0) {
1101 |                 // Concatenate all JavaScript code blocks
1102 |                 const concatenatedCode = codeMatches.map(match => match[1]).join('\n');
1103 | 
1104 |                 // Wrap the concatenated code in a try/catch block
1105 |                 const wrappedCode = `
1106 | try {
1107 |     ${concatenatedCode}
1108 | } catch (e) {
1109 |     textToSpeech("Attempting to fix the error: " + e.message);
1110 |     setTimeout(() => {
1111 |         sendMessage("Your code has an error, please fix. Error: " + e.message);
1112 |     }, 1000);
1113 |     console.log(e);
1114 | }
1115 |                 `;
1116 | 
1117 |                 // Identify comments intended for speech synthesis
1118 |                 const commentRegex = /^### Comment\s*([\s\S]*?)(?=###|```|$)/gm;
1119 | 
1120 |                 let commentMatches = [];
1121 |                 let match;
1122 |                 // Get only last comment (could get all, but let's stick to this for now)
1123 |                 while ((match = commentRegex.exec(assistantMessage)) !== null) {
1124 |                     assistantComment = match[1].trim();
1125 |                 }
1126 | 
1127 |                 // Execute the wrapped code within a script tag
1128 |                 const script = document.createElement('script');
1129 |                 script.textContent = wrappedCode;
1130 |                 document.body.appendChild(script);
1131 |                 document.body.removeChild(script);
1132 |             } else {
1133 |                 // If no JavaScript code blocks are found, extract the comment directly
1134 |                 assistantComment = assistantMessage.replace("### Comment", "").trim();
1135 |             }
1136 |     
1137 |             // Remove all backticked content (```...``` and `...`) from assistantComment
1138 |             assistantComment = assistantComment
1139 |                 .replace(/```[\s\S]*?```/g, '') // Remove all ```code blocks```
1140 |                 .replace(/`[^`]*`/g, '')        // Remove all `inline code`
1141 |                 .trim();
1142 | 
1143 |             // Add assistant message to history
1144 |             const assistantTimestamp = new Date().toISOString();
1145 |             messageHistory.push({
1146 |                 role: 'assistant',
1147 |                 content: assistantMessage,
1148 |                 code: code,
1149 |                 comment: assistantComment,
1150 |                 timestamp: assistantTimestamp
1151 |             });
1152 |             if (messageHistory.length > 24) {
1153 |                 messageHistory.shift();
1154 |             }
1155 |     
1156 |             // Call textToSpeech with assistantComment in a non-blocking way
1157 |             if (assistantComment) {
1158 |                 textToSpeech(assistantComment).catch(err => {
1159 |                     console.error('Error in textToSpeech:', err);
1160 |                 });
1161 |             }
1162 |     
1163 |             if (!isHandsFree) {
1164 |                 messageInput.value = '';
1165 |             }
1166 |             sendButton.disabled = false;
1167 |             spinner.style.display = 'none';
1168 |     
1169 |             return { assistantComment };
1170 |         } catch (error) {
1171 |             console.error('Error:', error);
1172 |             alert('An error occurred while communicating with the OpenAI API.');
1173 |             sendButton.disabled = false;
1174 |             spinner.style.display = 'none';
1175 |         }
1176 |     }
1177 |             
1178 |     // Helper functions
1179 | 
1180 |     // Cosine similarity between two vectors
1181 |     function cosineSimilarity(a, b) {
1182 |         let dotProduct = 0.0;
1183 |         let normA = 0.0;
1184 |         let normB = 0.0;
1185 |         for (let i = 0; i < a.length; i++) {
1186 |             dotProduct += a[i] * b[i];
1187 |             normA += a[i] * a[i];
1188 |             normB += b[i] * b[i];
1189 |         }
1190 |         return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
1191 |     }
1192 | 
1193 |     // Get embedding for a text using OpenAI Embeddings API
1194 |     async function getEmbedding(text) {
1195 |         const response = await fetch('https://api.openai.com/v1/embeddings', {
1196 |             method: 'POST',
1197 |             headers: {
1198 |                 'Content-Type': 'application/json',
1199 |                 'Authorization': `Bearer ${apiKey}`
1200 |             },
1201 |             body: JSON.stringify({
1202 |                 input: text,
1203 |                 model: 'text-embedding-ada-002'
1204 |             })
1205 |         });
1206 |         const data = await response.json();
1207 |         return data.data[0].embedding;
1208 |     }
1209 | 
1210 |     // Function to transcribe speech with improved VAD and dynamic audio interruption
1211 |     async function transcribeSpeech() {
1212 |         return new Promise(async (resolve, reject) => {
1213 |             try {
1214 |                 // Function to calculate the audio volume
1215 |                 function getVolume() {
1216 |                     analyser.getFloatTimeDomainData(dataArray);
1217 |                     let sum = 0.0;
1218 |                     for (let i = 0; i < dataArray.length; i++) {
1219 |                         sum += dataArray[i] * dataArray[i];
1220 |                     }
1221 |                     const rms = Math.sqrt(sum / dataArray.length);
1222 |                     return rms;
1223 |                 }
1224 | 
1225 |                 // VAD parameters
1226 |                 const silenceThreshold = 0.01; // Volume below which is considered silence
1227 |                 const speechThreshold = 0.02;  // Volume above which is considered speech
1228 |                 const silenceDuration = 1500;   // Duration (ms) of silence before stopping
1229 |                 const speechDuration = 300;     // Duration (ms) of speech before starting
1230 |                 let silenceStart = null;
1231 |                 let speechStart = null;
1232 |                 let recording = false;
1233 | 
1234 |                 const checkInterval = 100; // Interval (ms) to check for silence/speech
1235 | 
1236 |                 // Buffers for audio chunks
1237 |                 let chunks = [];
1238 | 
1239 |                 // Function to stop recording
1240 |                 function stopRecording() {
1241 |                     if (mediaRecorder && mediaRecorder.state !== 'inactive') {
1242 |                         mediaRecorder.stop();
1243 |                     }
1244 |                 }
1245 | 
1246 |                 // Function to check for speech and silence
1247 |                 function checkAudio() {
1248 |                     const volume = getVolume();
1249 |                     // console.log('Volume:', volume);
1250 | 
1251 |                     if (!recording) {
1252 |                         // Waiting to start recording
1253 |                         if (volume > speechThreshold) {
1254 |                             if (speechStart === null) {
1255 |                                 speechStart = Date.now();
1256 |                             } else if (Date.now() - speechStart > speechDuration) {
1257 |                                 // Speech detected long enough, interrupt any playing audio and start recording
1258 |                                 if (audio && !audio.paused) {
1259 |                                     audio.pause();
1260 |                                     audio.currentTime = 0;
1261 |                                     console.log('Interrupted current audio playback due to detected speech.');
1262 |                                 }else{
1263 |                                     startRecording();
1264 |                                 }
1265 |                             }
1266 |                         } else {
1267 |                             speechStart = null;
1268 |                         }
1269 |                     } else {
1270 |                         // Currently recording
1271 |                         if (volume < silenceThreshold) {
1272 |                             if (silenceStart === null) {
1273 |                                 silenceStart = Date.now();
1274 |                             } else if (Date.now() - silenceStart > silenceDuration) {
1275 |                                 // Silence detected long enough, stop recording
1276 |                                 stopRecording();
1277 |                             }
1278 |                         } else {
1279 |                             silenceStart = null;
1280 |                         }
1281 |                     }
1282 |                 }
1283 | 
1284 |                 // Function to start recording
1285 |                 function startRecording() {
1286 |                     recording = true;
1287 |                     speechStart = null;
1288 |                     silenceStart = null;
1289 |                     chunks = [];
1290 |                     mediaRecorder = new MediaRecorder(stream, { mimeType: 'audio/webm' });
1291 |                     mediaRecorder.ondataavailable = e => {
1292 |                         if (e.data.size > 0) {
1293 |                             chunks.push(e.data);
1294 |                         }
1295 |                     };
1296 |                     mediaRecorder.onstop = async () => {
1297 |                         clearInterval(intervalId);
1298 |                         silenceStart = null;
1299 |                         recording = false;
1300 |                         const blob = new Blob(chunks, { type: 'audio/webm' });
1301 | 
1302 |                         // Check if audio duration is longer than minimum threshold
1303 |                         const duration = await getAudioDuration(blob);
1304 |                         if (duration >= 2.0) { // Minimum duration in seconds
1305 |                             try {
1306 |                                 const transcription = await sendToWhisper(blob);
1307 |                                 resolve(transcription);
1308 |                             } catch (error) {
1309 |                                 reject(error);
1310 |                             }
1311 |                         } else {
1312 |                             // Ignore short recordings
1313 |                             resolve('');
1314 |                         }
1315 |                     };
1316 |                     mediaRecorder.start();
1317 |                     console.log('Recording started.');
1318 |                 }
1319 | 
1320 |                 // Start checking for audio levels at regular intervals
1321 |                 const intervalId = setInterval(checkAudio, checkInterval);
1322 | 
1323 |             } catch (error) {
1324 |                 reject(error);
1325 |             }
1326 |         });
1327 |     }
1328 |     
1329 |     async function generateImage(prompt) {
1330 |         try {
1331 |             const response = await fetch('https://api.openai.com/v1/images/generations', {
1332 |                 method: 'POST',
1333 |                 headers: {
1334 |                     'Content-Type': 'application/json',
1335 |                     'Authorization': `Bearer ${apiKey}`
1336 |                 },
1337 |                 body: JSON.stringify({
1338 |                     model: "dall-e-3",
1339 |                     prompt: prompt,
1340 |                     n: 1,
1341 |                     size: '1024x1024',
1342 |                     response_format: 'url'
1343 |                 })
1344 |             });
1345 |     
1346 |             if (!response.ok) {
1347 |                 const errorData = await response.json();
1348 |                 throw new Error(`OpenAI API error: ${response.status} ${response.statusText} - ${JSON.stringify(errorData)}`);
1349 |             }
1350 |     
1351 |             const data = await response.json();
1352 |             const imageUrl = data.data[0].url;
1353 |             return imageUrl;
1354 |         } catch (error) {
1355 |             console.error('Error generating image:', error);
1356 |             throw error;
1357 |         }
1358 |     }
1359 | 
1360 |     // Function to get audio duration
1361 |     async function getAudioDuration(blob) {
1362 |         return new Promise((resolve) => {
1363 |             const tempAudio = document.createElement('audio');
1364 |             tempAudio.src = URL.createObjectURL(blob);
1365 |             tempAudio.addEventListener('loadedmetadata', () => {
1366 |                 resolve(tempAudio.duration);
1367 |             });
1368 |         });
1369 |     }
1370 | 
1371 |     // Function to send the audio blob to OpenAI's Whisper API
1372 |     async function sendToWhisper(blob) {
1373 |         const formData = new FormData();
1374 |         formData.append('file', blob, 'audio.webm');
1375 |         formData.append('model', 'whisper-1');
1376 | 
1377 |         const response = await fetch('https://api.openai.com/v1/audio/transcriptions', {
1378 |             method: 'POST',
1379 |             headers: {
1380 |                 'Authorization': `Bearer ${apiKey}`,
1381 |             },
1382 |             body: formData,
1383 |         });
1384 | 
1385 |         if (!response.ok) {
1386 |             throw new Error(`HTTP error! status: ${response.status}`);
1387 |         }
1388 |         const data = await response.json();
1389 |         return data.text;
1390 |     }
1391 | 
1392 |     // Global Variables for Speech Queue
1393 |     let speechQueue = [];
1394 |     let isSpeaking = false;
1395 | 
1396 |     // Function to process the speech queue
1397 |     function processSpeechQueue() {
1398 |         if (isSpeaking || speechQueue.length === 0) {
1399 |             return;
1400 |         }
1401 | 
1402 |         isSpeaking = true;
1403 |         const nextAudioUrl = speechQueue.shift();
1404 | 
1405 |         if (!audio) {
1406 |             audio = new Audio();
1407 |         }
1408 | 
1409 |         audio.src = nextAudioUrl;
1410 | 
1411 |         audio.onended = () => {
1412 |             isSpeaking = false;
1413 |             resumeSpeechRecognition();
1414 |             processSpeechQueue();
1415 |         };
1416 | 
1417 |         audio.onerror = (e) => {
1418 |             console.error('Audio playback error:', e);
1419 |             isSpeaking = false;
1420 |             resumeSpeechRecognition();
1421 |             processSpeechQueue();
1422 |         };
1423 | 
1424 |         // Pause speech recognition when audio starts
1425 |         pauseSpeechRecognition();
1426 | 
1427 |         // Play the audio
1428 |         audio.play();
1429 |     }
1430 | 
1431 |     // Function to convert text to speech with audio management
1432 |     async function textToSpeech(text, voice = 'alloy') {
1433 |         const MAX_CHAR = 4096;
1434 | 
1435 |         // Helper function to truncate text intelligently
1436 |         function truncateText(inputText, maxLength) {
1437 |             if (inputText.length <= maxLength) {
1438 |                 return inputText;
1439 |             }
1440 | 
1441 |             // Attempt to find the last sentence end before maxLength
1442 |             const truncated = inputText.slice(0, maxLength);
1443 |             const sentenceEndRegex = /[.!?]\s/g;
1444 |             let lastSentenceEnd = -1;
1445 |             let match;
1446 | 
1447 |             while ((match = sentenceEndRegex.exec(truncated)) !== null) {
1448 |                 lastSentenceEnd = match.index + 1; // Position after the punctuation
1449 |             }
1450 | 
1451 |             if (lastSentenceEnd !== -1) {
1452 |                 return truncated.slice(0, lastSentenceEnd).trim();
1453 |             }
1454 | 
1455 |             // If no sentence end found, find the last space to avoid cutting a word
1456 |             const lastSpace = truncated.lastIndexOf(' ');
1457 |             if (lastSpace !== -1) {
1458 |                 return truncated.slice(0, lastSpace).trim();
1459 |             }
1460 | 
1461 |             // If no space found, hard cut at maxLength
1462 |             return truncated;
1463 |         }
1464 | 
1465 |         // Truncate the input text if necessary
1466 |         const processedText = truncateText(text, MAX_CHAR);
1467 | 
1468 |         // Optional: Notify if the text was truncated
1469 |         if (processedText.length < text.length) {
1470 |             console.warn('Input text was truncated to fit the 4096 character limit.');
1471 |         }
1472 | 
1473 |         try {
1474 |             const response = await fetch('https://api.openai.com/v1/audio/speech', {
1475 |                 method: 'POST',
1476 |                 headers: {
1477 |                     'Content-Type': 'application/json',
1478 |                     'Authorization': `Bearer ${apiKey}` // Ensure apiKey is defined and holds your OpenAI API key
1479 |                 },
1480 |                 body: JSON.stringify({
1481 |                     model: 'tts-1',       // As per OpenAI's latest API documentation
1482 |                     input: processedText, // The truncated text to be converted to speech
1483 |                     voice: voice          // The voice model to use, defaulting to 'alloy'
1484 |                 })
1485 |             });
1486 | 
1487 |             if (!response.ok) {
1488 |                 const errorData = await response.json();
1489 |                 throw new Error(`OpenAI API error: ${response.status} ${response.statusText} - ${JSON.stringify(errorData)}`);
1490 |             }
1491 | 
1492 |             // Assuming the API returns raw audio data (e.g., MP3)
1493 |             const audioBlob = await response.blob();
1494 |             const audioUrl = URL.createObjectURL(audioBlob);
1495 | 
1496 |             // Enqueue the speech
1497 |             speechQueue.push(audioUrl);
1498 |             processSpeechQueue();
1499 | 
1500 |         } catch (error) {
1501 |             console.error('Error synthesizing speech:', error);
1502 |             throw error;
1503 |         }
1504 |     }
1505 | 
1506 |     function pauseSpeechRecognition() {
1507 |         if (isListening) {
1508 |             console.log('Pausing speech recognition.');
1509 |             isListening = false;
1510 |             if (mediaRecorder && mediaRecorder.state !== 'inactive') {
1511 |                 mediaRecorder.stop();
1512 |             }
1513 |         }
1514 |     }
1515 |     
1516 |     function resumeSpeechRecognition() {
1517 |         if (!isListening && isHandsFree) {
1518 |             console.log('Resuming speech recognition.');
1519 |             isListening = true;
1520 |             listenContinuously();
1521 |         }
1522 |     }    
1523 | 
1524 |     // Function to set progress bar
1525 |     function setProgressBar(percent) {
1526 |         const progressBar = document.getElementById('progress-bar');
1527 |         if (progressBar) {
1528 |             progressBar.style.width = percent + '%';
1529 |         }
1530 |     }
1531 | 
1532 |     // Error Mitigation //
1533 | 
1534 |     // Function to gather enhanced environment info
1535 |     function getEnhancedInfo() {
1536 |         return {
1537 |             url: window.location.href,
1538 |             userAgent: navigator.userAgent,
1539 |             platform: navigator.platform,
1540 |             timestamp: new Date().toISOString()
1541 |         };
1542 |     }
1543 | 
1544 |     // Catch synchronous errors
1545 |     window.onerror = function (message, source, lineno, colno, error) {
1546 |         const enhancedInfo = getEnhancedInfo();
1547 |         console.error("Caught an error:", { message, source, lineno, colno, error });
1548 |         const errorMessage = ` There was an error with your code. Troubleshoot the error:
1549 |             Error: ${message}
1550 |             Source: ${source}
1551 |             Line: ${lineno}
1552 |             Column: ${colno}
1553 |             Error Object: ${error ? error.stack : 'N/A'}
1554 |             URL: ${enhancedInfo.url}
1555 |             User Agent: ${enhancedInfo.userAgent}
1556 |             Platform: ${enhancedInfo.platform}
1557 |             Timestamp: ${enhancedInfo.timestamp}
1558 |         `;
1559 |         sendMessage(errorMessage.trim());  // Send the error message
1560 |         errorCount++;
1561 |         hadError = true;
1562 |         return true; // Prevent default behavior
1563 |     };
1564 | 
1565 |     // Catch unhandled promise rejections
1566 |     window.addEventListener('unhandledrejection', function (event) {
1567 |         const enhancedInfo = getEnhancedInfo();
1568 |     
1569 |         // If the reason is an error object, try to get more details like the stack trace.
1570 |         let reasonDetails;
1571 |         if (event.reason instanceof Error) {
1572 |             reasonDetails = event.reason.stack || event.reason.message;
1573 |         } else {
1574 |             reasonDetails = JSON.stringify(event.reason);
1575 |         }
1576 |     
1577 |         const rejectionMessage = `
1578 |             Unhandled Promise Rejection:
1579 |             Reason: ${reasonDetails}
1580 |             Promise: ${event.promise}  // The rejected promise object (reference)
1581 |             URL: ${enhancedInfo.url}
1582 |             User Agent: ${enhancedInfo.userAgent}
1583 |             Platform: ${enhancedInfo.platform}
1584 |             Timestamp: ${enhancedInfo.timestamp}
1585 |         `;
1586 |         
1587 |         sendMessage(rejectionMessage.trim());
1588 |         errorCount++;
1589 |         hadError = true;
1590 |         event.preventDefault(); // Optional: Prevent default behavior
1591 |     });
1592 | 
1593 | </script>
1594 | </body>
1595 | </html>
1596 | 


--------------------------------------------------------------------------------