├── .gitignore ├── .vscode └── tasks.json ├── 404.html ├── CNAME ├── Gemfile ├── LICENSE.md ├── README.md ├── _config.yml ├── _sass └── color_schemes │ └── brilliant.scss ├── community.md ├── favicon.ico ├── frame ├── charging-cradle-schematics.pdf ├── charging-cradle.stl ├── frame-lenses-mechanical-drawing.pdf ├── frame-mechanical-drawing.pdf ├── frame-rx-clip.stl ├── frame-schematics.pdf ├── frame-sdk-bluetooth-specs.md ├── frame-sdk-flutter.md ├── frame-sdk-lua.md ├── frame-sdk-python.md ├── frame-sdk.md ├── frame.md ├── frame.stl ├── hardware.md └── images │ ├── frame-bluetooth-connection-diagram.drawio.svg │ ├── frame-bluetooth-sending-bytes-diagram.drawio.svg │ ├── frame-bluetooth-sending-lua-diagram.drawio.svg │ ├── frame-camera.png │ ├── frame-charging-cradle.png │ ├── frame-charging-diagram.png │ ├── frame-debug-pinout.png │ ├── frame-display.png │ ├── frame-exploded-view.png │ ├── frame-hardware-block-diagram.drawio.svg │ ├── frame-imu.png │ ├── frame-microphone.png │ ├── frame-nose-bridge-diagram.png │ ├── frame-pairing-diagram.png │ ├── frame-splash.png │ ├── frame-sprite-engine.drawio.png │ ├── frame-vs-code-extension.png │ └── frame-ycbcr-colorspace.png ├── images ├── discord-link.png ├── frame-link.png ├── logo.svg └── monocle-link.png ├── index.md ├── monocle ├── building-apps.md ├── hardware.md ├── images │ ├── bluetooth-raw-data-service-sequence-diagram.svg │ ├── bluetooth-serial-service-sequence-diagram.svg │ ├── micropython-justify.drawio.svg │ ├── micropython-web-repl.png │ ├── monocle-back-cover.png │ ├── monocle-camera.png │ ├── monocle-cc-dimensions.png │ ├── monocle-charging-case.png │ ├── monocle-charging-pads.png │ ├── monocle-dimensions.png │ ├── monocle-display-fov.png │ ├── monocle-exploded-view.png │ ├── monocle-hw-block-diagram.svg │ ├── monocle-jtag.png │ ├── monocle-leds.png │ ├── monocle-microphone.png │ ├── monocle-network-flow.png │ ├── monocle-power-diagram.svg │ ├── monocle-ship-mode.png │ ├── monocle-splash.png │ ├── monocle-swd.png │ ├── monocle-touch-interface.png │ ├── noa-for-ios-screens.png │ └── vs-code-extension.png ├── micropython.md ├── monocle-body.step ├── monocle-body.stl ├── monocle-cc-dimensions.pdf ├── monocle-cc-schematics.pdf ├── monocle-dimensions.pdf ├── monocle-schematics.pdf ├── monocle.md └── update.md ├── noa ├── delete-account.md └── images │ └── delete-account.png └── robots.txt /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | # Files to ignore in source control 3 | *.gem 4 | .bundle 5 | .ruby-version 6 | .jekyll-cache 7 | .sass-cache 8 | _site 9 | Gemfile.lock 10 | node_modules 11 | 12 | # MacOS cache files 13 | .DS_Store 14 | 15 | # VSCode files 16 | .vscode/settings.json 17 | .brilliant/ 18 | 19 | # Diagrams.net backup and temp files 20 | *.bkp 21 | *.dtmp 22 | -------------------------------------------------------------------------------- /.vscode/tasks.json: -------------------------------------------------------------------------------- 1 | { 2 | "version": "2.0.0", 3 | "tasks": [ 4 | { 5 | "label": "Launch site", 6 | "type": "shell", 7 | "command": "bundle exec jekyll serve --livereload --open-url", 8 | "group": { 9 | "kind": "build", 10 | "isDefault": true 11 | } 12 | } 13 | ] 14 | } -------------------------------------------------------------------------------- /404.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | Page Not Found 6 | 7 | 11 | 12 | 13 |

Redirecting to the home page...

14 | 15 | 16 | 17 | -------------------------------------------------------------------------------- /CNAME: -------------------------------------------------------------------------------- 1 | docs.brilliant.xyz 2 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | # After modifying, run `bundle update`, then `bundle install` 2 | # Then, star the sit with: `bundle exec jekyll serve --livereload --open-url`` 3 | 4 | source "https://rubygems.org" 5 | 6 | gem "github-pages", "~> 228", group: :jekyll_plugins 7 | 8 | group :jekyll_plugins do 9 | gem 'jekyll-include-cache', "~> 0.2.1" 10 | gem "jekyll-redirect-from", "~> 0.16.0" 11 | gem "jekyll-seo-tag", "~> 2.8.0" 12 | gem "jekyll-sitemap", "~> 1.4.0" 13 | end 14 | 15 | # Windows and JRuby does not include zoneinfo files, so bundle the tzinfo-data 16 | # gem and associated library. 17 | platforms :mingw, :x64_mingw, :mswin, :jruby do 18 | gem "tzinfo", "~> 1.2" 19 | gem "tzinfo-data" 20 | 21 | # this was required on a fresh install of Ruby on Windows (see https://github.com/jekyll/jekyll/issues/8523): 22 | gem "webrick", "~> 1.8.1" 23 | end 24 | 25 | # Performance-booster for watching directories on Windows 26 | gem "wdm", "~> 0.1.1", :platforms => [:mingw, :x64_mingw, :mswin] 27 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | ISC License 2 | 3 | Copyright © 2024 Brilliant Labs Ltd. 4 | 5 | Permission to use, copy, modify, and/or distribute this software for any purpose 6 | with or without fee is hereby granted, provided that the above copyright notice 7 | and this permission notice appear in all copies. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH 10 | REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND 11 | FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, 12 | INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS 13 | OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER 14 | TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF 15 | THIS SOFTWARE. 16 | 17 | Unless specified otherwise (see below), the above license and copyright applies 18 | to all files within this repository. 19 | 20 | Individual files or subdirectories may include additional copyright holders, or 21 | may be licensed under different terms. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Brilliant Docs - Source 2 | 3 | This site is built with [Jekyll](https://jekyllrb.com) and [Just the Docs](https://github.com/just-the-docs/just-the-docs). It's all hosted here on GitHub using the GitHub's [Pages](https://pages.github.com) feature. 4 | 5 | **If you spot any errors** in our documentation, feel free to make an [issue](https://github.com/brilliantlabsAR/docs/issues). 6 | 7 | If you'd like to do some extensive editing, you can also fork/clone this repository and view the pages live editing. 8 | 9 | ## To set it up: 10 | 11 | 1. Ensure you have [Ruby installed](https://www.ruby-lang.org/en/documentation/installation/). On MacOS, ruby is already installed and ready to go. 12 | 13 | 1. Clone this repository: 14 | 15 | ```bash 16 | git clone https://github.com/brilliantlabsAR/docs.git 17 | ``` 18 | 19 | 1. Set up the environment: 20 | 21 | ```bash 22 | cd docs 23 | bundle install 24 | ``` 25 | 26 | 1. Open the project in your browser: 27 | 28 | ```bash 29 | bundle exec jekyll serve --livereload --open-url 30 | ``` 31 | 32 | That's it! As you edit the pages. The website will automatically refresh. Be sure to keep an eye on your terminal to spot any error messages while you're developing. -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | # Site settings 2 | title: Brilliant Documentation 3 | description: Technical documentation for all the Brilliant products. 4 | logo: /images/logo.svg 5 | remote_theme: just-the-docs/just-the-docs 6 | color_scheme: brilliant 7 | permalink: pretty 8 | heading_anchors: true 9 | url: https://docs.brilliant.xyz 10 | 11 | # Plugins 12 | plugins: 13 | - jekyll-include-cache 14 | - jekyll-redirect-from 15 | - jekyll-seo-tag 16 | - jekyll-sitemap 17 | 18 | # Header 19 | search_enabled: true 20 | aux_links_new_tab: true 21 | aux_links: 22 | YouTube: 23 | - https://www.youtube.com/@brilliantlabsAR 24 | Discord: 25 | - https://discord.com/invite/vDS9X7gdwg 26 | GitHub: 27 | - https://github.com/brilliantlabsAR 28 | Homepage: 29 | - https://www.brilliant.xyz 30 | 31 | # Footer 32 | footer_content: Copyright © 2024 Brilliant Labs 33 | last_edit_timestamp: true 34 | last_edit_time_format: "%b %e %Y" 35 | 36 | # Mermaid diagrams 37 | mermaid: 38 | version: "9.2.2" # Pick an available version from https://cdn.jsdelivr.net/npm/mermaid/ 39 | 40 | # Call-outs 41 | callouts: 42 | warning: 43 | title: Caution 44 | color: red 45 | note: 46 | title: Note 47 | color: blue 48 | 49 | # SEO settings 50 | twitter: 51 | username: opensourceeyes 52 | # card: 53 | social: 54 | name: Brilliant Labs 55 | links: 56 | - https://twitter.com/opensourceeyes 57 | - https://www.instagram.com/opensourceeyes/ 58 | - https://discord.com/invite/vDS9X7gdwg 59 | 60 | # Google analytics 61 | # ga_tracking: UA-162687810-2 62 | # ga_tracking_anonymize_ip: true # Use GDPR compliant Google Analytics settings 63 | 64 | # Exclude files 65 | exclude: 66 | - CNAME 67 | - Gemfile 68 | - README.md 69 | - LICENSE.md 70 | 71 | # Optimizations 72 | compress_html: 73 | clippings: all 74 | comments: all 75 | endings: all 76 | startings: [] 77 | blanklines: false 78 | profile: false 79 | -------------------------------------------------------------------------------- /_sass/color_schemes/brilliant.scss: -------------------------------------------------------------------------------- 1 | $color-scheme: dark; 2 | $body-background-color: #0F0F0F; 3 | $body-heading-color: $grey-lt-000; 4 | $body-text-color: $grey-lt-300; 5 | $link-color: #f288bf; 6 | $nav-child-link-color: $grey-dk-000; 7 | $sidebar-color: #0F0F0F; 8 | $base-button-color: $grey-dk-250; 9 | $btn-primary-color: #F288BF; 10 | $code-background-color: #31343f; // OneDarkJekyll default for syntax-one-dark-vivid 11 | $code-linenumber-color: #dee2f7; // OneDarkJekyll .nf for syntax-one-dark-vivid 12 | $feedback-color: darken($sidebar-color, 3%); 13 | $table-background-color: #1d1f23; 14 | $search-background-color: $grey-dk-250; 15 | $search-result-preview-color: $grey-dk-000; 16 | $border-color: $grey-dk-200; 17 | 18 | @import "./vendor/OneDarkJekyll/syntax"; // this is the one-dark-vivid atom syntax theme -------------------------------------------------------------------------------- /community.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Community Projects 3 | description: Technical documentation for Brilliant's AR devices. 4 | image: /images/monocle/monocle-splash.png 5 | nav_order: 5 6 | --- 7 | 8 | # Community Projects, Libraries and Resources 9 | {: .no_toc } 10 | 11 | --- 12 | 13 | Here you'll find some useful resources created and maintained by the Brilliant community. Use them for inspiration, or as a part of your own project. 14 | 15 | If you'd like to share your project too, feel free to edit this doc, and create a [pull request](https://github.com/brilliantlabsAR/docs/pulls). 16 | 17 | ## Apps 18 | 19 | - [Teleprompter for Google Slides](https://github.com/milesprovus/Monocle-Teleprompter) by milesprovus 20 | - [QR-code Scanner](https://github.com/milesprovus/Monocle-QR-Reader) by milesprovus 21 | - [OpenAI GPT client for Monocle](https://github.com/acui51/monocle-rizz) by acui51 22 | - [A heads up display workout app](https://github.com/simonevetere/monocle) by simonevetere and jdc-cunningham 23 | - [Web based drawing tool for Monocle](https://github.com/jdc-cunningham/oled-pixels-to-mpython) by jdc-cunningham 24 | - [Monocle Navigation App](https://github.com/semtexzv/MonoNav) by semtexzv. [Try it here](https://semtexzv.github.io/MonoNav/) 25 | - [WebRTC video streaming demo](https://github.com/jdc-cunningham/bl-monocle-video-stream) by jdc-cunningham 26 | - [Various MicroPython snippets and examples](https://github.com/simonevetere/micropython) by simonevetere 27 | - [Question Visualizer via Google Sheets](https://github.com/ironmanfpv/Project-Question-Visualizer-for-Monocle) by ironmanfpv 28 | - [Fitness Odometer for Monocle](https://github.com/ironmanfpv/Fitness-Odometer-for-Monocle) by ironmanfpv [Try it here](https://ironmanfpv.github.io/Fitness-Odometer-for-Monocle/index.html) 29 | - [Simple Weather & Time App](https://github.com/siyengar04/weatherapp) by siyengar04 30 | 31 | ## Libraries & Tools 32 | 33 | - [Monocle communication driver for Python](https://github.com/fixermark/brilliant-monocle-driver-python) by fixermark 34 | - [MicroPython vector graphics library](https://github.com/sathibault/vgrs/) by sathibault 35 | - [Graphical tool for building custom FPGA apps](https://github.com/sathibault/streamlogic-monocle-micropython/) by sathibault 36 | - [Dynamic file loader for Monocle](https://github.com/sintezcs/monocle-python-chunks-demo) by sintezcs 37 | - [Web based MicroPython code and snippet editor for Monocle](https://github.com/jdc-cunningham/bl-monocle-reactjs-pwa) by jdc-cunningham 38 | - [Web based Monocle snippets editor](https://monoclejs.glitch.me/) by [Fabien Benetou](https://twitter.com/utopiah/status/1650733250489221123) 39 | - [NodeJS communication library for Monocle](https://github.com/endanke/monocle-node-cli) by endanke 40 | - [Android image transfer library for Monocle](https://github.com/elizagamedev/monocleprototype) 41 | - [Flutter communication library for Monocle and Frame](https://pub.dev/packages/brilliant_ble) by [Uma Shankar](github.com/uma-shankar-gupta) 42 | - [Go library and battery life tester for Monocle](https://github.com/floren/monocle) by floren 43 | 44 | ## Hardware & Accessories 45 | 46 | - [3D printed accessories for Monocle](https://github.com/jdc-cunningham/bl-monocle-stuff) by jdc-cunningham 47 | 48 | ## AI Resources 49 | 50 | We're hard at work developing tools and apps for AI enabled workflows on Monocle. In the meantime, if you'd like to build your own applications, the links below might help you get started: 51 | 52 | - [GoWin GoAI IP](https://www.gowinsemi.com/en/market/featured_detail/11/) 53 | - [Tensorflow Lite for microcontrollers](https://www.tensorflow.org/lite) 54 | -------------------------------------------------------------------------------- /favicon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/favicon.ico -------------------------------------------------------------------------------- /frame/charging-cradle-schematics.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/charging-cradle-schematics.pdf -------------------------------------------------------------------------------- /frame/charging-cradle.stl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/charging-cradle.stl -------------------------------------------------------------------------------- /frame/frame-lenses-mechanical-drawing.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/frame-lenses-mechanical-drawing.pdf -------------------------------------------------------------------------------- /frame/frame-mechanical-drawing.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/frame-mechanical-drawing.pdf -------------------------------------------------------------------------------- /frame/frame-rx-clip.stl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/frame-rx-clip.stl -------------------------------------------------------------------------------- /frame/frame-schematics.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/frame-schematics.pdf -------------------------------------------------------------------------------- /frame/frame-sdk-bluetooth-specs.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Talking to Frame Over Bluetooth 3 | description: A reference on how to manually communicate with Frame over Bluetooth Low Energy 4 | image: /images/frame/frame-splash.png 5 | nav_order: 4 6 | parent: Frame SDK 7 | grand_parent: Frame 8 | --- 9 | 10 | # Talking to Frame Over Bluetooth 11 | {: .no_toc } 12 | 13 | --- 14 | 15 | You may want to communicate with Frame manually over Bluetooth Low Energy if we don't yet have an SDK available for your preferred development language, or if you just want to have full control of what's going on under the hood. 16 | 17 | Once you're communicating with Frame over Bluetooth, you can use the [Lua API Reference](/frame/frame-sdk-lua) to see everything you can tell it to do. 18 | 19 | 1. TOC 20 | {:toc} 21 | 22 | --- 23 | 24 | ## Pairing & Connecting 25 | 26 | Frame uses BLE bonding and must pair with a host device before any communication can take place. When attempting to connect to Frame using an SDK, Frame will automatically initiate pairing, which the OS handles and the user will have to agree to. 27 | 28 |
29 | Full Bluetooth Connection Details 30 | 31 | ### Un-Pairing 32 | If already paired, Frame must be reset using the pinhole button on the charging dock before it can be paired to a new host side device. Host-side devices must also remove previous bonding before a new bonding can be accepted. 33 | 34 | ### Bluetooth Connection Technical Diagram 35 | 36 | ![Bluetooth connection sequence diagram](/frame/images/frame-bluetooth-connection-diagram.drawio.svg) 37 |
38 | 39 | ## Bluetooth Service & Characteristics 40 | 41 | Frame has a single BLE service containing two characteristics: one for transmitting and one for receiving data. 42 | 43 | Lua strings can be sent on the TX characteristic as UTF-8 strings, and responses are returned back on the RX characteristic. The host device must enable notification on the RX characteristic in order to receive data. 44 | 45 | - Service UUID: 7A230001-5475-A6A4-654C-8431F6AD49C4 46 | - TX characteristic UUID: 7A230002-5475-A6A4-654C-8431F6AD49C4 47 | - RX characteristic UUID: 7A230003-5475-A6A4-654C-8431F6AD49C4 48 | 49 | ## Executing Lua Statements Over Bluetooth 50 | 51 | You can run Lua commands on Frame over Bluetooth using a REPL-like interaction. Frame does not contain a complete Lua REPL, but rather evaluates every message and only returns a message if it resulted in the Lua `print()` function being called, or an error. 52 | 53 | ```lua 54 | print('hello world') -- Returns 'hello world' on the RX characteristic 55 | 56 | print(1 + 2) -- Returns 3 57 | 58 | a = 1 + 2 -- Evaluates 1 + 2 and stores it in a, but does not return anything 59 | 60 | 1 + 2 -- Returns an error because 1 + 2 hasn't been assigned to anything 61 | ``` 62 | 63 | The length of TX data is limited to a maximum size of the host's chosen MTU length. This number also determines the maximum length of a Lua string that can be evaluated at a time. For larger scripts, they must first be saved to the device using the [file system API](/frame/frame-sdk-lua#file-system) using several `f:write()` operations, and then executed using `require()`. 64 | 65 | Responses are also limited to the MTU length. Should you want to return very long strings, they should be broken up into smaller `print()` statements. 66 | 67 | Common MTU lengths are from 27 to 251 bytes, depending on OS version, BT version, and more. To determine the MTU length that applies to your connection, use the following Lua function on Frame: `frame.bluetooth.max_length()`. More details in the [Lua Bluetooth API reference](/frame/frame-sdk-lua#bluetooth). To determine the MTU length on the host side, consult the documentation for your bluetooth library. 68 | 69 | ### Bluetooth Lua Execution Sequence Diagram 70 | 71 | ![Bluetooth Lua execution sequence diagram](/frame/images/frame-bluetooth-sending-lua-diagram.drawio.svg) 72 | 73 | ## Sending Data 74 | 75 | Sometimes it may be more efficient to send raw byte data rather than strings: for example, when transmitting graphical or microphone data. These can be sent on the same TX characteristic by simply prepending a byte of value `0x01` at the start of the payload. This will trigger a callback function if one has already been assigned by `frame.bluetooth.receive_callback()`. The total length of byte data that can be transmitted is therefore MTU - 4 bytes. 76 | 77 | Raw byte data can also be returned to the host device using the `frame.bluetooth.send()` function. This data is also limited to MTU - 4 bytes in length, and will be prepended by a value of `0x01` in the first byte of the payload. 78 | 79 | ### Bluetooth Raw Data Exchange Sequence Diagram 80 | 81 | ![Bluetooth Raw Data Exchange Sequence Diagram](/frame/images/frame-bluetooth-sending-bytes-diagram.drawio.svg) 82 | 83 | ## Control Characters 84 | 85 | While a Lua script is running, Frame will ignore any other Lua strings that are sent over Bluetooth (except [raw byte data](#sending-data) which can still be sent to trigger callbacks in the user logic - see below). To interrupt a running script, the following control signals can be sent on the TX characteristic to exit or restart the script: 86 | 87 | - Sending a single byte of value `0x03` ("break signal") will terminate any running script or loop. 88 | - Sending a single byte of value `0x04` ("reset signal") will clear all variables, and run `main.lua` if it exists. Equivalent to a reboot, or dipping in and out of the charging cradle. 89 | 90 | ## Lua Main Loop - A Special Case 91 | 92 | The Lua command `require('my_module')` executes the whole file `my_module.lua`. If this file contains a main loop, it may never return. Any subsequent Lua command sent on the TX characteristic *will not execute in the REPL* because the previous command has not returned - however raw data transmitted (with the `0x01` prefix) will still be handled by the `receive_callback()`. 93 | 94 | **This behavior is the basis of Frameside applications**: when a Lua file is sent to Frame and then run (`require`d), Frame no longer accepts individual Lua REPL commands, but thereafter all communication between the Host and Frame takes place with raw data messages on the TX characteristic (with the `0x01` prefix). The primary purpose of the Lua main loop (in conjunction with the data characteristic `receive_callback()`) is to receive messages from the Host and trigger corresponding Frameside behavior. In addition, data from Frame can be sent back on the RX characteristic (RX from the point of view of the Host) to trigger its action. 95 | 96 | For example, an application that takes photos from Frame's camera, sends them to the Host, runs a vision model, and then sends a text caption to Frame, might require a main Lua loop that receives 2 message types from the host: (1) capture a photo (and send to Host), and (2) display the specified text. The `frame-msg` package in the Frame SDK works precisely in this way. 97 | 98 | Lua loops can be interrupted with the break and reset signals outlined in the section above: [Control Characters](#control-characters). When first connecting to Frame, it may not be known in advance if a main loop is already running, so a program may need to send a break signal to break out of any running loop to ensure that subsequent Lua REPL statements execute (e.g. to send new application files.) 99 | 100 | Note: if a file named `main.lua` exists on the filesystem, it will automatically be run (`require`d) after reboot (or reset signal). If that file has a main loop, a break control character will need to be sent after the reboot (and after a short delay) to ensure that further Lua commands can be successfully executed. 101 | 102 | ## Firmware Updates 103 | 104 | Frame contains a mechanism for updating the complete system firmware via a built in bootloader. To enter the bootloader, call `frame.update()`. The device will then reboot and advertise with the name "Frame Update". 105 | 106 | To understand how the bootloader process works in more detail, visit the [Nordic DFU documentation](https://infocenter.nordicsemi.com/topic/sdk_nrf5_v17.1.0/lib_bootloader_modules.html). 107 | 108 | If no activity commences on the bootloader for more than 5 minutes, it will timeout and return back to the application firmware. If the firmware update process is interrupted, and the application firmware becomes invalid, the bootloader will wait forever until a valid firmware update is completed. 109 | 110 | Unlike the application firmware, the bootloader will continue to operate even while Frame is on charge. It's therefore possible to update while the device is docked to the charging cradle. In this case, once the update completes, the firmware will reload and then go to sleep right away. 111 | 112 | The latest release file of the official firmware is available from the [Frame codebase releases page](https://github.com/brilliantlabsAR/frame-codebase/releases) which you can include in your own host apps along with the DFU mechanism. 113 | -------------------------------------------------------------------------------- /frame/frame-sdk-flutter.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Flutter 3 | description: An overview of the Frame SDK for building Flutter mobile apps that talk to Frame AR glasses. 4 | image: /images/frame/frame-splash.png 5 | nav_order: 2 6 | parent: Frame SDK 7 | grand_parent: Frame 8 | nav_enabled: true 9 | 10 | --- 11 | 12 | # Frame SDK for Flutter 13 | {: .no_toc } 14 | 15 | --- 16 | 17 | The Frame SDK for Flutter is available in a low-level library ([frame_ble](https://pub.dev/packages/frame_ble/)) for Bluetooth LE connectivity using [Flutter Blue Plus](https://pub.dev/packages/flutter_blue_plus) with a limited Lua REPL, and an application-level library ([frame_msg](https://pub.dev/packages/frame_msg/)) for passing rich objects between your host program and Frame, such as images, streamed audio, IMU data and rasterized text. 18 | 19 | Additionally for Flutter developers the community has contributed even higher-level options for getting started quickly with single-page demo Frame applications, including specific helpers for computer vision pipelines. 20 | 21 | --- 22 | 23 | 1. TOC 24 | {:toc} 25 | 26 | --- 27 | 28 | ## `frame_ble` package: low-level connectivity 29 | 30 | | [![Available on pub.dev](https://img.shields.io/pub/v/frame_ble)](https://pub.dev/packages/frame_ble){:target="_blank"} | [API Reference](https://pub.dev/documentation/frame_ble/latest/){:target="_blank"} | [Source](https://github.com/CitizenOneX/frame_ble){:target="_blank"} | [Example](https://github.com/CitizenOneX/frame_ble/tree/main/example){:target="_blank"} | 31 | 32 | The `BrilliantBluetooth` class allows a host to connect to Frame, initiating pairing if required. A sequence of Lua strings can be sent to Frame for execution, optionally returning their results as strings. Transmitted and received strings must be shorter than the Bluetooth MTU (`BrilliantDevice.maxStringLength()`). 33 | 34 | [Frame-specific Lua APIs](frame-sdk-lua.md) can be called to draw on the display, sample the accelerometer, put Frame into firmware update mode, set the tap and user data callbacks, and more. 35 | 36 | In addition to Lua strings, `BrilliantDevice` can also send control commands (break and reset), as well as custom user data messages that can be handled by a user-defined data handler function on Frame. See [Talking to Frame Over Bluetooth](frame-sdk-bluetooth-specs.md) for more details. 37 | 38 | Consult the [API Reference](https://pub.dev/documentation/frame_ble/latest/){:target="_blank"} for the complete list of functions available in `frame_ble`, and the [Example](https://github.com/CitizenOneX/frame_ble/blob/main/example/lib/main.dart){:target="_blank"} to see them in action. Don't see an example you need? Ask in [Discord](https://discord.com/invite/vDS9X7gdwg){:target="_blank"} to have one added. 39 | 40 | Note: It is expected that most application developers will prefer to use the [frame_msg package](https://pub.dev/packages/frame_msg) to make use of first-class messages and efficient transport for sprites, images, audio clips, and to avoid having to wrangle Bluetooth packet fragmentation and assembly. The `frame_ble` package, with only a dependency on the `flutter_blue_plus` library, is still a handy choice for `"Hello, World!"`, simple utilities, or as the basis of a custom application framework in Flutter. 41 | 42 | Note: Frame SDK for Flutter is under active development and it is expected that some `frame_ble` classes may change to harmonize with the Python SDK naming, while retaining some differences due to the mobile context. 43 | 44 | ### Example usage 45 | {: .no_toc } 46 | ```dart 47 | // ... 48 | Future _showCounterOnFrame() async { 49 | // connect to Frame the first time 50 | if (_frame == null) { 51 | 52 | // request permission to use Bluetooth 53 | try { 54 | await BrilliantBluetooth.requestPermission(); 55 | } catch (e) { 56 | print('Error requesting permission to use Bluetooth: $e'); 57 | return; 58 | } 59 | 60 | // look for a Frame and connect to it 61 | BrilliantScannedDevice? scannedFrame; 62 | try { 63 | scannedFrame = await BrilliantBluetooth.scan().first; 64 | } catch (e) { 65 | print('Error scanning for Frame: $e'); 66 | return; 67 | } 68 | 69 | try { 70 | // connect to the scanned Frame 71 | _frame = await BrilliantBluetooth.connect(scannedFrame); 72 | 73 | print('Found Frame: $_frame'); 74 | } 75 | catch (e) { 76 | print('Error connecting to Frame: $e'); 77 | return; 78 | } 79 | 80 | // send a break signal to stop any running Lua main loop 81 | await _frame!.sendBreakSignal(); 82 | } 83 | 84 | // send the counter to Frame for display 85 | if (_frame?.state == BrilliantConnectionState.connected) { 86 | await _frame!.sendString( 87 | 'frame.display.text("Hello, World! ($_counter)", 1, 1) frame.display.show()', 88 | awaitResponse: false 89 | ); 90 | } 91 | else { 92 | print('Frame is not connected'); 93 | _frame = null; 94 | } 95 | } 96 | // ... 97 | ``` 98 | 99 | | Selected Examples | 100 | |-------------------| 101 | | ["Hello, World!"](https://github.com/CitizenOneX/frame_ble/blob/main/example/lib/main.dart){:target="_blank"} | 102 | 103 | --- 104 | 105 | ## `frame_msg` package: application-level messaging 106 | 107 | | [![Available on pub.dev](https://img.shields.io/pub/v/frame_msg)](https://pub.dev/packages/frame_msg){:target="_blank"} | [API Reference](https://pub.dev/documentation/frame_msg/latest/){:target="_blank"} | [Source](https://github.com/CitizenOneX/frame_msg){:target="_blank"} | Example | 108 | 109 | `frame-msg` enables both the hostside and Frameside halves of an application to **communicate using richly typed messages** - for example transmitting Sprites to Frame with width, height, palette data and pixel data attributes. (See the code below for an example.) 110 | 111 | Additionally, `frame-msg` uses a programming model in which **a main event loop is running on Frame**. As a result, messages are exchanged asychronously and handled in an event-driven way. This is in contrast to the "Lua REPL" model, in which single statements are sent to Frame to be executed, and the host attempts to block until execution has completed. In a REPL-style model, it would not be practical to create an application that for example continuously streams audio from Frame to host, and simultaneously sends text back for display (a live translation app). In contrast, with a main event loop running on Frame it is possible to send some audio back to the host if some is available, and check for messages indicating text is ready for display, then loop around again. 112 | 113 | ### How it works 114 | {: .no_toc } 115 | 116 | In order to **serialize and deserialize large objects**, use `frame_ble`'s `BrilliantDevice.send_message()` function that **fragments large payloads** into multiple Bluetooth packets marked with a matching **`msg_code`**, and **efficiently reassembles them on Frame**. After parsing the reconstructed payload into the desired type, a complete object is made available for the application's main loop in Lua. 117 | 118 | Handling **rich message types** is done by providing symmetrical **packing code and parsing code** for standard types. 119 | 120 | * For "Transmit" (`Tx`) message types, the packing (serialization) code is in the Dart/Flutter type (e.g. `TxSprite.pack()`) and the parsing (deserialization) code is in the corresponding Lua module (`sprite.lua` / `parse_sprite()`). 121 | 122 | * For "Receive" (`Rx`) message types, e.g. `RxPhoto`, the serialization code is present implicitly in the Lua module `camera.lua` in `capture_and_send()`, and the deserialization code is present in the `RxPhoto` class. 123 | 124 | This approach presents a natural extension point for user-defined types too: a custom application-specific message might contain an icon, a heading and some detailed text - and this can all be sent together and used all at once on Frame to update the display. By implementing the packing and parsing code for user-defined types, custom objects can be used in applications in the same way as the standard types. 125 | 126 | ### Startup sequence 127 | {: .no_toc } 128 | 129 | `frame-msg` applications tend to follow a startup sequence that ensures Frame is in a known state for the application to run, namely: 130 | * Optionally send a break/reset/break signal sequence to ensure that Frame is in Lua REPL mode and its memory is freshly cleared 131 | * Upload standard Lua libraries to Frame: `data` for handling all messaging, plus any others to support specific messages the application uses (e.g. `sprite`, `plain_text`, etc.) 132 | * Upload the main Frame application Lua file (e.g. `my_frame_app.lua`). This can often be an exact copy of an example main Lua file that handles the same message types. 133 | * Start the main application loop on Frame (after which no Lua REPL commands will be accepted by Frame). 134 | 135 | After the Frameside application reports that it has started, the hostside and Frameside application halves begin messaging each other asynchronously: passing text, images, audio etc. 136 | 137 | 138 | ## Example usage 139 | {: .no_toc } 140 | `sprite_display.dart` 141 | ```dart 142 | // ... 143 | // connect to Frame using `frame-ble` in the usual way 144 | // frame = await BrilliantBluetooth.connect(scannedFrame); 145 | // ... 146 | Future run() async { 147 | // Open the file picker 148 | FilePickerResult? result = await FilePicker.platform.pickFiles( 149 | type: FileType.custom, 150 | allowedExtensions: ['png'], 151 | ); 152 | 153 | if (result != null) { 154 | File file = File(result.files.single.path!); 155 | Uint8List pngBytes = await file.readAsBytes(); 156 | 157 | // send the Sprite to Frame 158 | var msg = TxSprite.fromPngBytes(pngBytes: pngBytes); 159 | await frame!.sendMessage(0x20, msg.pack()); 160 | } 161 | } 162 | // ... 163 | ``` 164 | 165 | `sprite_frame_app.lua` 166 | ```lua 167 | local data = require('data.min') 168 | local sprite = require('sprite.min') 169 | 170 | -- Phone to Frame flags 171 | USER_SPRITE = 0x20 172 | 173 | -- register the message parsers so they are automatically called when matching data comes in 174 | data.parsers[USER_SPRITE] = sprite.parse_sprite 175 | 176 | -- Main app loop 177 | function app_loop() 178 | frame.display.text('Frame App Started', 1, 1) 179 | frame.display.show() 180 | 181 | -- tell the host program that the frameside app is ready (waiting on await_print) 182 | print('Frame app is running') 183 | 184 | while true do 185 | rc, err = pcall( 186 | function() 187 | -- process any raw data items, if ready 188 | local items_ready = data.process_raw_items() 189 | 190 | -- one or more full messages received 191 | if items_ready > 0 then 192 | 193 | if data.app_data[USER_SPRITE] ~= nil then 194 | local spr = data.app_data[USER_SPRITE] 195 | 196 | -- set the palette in case it's different to the standard palette 197 | sprite.set_palette(spr.num_colors, spr.palette_data) 198 | 199 | -- show the sprite 200 | frame.display.bitmap(1, 1, spr.width, 2^spr.bpp, 0, spr.pixel_data) 201 | frame.display.show() 202 | 203 | -- clear the object and run the garbage collector right away 204 | data.app_data[USER_SPRITE] = nil 205 | collectgarbage('collect') 206 | end 207 | 208 | end 209 | 210 | -- can't sleep for long, might be lots of incoming bluetooth data to process 211 | frame.sleep(0.001) 212 | end 213 | ) 214 | -- Catch an error (including the break signal) here 215 | if rc == false then 216 | -- send the error back on the stdout stream and clear the display 217 | print(err) 218 | frame.display.text(' ', 1, 1) 219 | frame.display.show() 220 | break 221 | end 222 | end 223 | end 224 | 225 | -- run the main app loop 226 | app_loop() 227 | ``` 228 | 229 | | Selected Examples | Flutter | Lua | 230 | |-------------------|---------|-----| 231 | | Display a Sprite (`simple_frame_app` example) | [Flutter](https://github.com/CitizenOneX/frame_sprite_viewer/blob/main/lib/main.dart){:target="_blank"} | [Lua](https://github.com/CitizenOneX/frame_sprite_viewer/blob/main/assets/frame_app.lua){:target="_blank"} | 232 | 233 | --- 234 | 235 | ## Higher-level application scaffolding 236 | 237 | In addition to `frame-msg` and `frame-ble`, the Frame developer community has provided some even higher-level scaffolding for creating demo apps and to experiment with ideas quickly, but are not recommended for production apps: 238 | 239 | ### Simple Frame App 240 | | [![Available on pub.dev](https://img.shields.io/pub/v/simple_frame_app)](https://pub.dev/packages/simple_frame_app){:target="_blank"} | [API Reference](https://pub.dev/documentation/simple_frame_app/latest/){:target="_blank"} | [Source](https://github.com/CitizenOneX/simple_frame_app){:target="_blank"} | 241 | 242 | Quickstart scaffolding for a single-page mobile application that handles basic connection to Frame and loading of Frameside application resources including Lua files and sprites. 243 | 244 | | Selected Examples | Flutter | Lua | 245 | |-------------------|---------|-----| 246 | | [Display a Sprite](https://github.com/CitizenOneX/frame_sprite_viewer){:target="_blank"} | [Flutter](https://github.com/CitizenOneX/frame_sprite_viewer/blob/main/lib/main.dart){:target="_blank"} | [Lua](https://github.com/CitizenOneX/frame_sprite_viewer/blob/main/assets/frame_app.lua){:target="_blank"} | 247 | | [Wikipedia Viewer](https://github.com/CitizenOneX/wiki_frame){:target="_blank"} | [Flutter](https://github.com/CitizenOneX/wiki_frame/blob/main/lib/main.dart){:target="_blank"} | [Lua](https://github.com/CitizenOneX/wiki_frame/blob/main/assets/frame_app.lua){:target="_blank"} | 248 | | [Pollinations.AI Image Generation](https://github.com/CitizenOneX/frame_pollinations){:target="_blank"} | [Flutter](https://github.com/CitizenOneX/frame_pollinations/blob/main/lib/main.dart){:target="_blank"} | [Lua](https://github.com/CitizenOneX/frame_pollinations/blob/main/assets/frame_app.lua){:target="_blank"} | 249 | | [Audio Clip Recorder](https://github.com/CitizenOneX/frame_audio_clip){:target="_blank"} | [Flutter](https://github.com/CitizenOneX/frame_audio_clip/blob/main/lib/main.dart){:target="_blank"} | [Lua](https://github.com/CitizenOneX/frame_audio_clip/blob/main/assets/frame_app.lua){:target="_blank"} | 250 | | [Live Streaming Transcription](https://github.com/CitizenOneX/frame_transcribe_googlespeech){:target="_blank"} | [Flutter](https://github.com/CitizenOneX/frame_transcribe_googlespeech/blob/main/lib/main.dart){:target="_blank"} | [Lua](https://github.com/CitizenOneX/frame_transcribe_googlespeech/blob/main/assets/frame_app.lua){:target="_blank"} | 251 | 252 | ### Frame Vision App 253 | | [![Available on pub.dev](https://img.shields.io/pub/v/simple_frame_app)](https://pub.dev/packages/simple_frame_app){:target="_blank"} | [API Reference](https://pub.dev/documentation/simple_frame_app/latest/){:target="_blank"} | [Source](https://github.com/CitizenOneX/simple_frame_app){:target="_blank"} | 254 | 255 | Even more specialized quickstart scaffolding within the `simple_frame_app` package, designed for computer vision prototyping. In addition to handling connection and application resources on Frame, the Frame Vision App mixin conveniently provides images from the Frame camera directly to your code for custom processing. 256 | 257 | | Selected Examples | Flutter | Lua | 258 | |-------------------|---------|-----| 259 | | [Live Camera Feed](https://github.com/CitizenOneX/live_camera_feed){:target="_blank"} | [Flutter](https://github.com/CitizenOneX/live_camera_feed/blob/main/lib/main.dart){:target="_blank"} | [Lua](https://github.com/CitizenOneX/live_camera_feed/blob/main/assets/frame_app.lua){:target="_blank"} | 260 | | [Hotdog or not hotdog](https://github.com/CitizenOneX/frame_vision_hotdog){:target="_blank"} | [Flutter](https://github.com/CitizenOneX/frame_vision_hotdog/blob/main/lib/main.dart){:target="_blank"} | [Lua](https://github.com/CitizenOneX/frame_vision_hotdog/blob/main/assets/frame_app.lua){:target="_blank"} | 261 | | [Mobile Image Classification](https://github.com/CitizenOneX/frame_vision_classification){:target="_blank"} | [Flutter](https://github.com/CitizenOneX/frame_vision_classification/blob/main/lib/main.dart){:target="_blank"} | [Lua](https://github.com/CitizenOneX/frame_vision_classification/blob/main/assets/frame_app.lua){:target="_blank"} | 262 | | [Mobile Vision Translation](https://github.com/CitizenOneX/frame_vision_translation){:target="_blank"} | [Flutter](https://github.com/CitizenOneX/frame_vision_translation/blob/main/lib/main.dart){:target="_blank"} | [Lua](https://github.com/CitizenOneX/frame_vision_translation/blob/main/assets/frame_app.lua){:target="_blank"} | 263 | 264 | ### “Mobile app with no mobile coding” 265 | 266 | | [Mobile App](https://github.com/CitizenOneX/frame_vision_api){:target="_blank"} | [Sample API Backend](https://github.com/CitizenOneX/frame_vision_api_impl){:target="_blank"} 267 | 268 | Just want to code your computer vision backend in Python, but want a mobile app experience? Use this simple mobile app as-is (or customize) and point it to your own API endpoint for image processing, sending the result back to Frame for display. 269 | 270 | 271 | --- 272 | 273 | ## Legacy SDKs 274 | 275 | Note: The legacy SDK package is available for [frame-sdk](https://pub.dev/packages/frame_sdk) as [previously documented here](https://github.com/brilliantlabsAR/docs/blob/c33420dc55eb12b27f5a804b9a5a82247d471ae9/frame/building-apps-sdk.md). 276 | 277 | In order to allow richer experiences on Frame including image display and audio streaming, the decision was made to move to an SDK model based on asynchronous message-passing between a main application loop running on the host, and a main event loop running on Frame. This approach is used in both the [Python](https://pypi.org/project/frame-msg/) and [Flutter](https://pub.dev/packages/frame_msg) SDKs. 278 | 279 | While this requires a slight adjustment from the way the legacy SDKs were used, there are many examples provided (e.g. [Sprite Viewer](https://github.com/CitizenOneX/frame_sprite_viewer), [Unicode Text Display](https://github.com/CitizenOneX/frame_textspriteblock), [Teleprompter](https://github.com/CitizenOneX/frame_teleprompter), [Mobile Image Classifier](https://github.com/CitizenOneX/frame_vision_classification), [Realtime Multimodal Assistant](https://github.com/brilliantlabsAR/frame_realtime_gemini_voicevision)) and a [helpful community in Discord](https://discord.com/invite/vDS9X7gdwg) to help you build your project. 280 | -------------------------------------------------------------------------------- /frame/frame-sdk-python.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Python 3 | description: An overview of the Frame SDK for building Python apps that talk to Frame AR glasses. 4 | image: /images/frame/frame-splash.png 5 | nav_order: 1 6 | parent: Frame SDK 7 | grand_parent: Frame 8 | nav_enabled: true 9 | 10 | --- 11 | 12 | # Frame SDK for Python 13 | {: .no_toc } 14 | 15 | --- 16 | 17 | The Frame SDK for Python is available in a low-level library ([frame-ble](https://pypi.org/project/frame-ble/)) for Bluetooth LE connectivity using [Bleak](https://github.com/hbldh/bleak) with a limited Lua REPL, and an application-level library ([frame-msg](https://pypi.org/project/frame-msg/)) for passing rich objects between your host program and Frame, such as images, streamed audio, IMU data and rasterized text. 18 | 19 | --- 20 | 21 | 1. TOC 22 | {:toc} 23 | 24 | --- 25 | 26 | ## `frame-ble` package: low-level connectivity 27 | 28 | | [![Available on PyPI](https://img.shields.io/pypi/v/frame-ble)](https://pypi.org/project/frame-ble/){:target="_blank"} | [API Reference](https://frame-ble-python.readthedocs.io/){:target="_blank"} | [Source](https://github.com/CitizenOneX/frame_ble_python){:target="_blank"} | [Examples](https://github.com/CitizenOneX/frame_examples_python/tree/main/frame_ble){:target="_blank"} | 29 | 30 | The `FrameBle` class allows a host to connect to Frame, initiating pairing if required. A sequence of Lua strings can be sent to Frame for execution, optionally returning their results as strings. Transmitted and received strings must be shorter than the Bluetooth MTU (`FrameBle.max_lua_payload()`). 31 | 32 | [Frame-specific Lua APIs](frame-sdk-lua.md) can be called to draw on the display, sample the accelerometer, put Frame into firmware update mode, set the tap and user data callbacks, and more. 33 | 34 | In addition to Lua strings, `FrameBle` can also send control commands (break and reset), as well as custom user data messages that can be handled by a user-defined data handler function on Frame. See [Talking to Frame Over Bluetooth](frame-sdk-bluetooth-specs.md) for more details. 35 | 36 | Consult the [API Reference](https://frame-ble-python.readthedocs.io/){:target="_blank"} for the complete list of functions available in `frame-ble`, and the [Examples repo](https://github.com/CitizenOneX/frame_examples_python/tree/main/frame_ble){:target="_blank"} to see them in action. Don't see an example you need? Ask in [Discord](https://discord.com/invite/vDS9X7gdwg){:target="_blank"} to have one added. 37 | 38 | Note: It is expected that most application developers will prefer to use `FrameMsg` from the [frame-msg package](https://pypi.org/project/frame-msg/) to make use of first-class messages and efficient transport for sprites, images, audio clips, and to avoid having to wrangle Bluetooth packet fragmentation and assembly. The `frame-ble` package, with only a dependency on the `bleak` library, is still a handy choice for `"Hello, World!"`, simple utilities, or as the basis of a custom application framework in Python. 39 | 40 | 41 | 42 | ### Example usage 43 | {: .no_toc } 44 | ```bash 45 | pip install frame-ble 46 | ``` 47 | 48 | ```python 49 | import asyncio 50 | from frame_ble import FrameBle 51 | 52 | async def main(): 53 | frame = FrameBle() 54 | 55 | try: 56 | await frame.connect() 57 | 58 | await frame.send_lua("frame.display.text('Hello, Frame!', 1, 1);frame.display.show();print(nil)", await_print=True) 59 | 60 | await frame.disconnect() 61 | 62 | except Exception as e: 63 | print(f"Not connected to Frame: {e}") 64 | return 65 | 66 | if __name__ == "__main__": 67 | asyncio.run(main()) 68 | ``` 69 | 70 | | Selected Examples | 71 | |-------------------| 72 | | ["Hello, World!"](https://github.com/CitizenOneX/frame_examples_python/blob/main/frame_ble/hello_world.py){:target="_blank"} | 73 | | [Clear the display](https://github.com/CitizenOneX/frame_examples_python/blob/main/frame_ble/clear_display.py){:target="_blank"} | 74 | | [Echo printed outputs from Lua code executing on Frame](https://github.com/CitizenOneX/frame_examples_python/blob/main/frame_ble/echo.py){:target="_blank"} | 75 | | [Upload and invoke a custom Lua function from a file](https://github.com/CitizenOneX/frame_examples_python/blob/main/frame_ble/custom_lua_functions.py){:target="_blank"} | 76 | | [Upload and invoke a function demonstrating lz4 decompression on Frame](https://github.com/CitizenOneX/frame_examples_python/blob/main/frame_ble/decompression.py){:target="_blank"} | 77 | 78 | --- 79 | 80 | ## `frame-msg` package: application-level messaging 81 | 82 | | [![Available on PyPI](https://img.shields.io/pypi/v/frame-msg)](https://pypi.org/project/frame-msg/){:target="_blank"} | [API Reference](https://frame-msg-python.readthedocs.io/){:target="_blank"} | [Source](https://github.com/CitizenOneX/frame_msg_python){:target="_blank"} | [Examples](https://github.com/CitizenOneX/frame_examples_python/tree/main/frame_msg){:target="_blank"} | 83 | 84 | `frame-msg` enables both the hostside and Frameside halves of an application to **communicate using richly typed messages** - for example transmitting Sprites to Frame with width, height, palette data and pixel data attributes. (See the code below for an example.) 85 | 86 | Additionally, `frame-msg` uses a programming model in which **a main event loop is running on Frame**. As a result, messages are exchanged asychronously and handled in an event-driven way. This is in contrast to the "Lua REPL" model, in which single statements are sent to Frame to be executed, and the host attempts to block until execution has completed. In a REPL-style model, it would not be practical to create an application that for example continuously streams audio from Frame to host, and simultaneously sends text back for display (a live translation app). In contrast, with a main event loop running on Frame it is possible to send some audio back to the host if some is available, and check for messages indicating text is ready for display, then loop around again. 87 | 88 | 89 | 90 | ### How it works 91 | {: .no_toc } 92 | 93 | In order to **serialize and deserialize large objects**, `frame-msg` makes use of `FrameBle`'s `send_message()` function that **fragments large payloads** into multiple Bluetooth packets marked with a matching **`msg_code`**, and **efficiently reassembles them on Frame**. After parsing the reconstructed payload into the desired type, a complete object is made available for the application's main loop in Lua. 94 | 95 | Handling **rich message types** is done by providing symmetrical **packing code and parsing code** for standard types. 96 | 97 | * For "Transmit" (`Tx`) message types, the packing (serialization) code is in the Python type (e.g. `TxSprite.pack()`) and the parsing (deserialization) code is in the corresponding Lua module (`sprite.lua` / `parse_sprite()`). 98 | 99 | * For "Receive" (`Rx`) message types, e.g. `RxPhoto`, the serialization code is present implicitly in the Lua module `camera.lua` in `capture_and_send()`, and the deserialization code is present in the `RxPhoto` class. 100 | 101 | This approach presents a natural extension point for user-defined types too: a custom application-specific message might contain an icon, a heading and some detailed text - and this can all be sent together and used all at once on Frame to update the display. By implementing the packing and parsing code for user-defined types, custom objects can be used in applications in the same way as the standard types. 102 | 103 | ### Startup sequence 104 | {: .no_toc } 105 | 106 | `frame-msg` applications tend to follow a startup sequence that ensures Frame is in a known state for the application to run, namely: 107 | * Optionally send a break/reset/break signal sequence to ensure that Frame is in Lua REPL mode and its memory is freshly cleared 108 | * Upload standard Lua libraries to Frame: `data` for handling all messaging, plus any others to support specific messages the application uses (e.g. `sprite`, `plain_text`, etc.) Unless specified otherwise, [minified Lua code files](https://en.wikipedia.org/wiki/Minification_(programming)) are uploaded, so make sure your Frameside app includes `sprite.min` in this case. 109 | * Upload the main Frame application Lua file (e.g. `my_frame_app.lua`). This can often be an exact copy of an example main Lua file that handles the same message types. 110 | * Start the main application loop on Frame (after which no Lua REPL commands will be accepted by Frame). 111 | 112 | After the Frameside application reports that it has started, the hostside and Frameside application halves begin messaging each other asynchronously: passing text, images, audio etc. 113 | 114 | 115 | 116 | ## Example usage 117 | {: .no_toc } 118 | ```bash 119 | pip install frame-msg 120 | ``` 121 | 122 | `sprite_display.py` 123 | ```python 124 | import asyncio 125 | from pathlib import Path 126 | 127 | from frame_msg import FrameMsg, TxSprite 128 | 129 | async def main(): 130 | """ 131 | Displays sample images on the Frame display. 132 | 133 | The images are indexed (palette) PNG images, in 2, 4, 134 | and 16 colors (that is, 1-, 2- and 4-bits-per-pixel). 135 | """ 136 | frame = FrameMsg() 137 | try: 138 | await frame.connect() 139 | 140 | # Let the user know we're starting 141 | await frame.print_short_text('Loading...') 142 | 143 | # send the std lua files to Frame that handle data accumulation and sprite parsing 144 | await frame.upload_stdlua_libs(lib_names=['data', 'sprite']) 145 | 146 | # Send the main lua application from this project to Frame that will run the app 147 | await frame.upload_frame_app(local_filename="lua/sprite_frame_app.lua") 148 | 149 | # attach the print response handler so we can see stdout from Frame Lua print() statements 150 | frame.attach_print_response_handler() 151 | 152 | # "require" the main frame_app lua file to run it, and block until it has started. 153 | await frame.start_frame_app() 154 | 155 | # send the 1-bit image to Frame in chunks 156 | sprite = TxSprite.from_indexed_png_bytes(Path("images/logo_1bit.png").read_bytes()) 157 | await frame.send_message(0x20, sprite.pack()) 158 | 159 | # send a 2-bit image 160 | sprite = TxSprite.from_indexed_png_bytes(Path("images/street_2bit.png").read_bytes()) 161 | await frame.send_message(0x20, sprite.pack()) 162 | 163 | # send a 4-bit image 164 | sprite = TxSprite.from_indexed_png_bytes(Path("images/hotdog_4bit.png").read_bytes()) 165 | await frame.send_message(0x20, sprite.pack()) 166 | 167 | await asyncio.sleep(5.0) 168 | 169 | # unhook the print handler 170 | frame.detach_print_response_handler() 171 | 172 | # break out of the frame app loop and reboot Frame 173 | await frame.stop_frame_app() 174 | 175 | except Exception as e: 176 | print(f"An error occurred: {e}") 177 | finally: 178 | # clean disconnection 179 | await frame.disconnect() 180 | 181 | if __name__ == "__main__": 182 | asyncio.run(main()) 183 | ``` 184 | 185 | `sprite_frame_app.lua` 186 | ```lua 187 | local data = require('data.min') 188 | local sprite = require('sprite.min') 189 | 190 | -- Phone to Frame flags 191 | USER_SPRITE = 0x20 192 | 193 | -- register the message parsers so they are automatically called when matching data comes in 194 | data.parsers[USER_SPRITE] = sprite.parse_sprite 195 | 196 | -- Main app loop 197 | function app_loop() 198 | frame.display.text('Frame App Started', 1, 1) 199 | frame.display.show() 200 | 201 | -- tell the host program that the frameside app is ready (waiting on await_print) 202 | print('Frame app is running') 203 | 204 | while true do 205 | rc, err = pcall( 206 | function() 207 | -- process any raw data items, if ready 208 | local items_ready = data.process_raw_items() 209 | 210 | -- one or more full messages received 211 | if items_ready > 0 then 212 | 213 | if data.app_data[USER_SPRITE] ~= nil then 214 | local spr = data.app_data[USER_SPRITE] 215 | 216 | -- set the palette in case it's different to the standard palette 217 | sprite.set_palette(spr.num_colors, spr.palette_data) 218 | 219 | -- show the sprite 220 | frame.display.bitmap(1, 1, spr.width, 2^spr.bpp, 0, spr.pixel_data) 221 | frame.display.show() 222 | 223 | -- clear the object and run the garbage collector right away 224 | data.app_data[USER_SPRITE] = nil 225 | collectgarbage('collect') 226 | end 227 | 228 | end 229 | 230 | -- can't sleep for long, might be lots of incoming bluetooth data to process 231 | frame.sleep(0.001) 232 | end 233 | ) 234 | -- Catch an error (including the break signal) here 235 | if rc == false then 236 | -- send the error back on the stdout stream and clear the display 237 | print(err) 238 | frame.display.text(' ', 1, 1) 239 | frame.display.show() 240 | break 241 | end 242 | end 243 | end 244 | 245 | -- run the main app loop 246 | app_loop() 247 | ``` 248 | 249 | | Selected Examples | Python | Lua | 250 | |-------------------|--------|-----| 251 | | Display a Sprite | [Python](https://github.com/CitizenOneX/frame_examples_python/blob/main/frame_msg/sprite_ind_png.py){:target="_blank"} | [Lua](https://github.com/CitizenOneX/frame_examples_python/blob/main/frame_msg/lua/sprite_frame_app.lua){:target="_blank"} | 252 | | Display Unicode Text | [Python](https://github.com/CitizenOneX/frame_examples_python/blob/main/frame_msg/text_sprite_block.py){:target="_blank"} | [Lua](https://github.com/CitizenOneX/frame_examples_python/blob/main/frame_msg/lua/text_sprite_block_frame_app.lua){:target="_blank"} | 253 | | Take a photo (auto exposure) | [Python](https://github.com/CitizenOneX/frame_examples_python/blob/main/frame_msg/camera.py){:target="_blank"} | [Lua](https://github.com/CitizenOneX/frame_examples_python/blob/main/frame_msg/lua/camera_frame_app.lua){:target="_blank"} | 254 | | Take a photo (manual exposure) | [Python](https://github.com/CitizenOneX/frame_examples_python/blob/main/frame_msg/manual_exposure.py){:target="_blank"} | [Lua](https://github.com/CitizenOneX/frame_examples_python/blob/main/frame_msg/lua/manualexp_frame_app.lua){:target="_blank"} | 255 | | Live camera feed | [Python](https://github.com/CitizenOneX/frame_examples_python/blob/main/frame_msg/live-camera-feed-with-params.py){:target="_blank"} | [Lua](https://github.com/CitizenOneX/frame_examples_python/blob/main/frame_msg/lua/live_camera_frame_app.lua){:target="_blank"} | 256 | | Stream audio | [Python](https://github.com/CitizenOneX/frame_examples_python/blob/main/frame_msg/audio_stream.py){:target="_blank"} | [Lua](https://github.com/CitizenOneX/frame_examples_python/blob/main/frame_msg/lua/audio_frame_app.lua){:target="_blank"} | 257 | 258 | --- 259 | 260 | ## Legacy SDKs 261 | 262 | Note: Legacy SDK packages are available for [frame-sdk](https://pypi.org/project/frame-sdk/) and [frameutils](https://pypi.org/project/frame-utilities-for-python/) as [previously documented here](https://github.com/brilliantlabsAR/docs/blob/c33420dc55eb12b27f5a804b9a5a82247d471ae9/frame/building-apps-sdk.md). 263 | 264 | In order to allow richer experiences on Frame including image display and audio streaming, the decision was made to move to an SDK model based on asynchronous message-passing between a main application loop running on the host, and a main event loop running on Frame. This approach is used in both the [Python](https://pypi.org/project/frame-msg/) and [Flutter](https://pub.dev/packages/frame_msg) SDKs. 265 | 266 | While this requires a slight adjustment from the way the legacy SDKs were used, there are [many examples provided](https://github.com/CitizenOneX/frame_python_examples) and a [helpful community in Discord](https://discord.com/invite/vDS9X7gdwg) to help you build your project. 267 | -------------------------------------------------------------------------------- /frame/frame-sdk.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Frame SDK 3 | description: A guide on how to develop your own applications for Frame. 4 | image: /images/frame/frame-splash.png 5 | nav_order: 2 6 | parent: Frame 7 | has_children: true 8 | redirect_from: 9 | - /frame/building-apps 10 | - /frame/building-apps-sdk 11 | - /frame/building-apps-lua 12 | - /frame/building-apps-bluetooth-specs 13 | - /frame/sdk 14 | --- 15 | 16 | # Building Apps with the Frame SDK 17 | {: .no_toc } 18 | 19 | --- 20 | 21 | 1. TOC 22 | {:toc} 23 | 24 | --- 25 | 26 | ## How Do Apps Run on Frame? 27 | 28 | Frame runs a power-efficient System-on-a-Chip (SoC) with relatively limited memory and processing power compared to modern smartphones. Rather than installing apps directly on Frame, it typically functions as a peripheral accessory for "host" apps running on computers or mobile devices. These host apps communicate with Frame via Bluetooth to control features like the camera, microphone, and display. 29 | 30 | While you may write Lua scripts that execute on Frame for specific behaviors, your host app primarily drives the logic while Frame runs a simple event handler loop. For example, the Noa mobile app connects to Frame, receives sensor data over Bluetooth, processes it, and sends content back to Frame for display. 31 | 32 | Frame doesn't have its own app launcher or traditional app installation system. To share your Frame apps with others, publish your host app through normal distribution channels like App Store, Google Play, or open source repositories. Of course if you build something cool, let us know and we will be happy to share it with the community. 33 | 34 | ## Getting Started 35 | 36 | For the quickest start: 37 | 38 | * If you're using a supported platform, begin with our SDK documentation for [Python](/frame/frame-sdk-python) or [Flutter](/frame/frame-sdk-python), then explore the [Lua API](/frame/frame-sdk-lua) for advanced features. 39 | * For other platforms, use the [Lua API](/frame/frame-sdk-lua) directly over Bluetooth with the [Bluetooth LE API documentation](/frame/frame-sdk-bluetooth-specs). 40 | * Note: Legacy SDKs ([frame-sdk-python](https://pypi.org/project/frame-sdk/), [frame-sdk-flutter](https://pub.dev/packages/frame_sdk), [frameutils-python](https://pypi.org/project/frameutils/)) remain functional, but new projects should use the updated SDK to access new features like image display and realtime streaming. 41 | 42 | # Development Options for Frame 43 | 44 | 1. Use Platform SDKs **(Recommended)** 45 | * Build with supported platforms: 46 | * [Python](/frame/frame-sdk-python) (Mac, Linux, Windows) 47 | * [Flutter](/frame/frame-sdk-flutter) (iOS/Android) 48 | * Benefits: Common tasks are simplified, many examples to follow 49 | 2. Communicate via Bluetooth LE 50 | * Use any platform with Bluetooth LE capabilities 51 | * Offers full control but requires Bluetooth LE development experience 52 | * See [Bluetooth LE API documentation](/frame/frame-sdk-bluetooth-specs) 53 | 3. Customize Firmware **(Advanced)** 54 | * Requires hardware development expertise 55 | * Provides complete control of Frame hardware 56 | * See [firmware customization and hardware documentation](/frame/hardware#customizing-the-firmware) 57 | 58 | The typical development pattern involves creating a Python desktop application or Flutter mobile app that uses the Frame SDK to communicate with Frame over Bluetooth, abstracting away many Lua and BTLE details. 59 | 60 | ## Platform SDK Development 61 | 62 | {: .note } 63 | This recommended approach offers quick and reliable Frame app development for Python and Flutter platforms, with each of these platform SDKs under active development. 64 | 65 | Our platform SDKs simplify the low-level Bluetooth and Lua APIs, eliminating the need to write boilerplate code for device discovery, connection, and communication. 66 | 67 | Familiarity with the [Lua API](/frame/frame-sdk-lua) remains useful for main application event loops, `main.lua` scripts that run at wakeup, or custom on-device processing. 68 | 69 | Check the [Python](/frame/frame-sdk-python) or [Flutter](/frame/frame-sdk-flutter) SDK documentation for implementation details. 70 | 71 | 72 | ## Direct Bluetooth LE Development 73 | 74 | For developers comfortable with Bluetooth LE, this approach offers maximum flexibility but requires a greater understanding of Bluetooth technology. 75 | 76 | After establishing BTLE communication, you'll use Lua to control Frame's functions. Frame provides a Lua virtual machine where you can execute scripts or use the Lua REPL over the BTLE interface. 77 | 78 | Refer to the [Bluetooth LE API documentation](/frame/frame-sdk-bluetooth-specs) and the [full Lua API reference](/frame/frame-sdk-lua) for details on Frame's capabilities. 79 | 80 | During development, you can also use the [AR Studio extension for VSCode](/frame/frame-sdk-lua#ar-studio) to directly write and execute Lua code on Frame. 81 | 82 | ## Firmware Customization 83 | 84 | Frame is developed by hackers for hackers, so even though we don't recommend it for most users, Frame's open-source firmware can be customized. Find the official firmware [on GitHub](https://github.com/brilliantlabsAR/frame-codebase). 85 | 86 | For more details on how to customize the firmware, see the [hardware documentation](/frame/hardware#customizing-the-firmware). -------------------------------------------------------------------------------- /frame/frame.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Frame 3 | description: Technical documentation for the Brilliant Frame. 4 | image: /images/frame/frame-splash.png 5 | nav_order: 2 6 | has_children: true 7 | --- 8 | 9 | ![Brilliant Frame](/frame/images/frame-splash.png) 10 | 11 | # Getting started with Frame 12 | 13 | Whether you're using Frame with Noa, or making your own apps, you'll find everything you need to know about your Frame hardware here. 14 | 15 | ## How do I charge my Frame? 16 | 17 | Your Frame will automatically power off, and start charging whenever it is placed onto the charging cradle (aka Mister Power). 18 | 19 | The charging cradle provides an extra helping of battery power for when you're on the go. To charge both Frame and the charging cradle, use any USB Type-C power supply. 20 | 21 | The cradle and Frame when docked together will take around 2 hours to fully charge. The white LED on the cradle will turn off to indicate that charging is complete. 22 | 23 | ![Charging Frame](/frame/images/frame-charging-diagram.png) 24 | 25 | ## How do I set the bridge height? 26 | 27 | Choosing the correct nose bridge adaptor is key to seeing the full display area. You can choose between 3mm, 2mm or no nose bridge adaptor to set the correct level. 28 | 29 | ![Frame nose bridge height](/frame/images/frame-nose-bridge-diagram.png) 30 | 31 | ## How do I connect to my Frame for the first time? 32 | 33 | Ensure your Frame is charged, and then with the charging cradle connected, use a sim card tool and hold the pairing button for 3 seconds. Then remove the dock, and from any Frame compatible app, such as Noa, press connect, and then tap pair. 34 | 35 | ![Pairing Frame to Noa](/frame/images/frame-pairing-diagram.png) 36 | 37 | ## How do I update my Frame? 38 | 39 | Whenever software updates are available for your Frame, you'll be notified in your Frame compatible apps. Ensure Frame is well charged before updating to prevent partial updates and having to start again. Be sure to keep the app open and within close range of your Frame during the entire update. 40 | 41 | ## Where can I download Noa? 42 | 43 | Noa is available on both Apple App Store for iOS and Google Play Store for Android. You can also download the [Android APK directly from our Github](https://github.com/brilliantlabsAR/noa-flutter/releases/latest). 44 | 45 |
46 | [Apple App Store badge](https://apps.apple.com/us/app/noa-for-frame/id6482980023) 47 |         48 | [Google Play Store badge](https://play.google.com/store/apps/details?id=xyz.brilliant.noaflutter) 49 |
50 | 51 |
52 | **Check out the getting started with Noa playlist [here](https://www.youtube.com/watch?v=olNhbZTx6Lw&list=PLfbaC5GRVJJgSPdN-KWndTld35tihu1Ic)** 53 |
54 | 55 | ## Developer? 56 | 57 | Get started making your own apps today. Follow the links below to familiarize yourself with the Frame hardware and Lua API. -------------------------------------------------------------------------------- /frame/frame.stl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/frame.stl -------------------------------------------------------------------------------- /frame/hardware.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Hardware 3 | description: A brief overview of how Frame works under the hood. 4 | image: /images/frame/frame-splash.png 5 | nav_order: 1 6 | parent: Frame 7 | --- 8 | 9 | # Frame Hardware Manual 10 | {: .no_toc } 11 | 12 | --- 13 | 14 | ![Frame exploded view](/frame/images/frame-exploded-view.png) 15 | 16 | Here you'll find all the details about how Frame works under the hood. For software related information, be sure to also check out the [Frame SDK](/frame/frame-sdk) including [Lua API](/frame/frame-sdk-lua/) pages. 17 | 18 | ## Key features 19 | {: .no_toc } 20 | - 640x400 color OLED display 21 | - 20° FOV optic 22 | - Optional personalized prescriptions 23 | - Thin and light 6mm lenses (not including optional prescription) 24 | - 720p low power color camera 25 | - Microphone 26 | - FPGA acceleration for graphics and imaging 27 | - Bluetooth 5.3 28 | - 210mAh built-in rechargeable li-ion battery 29 | - 3 axis accelerometer with tap detection 30 | - 3 axis e-compass 31 | - Full featured Lua based OS 32 | - Charging dock with USB Type-C & 140mAh battery 33 | 34 | ## Example uses: 35 | {: .no_toc } 36 | - Generative AI on the go 37 | - ML based image augmentation 38 | - Computer vision research 39 | - QR code & barcode detection 40 | - Heads up telemetry 41 | - AR app & game design 42 | 43 | ## Contents 44 | {: .no_toc } 45 | 46 | 1. TOC 47 | {:toc} 48 | 49 | ## Hardware block diagram 50 | 51 | This diagram shows a general overview of the Frame internal hardware architecture. 52 | 53 | ![Frame block diagram](/frame/images/frame-hardware-block-diagram.drawio.svg) 54 | 55 | --- 56 | 57 | ## Bluetooth MCU 58 | 59 | the Bluetooth MCU serves as the main processor for Frame. It handles all network communication and running of user logic. The MCU used is the [nRF52840](https://www.nordicsemi.com/products/nrf52840) from Nordic. It contains a 32bit ARM Cortex-M4F CPU running at 64MHz, 1 MB of flash storage, 256 KB of RAM, and features Bluetooth 5.3 connectivity. 60 | 61 | As standard, Frame comes with a feature rich Lua based OS that allows for scripting and remote access fully over Bluetooth. No programming cables, or proprietary software is needed to use the device. 62 | 63 | The firmware is fully updatable over the air. If you're using a Frame compatible app, these updates should be automatic, however if you're developing your own apps, take a look at the [Frame SDK Bluetooth Specs page](/frame/frame-sdk-bluetooth-specs/#firmware-updates) to see how you can implement firmware update capability into your own apps. 64 | 65 | ### Customizing the firmware 66 | 67 | Frame is intended to be used with the officially provided firmware from Brilliant, however it is possible to customize the firmware if desired. Note that this will require destructive disassembly of your Frame hardware to access the physical debug port. 68 | 69 | If you wish to view the source code, or use it as a starting point for your custom firmware, visit the [Frame codebase repository](https://github.com/brilliantlabsAR/frame-codebase). 70 | 71 | The physical debug port of the Bluetooth MCU is located on the back of the Frame PCB. A total of five wires can be connected to a suitable ARM SWD debugger such as a [J-Link probe](https://www.segger.com/products/debug-probes/j-link/) or [Black-Magic probe](https://black-magic.org) to allow for debugging. 72 | 73 | ![Serial wire debug (SWD) of the Frame PCB](/frame/images/frame-debug-pinout.png) 74 | 75 | --- 76 | 77 | ## FPGA 78 | 79 | The FPGA is used for graphics acceleration of the display, as well as interfacing to the 720p camera sensor. It communicates to the Bluetooth MCU via SPI and is dynamically shutdown off to save power. 80 | 81 | The FPGA used is the [Crosslink-NX LIFCL-17](https://www.latticesemi.com/Products/FPGAandCPLD/CrossLink-NX) from Lattice. It features 17k logic cells, 432kb of embedded block RAM, and 2.56Mb of large RAM. 82 | 83 | If you wish to view the source code, or use it as a starting point for a custom RTL design, visit the [FPGA section](https://github.com/brilliantlabsAR/frame-codebase/tree/main/source/fpga) of the Frame codebase repository. 84 | 85 | The FPGA does not expose a physical programming interface. Therefore the FPGA application is uploaded solely via SPI at boot-up. 86 | 87 | --- 88 | 89 | ## Display 90 | 91 | The display used in Frame is a 0.23" micro OLED. It features 640x400 RGB pixels and is optically bonded to the prism assembly which directs the image into the user's eye. The result is a transparent floating display with a 20° field of view, and appears about the size of a tablet display at arms length. 92 | 93 | The display is connected to the FPGA in 8bit YCbCr mode. 4 wires for the Y channel, and 2 wires each for the Cb and Cr channels. This allows for a total of 255 possible colors shown on the display. In turn, to save memory and ensure fast frame rates, the FPGA RTL is optimized to display a maximum of 16 colors per frame. These colors can be changed on the fly, allowing for large sprite sets and fonts to be efficiently stored within Frame's embedded memory. To better understand how the graphics subsystem of Frame works, check out the [graphics section](https://github.com/brilliantlabsAR/frame-codebase/blob/main/docs/fpga-architecture.md#graphics) of the Frame codebase repository. 94 | 95 | ![Optical design and pathway of the display in Frame](/frame/images/frame-display.png) 96 | 97 | --- 98 | 99 | ## Camera 100 | 101 | The front facing camera sensor on Frame is the incredibly small and power efficient [OV09734](https://www.ovt.com/products/ov09734-h16a-2a/) from Omnivision. 102 | 103 | The FPGA RTL optimizes images for AI applications such as image recognition and allows the user a large amount of control over resolution, gain and exposure for maximum flexibility. 104 | 105 | Images are captured as 1280x720 RGB, but then cropped to 720x720 and converted to YCbCr for memory efficiency and fast downloading over Bluetooth. To better understand how the camera subsystem of Frame works, check out the [camera section](https://github.com/brilliantlabsAR/frame-codebase/blob/main/docs/fpga-architecture.md#camera) of the Frame codebase repository. 106 | 107 | ![Position of the front facing camera on Frame](/frame/images/frame-camera.png) 108 | 109 | --- 110 | 111 | ## Microphone 112 | 113 | Frame features a single [ICS-41351](https://product.tdk.com/system/files/dam/doc/product/sw_piezo/mic/mems-mic/data_sheet/ds-000157-ics-41351-v1.4.pdf) MEMS microphone from TDK. It has a wide dynamic range from -35.5dB to 129.5dB, allowing it to hear everything from soft speech, to loud booming noises. 114 | 115 | The microphone is connected directly to the Bluetooth MCU which allows for low power operation and applications such as periodic recording and wake detection. The standard firmware allows for a wide range of recording formats from 4bits to 16bits sample depth, and 4kHz to 20kHz sample rate. 116 | 117 | ![Position of the microphone on Frame](/frame/images/frame-microphone.png) 118 | 119 | --- 120 | 121 | ## Motion Sensor (IMU) 122 | 123 | The IMU on Frame is the 6 axis [MC6470](https://eu.mouser.com/datasheet/2/821/MC6470_Datasheet_APS_048_0033v1_7_1-3003085.pdf) sensor from MEMSIC. It features both an accelerometer and e-compass in a tiny package consuming very little power. Further processing on the Bluetooth MCU calculates the raw X, Y and Z values of both sensing elements into angular values for detecting head position. 124 | 125 | An accelerator interrupt line from the IMU is also directly connected to the Bluetooth MCU. This allows for always on detections of taps that can be used to navigate and select UI elements within the user's apps. 126 | 127 | ![Motion sensing capabilities of Frame](/frame/images/frame-imu.png) 128 | 129 | --- 130 | 131 | ## Power 132 | 133 | Power is internally distributed within Frame via the [MAX77654](https://www.analog.com/media/en/technical-documentation/data-sheets/max77654.pdf) power management IC from Analog Devices. Each rail is carefully managed and monitored to both protect the components within Frame, as well as ensure lasting performance of the battery. 134 | 135 | {: .warning } 136 | > The PMIC is configured from C code baked into the Bluetooth MCU firmware. If you're creating a custom firmware, it's recommended to avoid changing any of the PMIC settings without carefully studying the schematics and PMIC datasheet. These settings can easily over-volt components damaging them, as well as damage the battery. 137 | 138 | ### Battery charging 139 | 140 | The PMIC includes an integrated battery charger for the two built in 105mAh li-ion cells. Regulation is based on time, temperature and the current operating state of Frame. An analog pin provides the ability to monitor battery voltage and current within the firmware, and can be used to estimate battery life. 141 | 142 | --- 143 | 144 | ## Charging cradle (Mister Power) 145 | 146 | The charging cradle functions to both charge Frame via the 5V terminal on the back of the glasses, as well as allowing for factory resetting and un-pairing of Frame from any connected device. 147 | 148 | It also contains a 140mAh rechargeable li-ion cell which allows for a top-up of Frame's internal battery while on the go. The charging cradle itself, and in turn Frame, can be charged using any USB Type-C power supply. 149 | 150 | ![Frame charging cradle aka Mister Power](/frame/images/frame-charging-cradle.png) 151 | 152 | --- 153 | 154 | ## Schematics 155 | 156 | ### Frame 157 | 158 |
159 | 160 | 161 | 162 |
163 | 164 | ### Charging cradle 165 | 166 |
167 | 168 | 169 | 170 |
171 | 172 | --- 173 | 174 | ## Mechanical 175 | 176 | **Details coming soon** 177 | 178 | ### Frame 179 | 180 | [Download the 3D Model in STL format](/frame/frame.stl) 181 | 182 |
183 | 184 | 185 | 186 |
187 | 188 |
189 | 190 | 191 | 192 |
193 | 194 | ### Charging cradle 195 | 196 | [Download the 3D Model in STL format](/frame/charging-cradle.stl) 197 | 198 | 203 | 204 | ### Prescription (Rx) clip 205 | 206 | [Download the 3D Model in STL format](/frame/frame-rx-clip.stl) 207 | 208 | 209 | --- 210 | 211 | ## Device characteristics 212 | 213 | Typical and absolute device characteristics are shown below. To get the best lifetime of Frame, it's recommended to keep within these limits. 214 | 215 | ### Typical characteristics 216 | 217 | | | Min | Typ | Max | 218 | |:------------------------|:------:|:-----:|:------:| 219 | | Frame operating current | 45mA | 80mA | 100mA | 220 | | Frame sleep current | - | 580uA | - | 221 | | Frame shutdown current | - | 132uA | - | 222 | | Frame charging current | 1.5mA | - | 225mA | 223 | | Charging cradle current | - | - | 400mA | 224 | | Bluetooth radio power | -20dBm | - | 8dBm | 225 | | Bluetooth sensitivity | - | - | -95dBm | 226 | 227 | ### Maximum ratings 228 | 229 | | | Min | Typ | Max | 230 | |:----------------------------|:-----:|:----:|:----:| 231 | | Frame charging voltage | -0.3V | 5.1V | 6.4V | 232 | | Charging cradle USB voltage | -0.3V | 5.1V | 7V | 233 | | Operating temperature | 0°C | - | 45°C | 234 | | Storage temperature | -20°C | - | 60°C | 235 | 236 | --- 237 | 238 | ## Safety & limitation of liability 239 | 240 | ### Safety 241 | 242 | Brilliant Labs' devices can obscure your vision and should not be used while driving or operating dangerous equipment. Additionally long periods of use may cause eye strain, headaches and motion sickness. Brilliant Labs' devices can also display bright flashing images so may not be suitable for those who are susceptible to light sensitivity. 243 | 244 | ### Critical applications 245 | 246 | Brilliant Labs' devices are intended for consumer and R&D applications. It is not verified for use where performance and accuracy would be critical to human health, safety or mission critical use. 247 | 248 | ### Lithium batteries 249 | 250 | Lithium batteries can be dangerous if mishandled. Do not expose Brilliant Labs' devices to excess temperatures, fire or liquids. Do not try to remove the battery as the terminals can become shorted and result in the battery overheating or catching fire. Once the product reaches the end of it's life, dispose it safely according to your local regulations, such as e-waste collection points where any volatile components can be properly contained and handled. 251 | 252 | ### FCC notice 253 | 254 | This device complies with part 15 of the FCC Rules. Operation is subject to the following two conditions: (1) This device may not cause harmful interference, and (2) this device must accept any interference received, including interference that may cause undesired operation. 255 | 256 | Any Changes or modifications not expressly approved by the party responsible for compliance could void the user's authority to operate the equipment. Note: This equipment has been tested and found to comply with the limits for a Class B digital device, pursuant to part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference in a residential installation. This equipment generates uses and can radiate radio frequency energy and, if not installed and used in accordance with the instructions, may cause harmful interference to radio communications. However, there is no guarantee that interference will not occur in a particular installation. If this equipment does cause harmful interference to radio or television reception, which can be determined by turning the equipment off and on, the user is encouraged to try to correct the interference by one or more of the following measures: 257 | 258 | - Reorient or relocate the receiving antenna. 259 | - Increase the separation between the equipment and receiver. 260 | - Connect the equipment into an outlet on a circuit different from that to which the receiver is connected. 261 | - Consult the dealer or an experienced radio/TV technician for help. 262 | 263 | The device has been evaluated to meet general RF exposure requirement. The device can be used in portable exposure condition without restriction. 264 | 265 | ### Limitation of liability 266 | 267 | Brilliant Labs provides technical data, including design resources, examples, applications, design advice, tools, safety information and other resources "as is" and disclaims all warranties, express and implied, including without limitation any implied warranties or merchantability, fitness for a particular purpose or non-infringement of third party intellectual property rights. 268 | 269 | These resources are intended for skilled developers. You are solely responsible for selecting the appropriate products for your application, designing, validating and testing your application, and ensuring your application meets applicable standards, and other safety, security, regulatory or other requirements. 270 | 271 | Brilliant Labs reserves the right to change the circuitry and specifications without notice at any time. The parametric values quoted in this manual are provided for guidance only. 272 | 273 | The resources and products are provided subject to our [terms and conditions](https://brilliant.xyz/pages/terms-conditions). 274 | -------------------------------------------------------------------------------- /frame/images/frame-bluetooth-sending-bytes-diagram.drawio.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 |
Host
Host
Frame
Frame
0x01 0x34 0x54 0x5E 0x7A
0x01 0x34 0x54 0x5E 0x7A
Sending bytes. E.g. 0x34 0x54 0x5E 0x7A
Sending bytes. E.g. 0x34 0x54 0x5E 0x7A
Receiving bytes
Receiving bytes
run callback: data = 0x34 0x54 0x5E 0x7A
run callback: data = 0x34 0x54 0x5E 0x7A
0x01 0x34 0x54 0x5E 0x7A
0x01 0x34 0x54 0x5E 0x7A
frame.bluetooth.send(0x34 0x54 0x5E 0x7A)
frame.bluetooth.send(0x34 0x54 0x5E 0x7A)Text is not SVG - cannot display -------------------------------------------------------------------------------- /frame/images/frame-bluetooth-sending-lua-diagram.drawio.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 |
Host
Host
Frame
Frame
print("hello world")
print("hello world")
"hello world"
"hello world"
f = frame.file.open("main.lua", "w")
f = frame.file.open("main.lua", "w")
responses on RX as per script
responses on RX as per script
f.write(...)
f.write(...)
f.write(...)
f.write(...)
f.close()
f.close()
require(main)
require(main)
Execute single lines of Lua
Execute single lines of Lua
Running larger scripts
Running larger scriptsText is not SVG - cannot display -------------------------------------------------------------------------------- /frame/images/frame-camera.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/images/frame-camera.png -------------------------------------------------------------------------------- /frame/images/frame-charging-cradle.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/images/frame-charging-cradle.png -------------------------------------------------------------------------------- /frame/images/frame-charging-diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/images/frame-charging-diagram.png -------------------------------------------------------------------------------- /frame/images/frame-debug-pinout.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/images/frame-debug-pinout.png -------------------------------------------------------------------------------- /frame/images/frame-display.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/images/frame-display.png -------------------------------------------------------------------------------- /frame/images/frame-exploded-view.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/images/frame-exploded-view.png -------------------------------------------------------------------------------- /frame/images/frame-imu.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/images/frame-imu.png -------------------------------------------------------------------------------- /frame/images/frame-microphone.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/images/frame-microphone.png -------------------------------------------------------------------------------- /frame/images/frame-nose-bridge-diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/images/frame-nose-bridge-diagram.png -------------------------------------------------------------------------------- /frame/images/frame-pairing-diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/images/frame-pairing-diagram.png -------------------------------------------------------------------------------- /frame/images/frame-splash.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/images/frame-splash.png -------------------------------------------------------------------------------- /frame/images/frame-sprite-engine.drawio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/images/frame-sprite-engine.drawio.png -------------------------------------------------------------------------------- /frame/images/frame-vs-code-extension.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/images/frame-vs-code-extension.png -------------------------------------------------------------------------------- /frame/images/frame-ycbcr-colorspace.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/frame/images/frame-ycbcr-colorspace.png -------------------------------------------------------------------------------- /images/discord-link.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/images/discord-link.png -------------------------------------------------------------------------------- /images/frame-link.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/images/frame-link.png -------------------------------------------------------------------------------- /images/logo.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | -------------------------------------------------------------------------------- /images/monocle-link.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/images/monocle-link.png -------------------------------------------------------------------------------- /index.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Welcome 3 | description: Technical documentation for Brilliant's AR devices. 4 | image: /images/frame/frame-splash.png 5 | nav_order: 1 6 | --- 7 | 8 | # Get started with your Brilliant products 9 | 10 | Here you'll find everything you need to know about using your Brilliant AR devices. For further help, be sure to join our [Discord](https://discord.gg/7w3DFxek4p) community where our support team will be able to help you directly. 11 | 12 | --- 13 | 14 | [![Brilliant Frame](/images/frame-link.png)](/frame/frame) 15 | 16 | --- 17 | 18 | [![Brilliant Monocle](/images/monocle-link.png)](/monocle/monocle) 19 | 20 | --- 21 | 22 | [![Brilliant Discord server](/images/discord-link.png)](https://discord.gg/7w3DFxek4p) 23 | -------------------------------------------------------------------------------- /monocle/building-apps.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Building Apps 3 | description: A guide on how to develop your own applications for Monocle. 4 | image: /images/monocle/monocle-splash.png 5 | nav_order: 3 6 | parent: Monocle 7 | redirect_from: 8 | - /building-apps 9 | --- 10 | 11 | # Building Apps 12 | {: .no_toc } 13 | 14 | --- 15 | 16 | ## Getting started with AR Studio for VSCode 17 | {: .no_toc } 18 | 19 | AR Studio for VSCode lets you develop, test and save apps directly onto your Monocle. Download it today from the [VSCode Marketplace](https://marketplace.visualstudio.com/items?itemName=brilliantlabs.brilliant-ar-studio) and start developing your Monocle projects. 20 | 21 | ![Brilliant AR Studio for VSCode](/monocle/images/vs-code-extension.png) 22 | 23 | - Once installed, connect to Monocle with `Ctrl-Shift-P` → `Brilliant AR Studio: Connect`. 24 | - Copy the example below, and save the file as `main.py`. 25 | 26 | ```python 27 | import touch 28 | import display 29 | 30 | def change_text(button): 31 | new_text = display.Text(f"Button {button} touched!", 0, 0, display.WHITE) 32 | display.show(new_text) 33 | 34 | touch.callback(touch.EITHER, change_text) 35 | 36 | initial_text = display.Text("Tap a touch button", 0, 0, display.WHITE) 37 | display.show(initial_text) 38 | ``` 39 | 40 | - Upload `main.py` by right clicking on the file in the Explorer and clicking `Brilliant AR Studio: Upload File to Device`. Ensure that you have a workspace or folder open in the Explorer so that you can see the file. 41 | - When prompted for the on-device name, keep it as `main.py` and press Enter. 42 | - Run main.py with `Ctrl-Shift-P` → `Brilliant AR Studio: Build`. 43 | - You should be able to touch each of the touch pads on Monocle, and see the display update. 44 | 45 | --- 46 | 47 | ## Connecting to the wider world 48 | 49 | MicroPython on Monocle simplifies the usage of complex protocols typically employed in Bluetooth devices. Just like Python on desktop, Monocle can be conveniently interacted with using the REPL (Read Evaluate Print Loop) interface. The key difference? It's completely wireless! 50 | 51 | This wireless functionality not only allows for effortless testing and application development on Monocle, but also enables Python commands to control Monocle from your customized iOS, Android, or Desktop app. 52 | 53 | ![Diagram of Monocle network flow](/images/monocle-network-flow.png) 54 | 55 | Below are some templates to help you get started building your own apps. 56 | 57 | ### iOS & Android 58 | 59 | Noa is our ChatGPT client for Monocle. Check out the source code for [iOS](https://github.com/brilliantlabsAR/noa-for-ios), as well as for [Android](https://github.com/brilliantlabsAR/noa-for-android) and use it as a template for your own mobile app. 60 | 61 | ![Brilliant Noa for iOS App](/images/noa-for-ios-screens.png) 62 | 63 | ### Javascript 64 | 65 | The [WebREPL](https://repl.brilliant.xyz) is a good starting point building your own web apps. Try it from Google Chrome on your PC, Mac, Android, or a Web Bluetooth compatible web browser on your iOS device such as [Bluefy](https://apps.apple.com/us/app/bluefy-web-ble-browser/id1492822055). 66 | 67 | Check out the full instructions and the source code [here](https://github.com/siliconwitchery/web-bluetooth-repl/). 68 | 69 | ![Accessing MicroPython on Monocle using the WebREPL interface](/monocle/images/micropython-web-repl.png) 70 | 71 | ### Community projects 72 | 73 | For more application examples and ideas, check out the [community projects](/community) section to see what others have built. -------------------------------------------------------------------------------- /monocle/hardware.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Hardware 3 | description: A brief overview of how Monocle works under the hood. 4 | image: /images/monocle/monocle-splash.png 5 | nav_order: 1 6 | parent: Monocle 7 | --- 8 | 9 | # Monocle Hardware Manual 10 | {: .no_toc } 11 | 12 | --- 13 | 14 | ![Monocle exploded view](/monocle/images/monocle-exploded-view.png) 15 | 16 | Monocle is a tiny heads-up display which clips onto your existing eye-wear. It's packed full of powerful hardware which is perfect for when you're on the go. It connects to your mobile phone over Bluetooth, and includes a few handy sensors such as touch buttons, camera and microphone. The included FPGA is perfect for computer vision, AI or graphics acceleration directly on the device. Monocle runs open source software and provides an easy way to get started using [MicroPython](/monocle/micropython). You can also write your own firmware from scratch using these docs as guidance. 17 | 18 | ## Key features: 19 | {: .no_toc } 20 | - 640x400 color OLED display 21 | - 20° FOV optical body 22 | - 5MP camera 23 | - Microphone 24 | - FPGA based acceleration for ML/CV 25 | - Bluetooth 5.2 26 | - 70mAh rechargeable Li-ion battery 27 | - Touch buttons 28 | - Full featured MicroPython based OS 29 | - Charging case with USB & 450mAh battery 30 | 31 | ## Example uses: 32 | {: .no_toc } 33 | - Generative AI on the go 34 | - ML based image augmentation 35 | - Computer vision research 36 | - QR code & barcode detection 37 | - Heads up telemetry 38 | - AR app & game design 39 | 40 | ## Contents 41 | {: .no_toc } 42 | 43 | 1. TOC 44 | {:toc} 45 | 46 | ## Block diagram 47 | 48 | This diagram shows a general overview of the Monocle architecture. 49 | 50 | ![Monocle block diagram](/monocle/images/monocle-hw-block-diagram.svg) 51 | 52 | --- 53 | 54 | ## Bluetooth MCU 55 | 56 | The Bluetooth MCU serves as the main processor for Monocle. It handles the majority of control over the device, and is used for networking and scripting. The MCU used is a [Nordic nRF52832](https://infocenter.nordicsemi.com/pdf/nRF52832_PS_v1.8.pdf) with **512KB of Flash** memory, and **64KB of RAM**. It supports **Bluetooth 5.2**, up to 2Mb/s. 57 | 58 | By default, the nRF comes preloaded with our [MicroPython](/monocle/micropython) firmware, however you are able to deploy your own [custom firmware](#developing-custom-firmware) if you wish to do so. 59 | 60 | ### Updating firmware over-the-air 61 | 62 | Over the air updates can be performed via Nordic's nRFConnect software either on [Desktop](https://www.nordicsemi.com/Products/Development-tools/nrf-connect-for-desktop), or on Mobile ([iOS](https://apps.apple.com/us/app/nrf-connect-for-mobile/id1054362403)/[Android](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjbtevyrf77AhUncPEDHfjiBSEQFnoECBIQAQ&url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dno.nordicsemi.android.mcp%26hl%3Den%26gl%3DUS&usg=AOvVaw26fnMv6YUhCDOx-ZZHre94)). nRFConnect requires a `.zip` DFU file to perform the update. 63 | 64 | **The latest MicroPython release for Monocle can be found on our [GitHub](https://github.com/brilliantlabsAR/monocle-micropython/releases) releases page.** 65 | 66 | --- 67 | 68 | ## FPGA 69 | 70 | The FPGA is used for graphics acceleration, as well as image processing of the 5MP camera. It's perfect for computer vision and AI tasks which can make direct use of the camera and microphone as inputs. The FPGA IC used is a [Gowin GW1N-LV9MG100C6/I5](http://cdn.gowinsemi.com.cn/DS117E.pdf) from the [Little Bee Family](https://www.gowinsemi.com/en/product/detail/46/). It contains **7k LUTs**, **468kb of block RAM**, as well as **608kb of Flash** memory. An additional [1 megabyte (8 megabits) SPI Flash IC](#flash) is also included in Monocle, and either flash space may be used for booting up the FPGA, or storing user data. For further details, see the [Package & Pinout Guide](http://cdn.gowinsemi.com.cn/UG103E.pdf). 71 | 72 | By default, the FPGA comes pre-loaded our [Graphics & Camera Accelerator IP](https://github.com/brilliantlabsAR/monocle-fpga) and can be accessed using MicroPython commands. Our IP is a perfect starting point if you wish to [create your own FPGA application](#developing-for-the-fpga). 73 | 74 | **The latest release for the FPGA IP can be updated via the Micropython [`update.fpga()`](/monocle/micropython#fpga--monocle-specific) command. It is also checked periodically if you are using the Brilliant App** 75 | 76 | To save power, the FPGA can be shutdown along with the camera and display when not needed. See the [power](#power) section for details. 77 | 78 | --- 79 | 80 | ## Memory 81 | 82 | Aside from the built-in memories of the Bluetooth MCU and the FPGA, Monocle contains three additional memory ICs. One for flash, and two for RAM. The RAM is accessible by the FPGA, and is great for storing display buffers, camera data, or working data for AI algorithms. The flash meanwhile, is great as a secondary source for loading FPGA bitstreams, logging data, and is accessible by both the Bluetooth MCU and FPGA. 83 | 84 | ### Flash 85 | 86 | The Flash IC used is the [Winbond W25Q80EWUXIE](https://www.winbond.com/hq/support/documentation/levelOne.jsp?__locale=en&DocNo=DA00-W25Q80EW). It's an 1 megabyte (8 megabits) serial Flash and is connected to the FPGA and nRF52 via an SPI bus. 87 | 88 | ### RAM 89 | 90 | The RAM ICs used are the [AP Memory APS256XXN](https://www.apmemory.com/wp-content/uploads/APM_PSRAM_OPI_Xccela-APS256XXN-OBRx-v1.0-PKG.pdf). A total of 64 megabytes (512 megabits) of memory is addressable, and can be accessed at up to 800Mb/s in DDR mode. 91 | 92 | --- 93 | 94 | ## Display 95 | 96 | The display used in Monocle is a 0.23" micro OLED. It features **640x400 RGB pixels**, and is optically bonded to the main housing which directs the image into the users eye. The result is a transparent floating display with a **20° field of view**. About the size of a tablet display at arms length. 97 | 98 | ![Diagram of the Monocle display light path](/monocle/images/monocle-display-fov.png) 99 | 100 | To save power, the display can be shutdown along with the camera and fpga when not needed. See the [power](#power) section for details. 101 | 102 | --- 103 | 104 | ## Camera 105 | 106 | The front facing camera on Monocle is an [Omnivision OV5640](https://cdn.sparkfun.com/datasheets/Sensors/LightImaging/OV5640_datasheet.pdf). It is a 5MP color sensor and features numerous convince features such as automatic exposure control, and automatic white balance. It is connected directly to the FPGA over a high speed MIPI CSI-2 interface. 107 | 108 | ![Annotation of the Monocle camera](/monocle/images/monocle-camera.png) 109 | 110 | To save power, the camera can be shutdown along with the fpga and display when not needed. See the [power](#power) section for details. 111 | 112 | --- 113 | 114 | ## Touch interface 115 | 116 | Monocle features two capacitive touch buttons which are fed into an [Azoteq IQS620A](https://www.azoteq.com/images/stories/pdf/iqs620_datasheet.pdf) touch controller. Each button can detect close proximity as well as touch events. With further software processing, double taps, long presses and other gestures can be detected. The touch controller is connected to the Blueooth MCU via I2C and an interrupt line to flag pending touch events. Our MicroPython firmware contains a built in library to access various types of touch events, and trigger actions when pressed. 117 | 118 | ![Annotation of the Monocle touch interface](/monocle/images/monocle-touch-interface.png) 119 | 120 | --- 121 | 122 | ## Microphone 123 | 124 | A [TDK/InvenSense ICS-41351](http://invensense.wpenginepowered.com/wp-content/uploads/2020/07/DS-000157-ICS-41351-v1.4.pdf) microphone is directly connected to the FPGA, and can be used to record audio, or for voice recognition algorithms. 125 | 126 | The audio port is located on the back side of Monocle, so is perfect for receiving the wearers voice, while avoiding excessive background sounds. 127 | 128 | ![Annotation of the Monocle microphone](/monocle/images/monocle-microphone.png) 129 | 130 | The microphone is connected via a PDM interface, and is on the same power domain as the FPGA. 131 | 132 | --- 133 | 134 | ## Power 135 | 136 | Monocle operates on the four power domains as shown below. The colored domains may be powered down by command of the Bluetooth MCU. If the FPGA is shutdown, all the connected peripherals including the display and camera will shutdown also. 137 | 138 | Alternatively, the analog supplies of the camera (2.7V) and display (10V) may be powered down independently of the FPGA. This method, while not being the most aggressive power saving method, allows for faster startup of the camera or display as the digital registers do not need to be reconfigured. 139 | 140 | Power supply diagram of the Monocle 141 | 142 | ### Regulation 143 | 144 | The [Maxim MAX77654 PMIC](https://www.analog.com/media/en/technical-documentation/data-sheets/max77654.pdf) is an efficient and low power management controller. Each rail is configurable and can be tuned for efficiency, stability or noise performance. Each rail can additionally be current limited to avoid overloading the battery. 145 | 146 | The PMIC communicates solely with the Bluetooth MCU over I2C. 147 | 148 | {: .warning } 149 | > It's possible to damage the Monocle hardware by setting the voltages levels too high. We recommend that you avoid diverging from the settings found in the Brilliant provided firmware, to prevent damaging your Monocle. 150 | 151 | ### Battery charging 152 | 153 | The PMIC also includes an integrated battery charger. Charge regulation voltage and current can be configured, as well as various timings. The battery level, and charge current can also be read via an analog pin provided from the PMIC to the Bluetooth MCU. 154 | 155 | {: .warning } 156 | > It's possible to damage the Monocle battery by misconfiguring the charge voltage or current. We recommend that you avoid diverging from the settings found in the Brilliant provided firmware and to prevent fire or damage to the Monocle battery. 157 | 158 | ### LEDs 159 | 160 | Monocle contains two LEDs (green and red) connected to GPIOs on the PMIC. They are easily accesible via PMIC registers from the Bluetooth MCU. 161 | 162 | ![Annotation of the Monocle LEDs](/monocle/images/monocle-leds.png) 163 | 164 | --- 165 | 166 | ## Charging case 167 | 168 | Monocle is provided with a compact charging case for taking on the go. When inserted into the case, Monocle will begin to recharge from the charging case battery. The case battery can provide several full charges of the Monocle and be itself recharged from a standard 500mA USB Type-C jack. While charging, the case will show an orange LED. Once the LED turns off, the case is fully charged. 169 | 170 | ![Annotation of the Monocle charging case](/monocle/images/monocle-charging-case.png) 171 | 172 | When placed into the charger, the Monocle will detect the charge voltage and shut down all peripherals to efficiently charge. Upon removal from the case, and touching of either of the touch pads, Monocle will return into its normally powered state. 173 | 174 | Note that once placed into the charging case, it can take up to 10s for charging to begin. During this time, Monocle will look as if it's awake. If Monocle does not go to sleep after 10 seconds, ensure that the charging case is sufficently charged (by placing it on USB-C charging), or ensure that `device.prevent_sleep()` is disabled. 175 | 176 | --- 177 | 178 | ## Developing custom firmware 179 | 180 | ### Developing for the Bluetooth MCU 181 | 182 | {: .warning } 183 | > It's possible to damage the Monocle hardware by misconfiguring the power supply controller. We recommend that you avoid changing any of those settings as we've already fine tuned them for you. 184 | 185 | A good place to start writing your custom nRF applications is the [Monocle MicroPython repository](https://github.com/brilliantlabsAR/monocle-micropython). All the hardware drivers can be found here and you can follow the `main()` flow to understand how Monocle is booted. 186 | 187 | For compiling projects, you will need to download the latest [ARM GCC compiler](https://developer.arm.com/downloads/-/gnu-rm). For Bluetooth connectivity, you will also need to download a compatible Bluetooth stack from Nordic (aka the [Softdevice](https://www.nordicsemi.com/Products/Development-software/nrf5-sdk/download)). This is a proprietary library from Nordic, so is not directly included within the Monocle MicroPython repository, however is free to download and use. 188 | 189 | ### Creating custom OTA (over-the-air) update packages 190 | 191 | {: .warning } 192 | > It's recommended that your application is well tested and can reliably return back into OTA mode after programming. If you flash a bad application, you'll have to dismantle your Monocle and manually reprogram it using a programmer. 193 | 194 | To generate the `.zip` file, you'll need to use the [nRF Util](https://www.nordicsemi.com/Products/Development-tools/nrf-util) command line application. You can read how to generate files [here](https://infocenter.nordicsemi.com/index.jsp?topic=%2Fug_nrfutils%2FUG%2Fnrfutils%2Fnrfutil_intro.html). You will need to use the [Brilliant OTA key](https://github.com/brilliantlabsAR/monocle-micropython/blob/main/bootloader/published_privkey.pem) in order to generate a compatible image. If you wish to change this key, you can create a new keypair, and add the public key within your [bootloader code](https://github.com/brilliantlabsAR/monocle-micropython/blob/main/bootloader/dfu_public_key.c). Subsequent updates will then require your new private key. **Note: without the key, it's not possible to do over-the-air updates, and you will need to revert to manual programming.** 195 | 196 | ### Developing for the FPGA 197 | 198 | [GoWin's EDA](https://www.gowinsemi.com/en/support/download_eda/) can be used to develop applications for the FPGA. You can request a free licence [here](https://www.gowinsemi.com/en/support/license/). 199 | 200 | We recommend having a look at our [Graphics & Camera Accelerator IP](https://github.com/brilliantlabsAR/monocle-fpga) to see how an FPGA project is set up and built. 201 | 202 | Generally it's convenient to use the [MicroPython FPGA module](/monocle/monocle/#fpga--monocle-specific) to wirelessly program the FPGA. However this method does not allow for JTAG debugging, and can be a slow process for iterative development. It's therefore also possible to program the FPGA manually using a programmer. 203 | 204 | **Note:** Be sure to check your [chip revision](#fpga-chip-revisions) prior to developing and programming your application. 205 | 206 | ### Manually programming 207 | 208 | {: .warning } 209 | > The internal hardware of the Monocle is very delicate, especially the OLED flex cable. This cable contains thirty tiny wires which can be easily be broken if the cable is creased. Additionally, the OLED is bonded to the optical prism for a clear picture. It's impossible to replace this OLED if it breaks. 210 | 211 | 1. To access the programming pads, you'll first need to remove the back cover. 212 | 213 | ![Removing Monocle back cover](/monocle/images/monocle-back-cover.png) 214 | 215 | 1. With the back cover removed, gently extract the main board, taking care not to damage the flex cables, or shorting any of the electronics. 216 | 217 | 1. To shutdown the power, you will need to short the two pads shown below. Metal tweezers work well, but be careful not to short any other components. 218 | 219 | ![Enabling Monocle ship mode](/monocle/images/monocle-ship-mode.png) 220 | 221 | 1. Your device should now be off, but **note the battery pins will still be live**. If you remove any components, **take note of which way they were inserted into the connectors**. Re-inserting the OLED cable or the camera module backwards will permanently damage them. 222 | 223 | 1. The programming pins for the Bluetooth MCU are shown below. It's recommended to use a J-Link compatible programmer such as the [nRF52 DK](https://www.nordicsemi.com/Products/Development-hardware/nrf52-dk). Nordic provide a helper utility which is correctly aware of important settings and flags when reprogramming. However other OpenOCD based SWD programmers may work with some careful tweaking. 224 | 225 | ![Monocle main board FPGA JTAG diagram](/monocle/images/monocle-swd.png) 226 | 227 | 1. The programming pins for the FPGA are shown below. You can use Gowin's GWU2X based dongles to program the FPGA directly from within the IDE. Alternatively, [openFPGAloader](https://github.com/trabucayre/openFPGALoader) works as an open source alternative and can be used with a variety of programmers, such as [FTDI FT2232 based dongles](https://www.seeedstudio.com/Sipeed-USB-JTAG-TTL-RISC-V-Debugger-ST-Link-V2-STM8-STM32-Simulator-p-2910.html). Note that you will need to perform [level shifting](https://en.wikipedia.org/wiki/Level_shifter) between the 3.3V of most tools and the 1.8V of the FPGA, to avoid damaging it. If programming the internal flash, care must be taken to not perform so while the firmware is starting, which could provoke damage to the internal flash (see [RECONFIG_N](http://cdn.gowinsemi.com.cn/UG290E.pdf)). 228 | 229 | ![Monocle main board FPGA JTAG diagram](/monocle/images/monocle-jtag.png) 230 | 231 | 1. To power the device, apply 5V to the charging pads. 232 | 233 | ![Monocle charging pads](/monocle/images/monocle-charging-pads.png) 234 | 235 | 1. To program the Bluetooth MCU, install the [Nordic command line tools](https://www.nordicsemi.com/Products/Development-tools/nrf-command-line-tools), and then run the command: 236 | 237 | ```bash 238 | nrfjprog --program firmware.hex --chiperase -f nrf52 --verify -r 239 | ``` 240 | 241 | 1. To program the FPGA with openFPGAloader, run the command: 242 | 243 | ```bash 244 | openFPGALoader --cable ft2232 --fpga-part 0x0100481b --write-flash bitstream.fs 245 | ``` 246 | 247 | --- 248 | 249 | ## Schematics 250 | 251 | [Download the PDF schematics for Monocle here](/monocle/monocle-schematics.pdf) 252 | 253 | [Download the PDF schematics for the Monocle charging case here](/monocle/monocle-cc-schematics.pdf) 254 | 255 | --- 256 | 257 | ## Mechanical 258 | 259 | ### Monocle 260 | 261 | [![PDF mechanical dimensions for Monocle](/monocle/images/monocle-dimensions.png)](/monocle/monocle-dimensions.pdf) 262 | 263 | [Download the 3D Model in STEP format](/monocle/monocle-body.step) 264 | 265 | [Download the 3D Model in STL format](/monocle/monocle-body.stl) 266 | 267 | ### Charging case 268 | 269 | [![PDF mechanical dimensions for the Monocle charging case](/monocle/images/monocle-cc-dimensions.png)](/monocle/monocle-cc-dimensions.pdf) 270 | 271 | --- 272 | 273 | ## Device characteristics 274 | 275 | Typical and absolute device characteristics are shown below. To get the best lifetime of your Monocle, it's recommended to keep within these limits. 276 | 277 | ### Typical characteristics 278 | 279 | | | Min | Typ | Max | 280 | |:--------------------------|:------:|:---:|:-----:| 281 | | Monocle charging current | - | - | 77mA | 282 | | Case USB charging current | - | - | 500mA | 283 | | Bluetooth radio power | -20dBm | - | 4dBm | 284 | | Bluetooth sensitivity | -96dBm | - | - | 285 | 286 | ### Maximum ratings 287 | 288 | | | Min | Typ | Max | 289 | |:--------------------------|:-----:|:----:|:----:| 290 | | Case USB charging voltage | -0.3V | 5.1V | 30V | 291 | | Monocle charging voltage | -0.3V | 5.1V | 28V | 292 | | Operating temperature | 0°C | - | 45°C | 293 | | Storage temperature | -20°C | - | 60°C | 294 | 295 | --- 296 | 297 | ## Safety & limitation of liability 298 | 299 | ### Safety 300 | 301 | Brilliant Labs' devices can obscure your vision and should not be used while driving or operating dangerous equipment. Additionally long periods of use may cause eye strain, headaches and motion sickness. Brilliant Labs' devices can also display bright flashing images so may not be suitable for those who are susceptible to light sensitivity. 302 | 303 | ### Critical applications 304 | 305 | Brilliant Labs' devices are intended for consumer and R&D applications. It is not verified for use where performance and accuracy would be critical to human health, safety or mission critical use. 306 | 307 | ### Lithium batteries 308 | 309 | Lithium batteries can be dangerous if mishandled. Do not expose Brilliant Labs' devices to excess temperatures, fire or liquids. Do not try to remove the battery as the terminals can become shorted and result in the battery overheating or catching fire. Once the product reaches the end of it's life, dispose it safely according to your local regulations, such as e-waste collection points where any volatile components can be properly contained and handled. 310 | 311 | ### Limitation of liability 312 | 313 | Brilliant Labs provides technical data, including design resources, examples, applications, design advice, tools, safety information and other resources "as is" and disclaims all warranties, express and implied, including without limitation any implied warranties or merchantability, fitness for a particular purpose or non-infringement of third party intellectual property rights. 314 | 315 | These resources are intended for skilled developers. You are solely responsible for selecting the appropriate products for your application, designing, validating and testing your application, and ensuring your application meets applicable standards, and other safety, security, regulatory or other requirements. 316 | 317 | Brilliant Labs reserves the right to change the circuitry and specifications without notice at any time. The parametric values quoted in this manual are provided for guidance only. 318 | 319 | The resources and products are provided subject to our [terms and conditions](https://brilliant.xyz/pages/terms-conditions). 320 | -------------------------------------------------------------------------------- /monocle/images/bluetooth-serial-service-sequence-diagram.svg: -------------------------------------------------------------------------------- 1 | Central
enable notifications
enable notifications
Connection
Connection
Short message (strlen() < MTU length - 3)
Short message (strlen() < MTU length - 3)Monocle
write on RX characteristic
write on RX characteristic
write on RX characteristic
write on RX characteristic
notify on TX characteristic
notify on TX characteristic
notify on TX characteristic
notify on TX characteristic
connect
connect
connected
connected
Accept connection
Accept connection
String received and processed by REPL
String received and processed by REPL
Response string generated
Response string generated
notify on TX characteristic
notify on TX characteristic
Long message (strlen() > MTU length - 3)
Long message (strlen() > MTU length - 3)
Split message into MTU-3 sized chunks
Split message into MTU-3 sized chunks
write on RX characteristic
write on RX characteristic
write on RX characteristic
write on RX characteristic
String received and processed by REPL
String received and processed by REPL
Response string generated
Response string generated
Asynchronous messages generated by the REPL
Asynchronous messages generated by the REPL
notify on TX characteristic
notify on TX characteristic
notify on TX characteristic
notify on TX characteristic
Messages received and processed
Messages received and processedText is not SVG - cannot display -------------------------------------------------------------------------------- /monocle/images/micropython-web-repl.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/micropython-web-repl.png -------------------------------------------------------------------------------- /monocle/images/monocle-back-cover.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/monocle-back-cover.png -------------------------------------------------------------------------------- /monocle/images/monocle-camera.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/monocle-camera.png -------------------------------------------------------------------------------- /monocle/images/monocle-cc-dimensions.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/monocle-cc-dimensions.png -------------------------------------------------------------------------------- /monocle/images/monocle-charging-case.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/monocle-charging-case.png -------------------------------------------------------------------------------- /monocle/images/monocle-charging-pads.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/monocle-charging-pads.png -------------------------------------------------------------------------------- /monocle/images/monocle-dimensions.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/monocle-dimensions.png -------------------------------------------------------------------------------- /monocle/images/monocle-display-fov.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/monocle-display-fov.png -------------------------------------------------------------------------------- /monocle/images/monocle-exploded-view.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/monocle-exploded-view.png -------------------------------------------------------------------------------- /monocle/images/monocle-jtag.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/monocle-jtag.png -------------------------------------------------------------------------------- /monocle/images/monocle-leds.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/monocle-leds.png -------------------------------------------------------------------------------- /monocle/images/monocle-microphone.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/monocle-microphone.png -------------------------------------------------------------------------------- /monocle/images/monocle-network-flow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/monocle-network-flow.png -------------------------------------------------------------------------------- /monocle/images/monocle-ship-mode.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/monocle-ship-mode.png -------------------------------------------------------------------------------- /monocle/images/monocle-splash.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/monocle-splash.png -------------------------------------------------------------------------------- /monocle/images/monocle-swd.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/monocle-swd.png -------------------------------------------------------------------------------- /monocle/images/monocle-touch-interface.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/monocle-touch-interface.png -------------------------------------------------------------------------------- /monocle/images/noa-for-ios-screens.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/noa-for-ios-screens.png -------------------------------------------------------------------------------- /monocle/images/vs-code-extension.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/images/vs-code-extension.png -------------------------------------------------------------------------------- /monocle/monocle-body.stl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/monocle-body.stl -------------------------------------------------------------------------------- /monocle/monocle-cc-dimensions.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/monocle-cc-dimensions.pdf -------------------------------------------------------------------------------- /monocle/monocle-cc-schematics.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/monocle-cc-schematics.pdf -------------------------------------------------------------------------------- /monocle/monocle-dimensions.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/monocle-dimensions.pdf -------------------------------------------------------------------------------- /monocle/monocle-schematics.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/monocle/monocle-schematics.pdf -------------------------------------------------------------------------------- /monocle/monocle.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Monocle 3 | description: Technical documentation for the Brilliant Monocle. 4 | image: /images/monocle/monocle-splash.png 5 | nav_order: 3 6 | has_children: true 7 | --- 8 | 9 | 10 | ![Brilliant Monocle use cases](/monocle/images/monocle-splash.png) 11 | 12 | # Technical Documentation 13 | 14 | Monocle was designed for hackers, by hackers. Take a moment to familiarize yourself with the [Monocle hardware](/monocle/hardware), and when you're ready to delve deeper, visit the [Building Apps](/building-apps) section to discover how you can construct your own augmented reality apps. 15 | 16 | ### How do I start my Monocle? 17 | 18 | As soon as you remove Monocle from its case, it will automatically power on. You'll notice the display turning on, and you can easily connect via Bluetooth. When you place Monocle back into the case, it will go to sleep and recharge automatically. 19 | 20 | ### How do I program my Monocle? 21 | 22 | Monocle understands Python and offers a convenient set of APIs to control the display, camera, touch, and microphone. By leveraging Brilliant's [AR Studio for VSCode](https://marketplace.visualstudio.com/items?itemName=brilliantlabs.brilliant-ar-studio), you can create Python applications and effortlessly test them on Monocle. To get started, explore the helpful examples in our [reference documentation](/monocle/micropython). Alternatively, you can browse the [community projects page](/community) to discover what others have built with Monocle. 23 | 24 | ### How do I update my Monocle? 25 | 26 | Brilliant AR Studio will promptly notify you whenever a new firmware update is available, ensuring that you stay up-to-date with the latest features and improvements. Additionally, if you prefer, you have the option to load custom FPGA images directly from within AR Studio. 27 | 28 | ### Are there any out of the box applications? 29 | 30 | Noa by Brilliant, brings ChatGPT directly to your Monocle. With just a tap and a spoken command, you can witness ChatGPT's response right in front of your eyes. 31 | 32 |
33 | [Apple App Store badge](https://apps.apple.com/us/app/argpt/id6450499355) 34 |         35 | [Google Play Store badge](https://play.google.com/store/apps/details?id=xyz.brilliant.argpt) 36 |
37 | -------------------------------------------------------------------------------- /monocle/update.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Firmware Updates 3 | description: A guide on how to update your Monocle AR device. 4 | image: /images/monocle/monocle-splash.png 5 | nav_order: 4 6 | parent: Monocle 7 | redirect_from: 8 | - /monocle/update 9 | --- 10 | 11 | # Updating your Monocle Firmware 12 | {: .no_toc } 13 | 14 | --- 15 | 16 | If you're using the [Brilliant AR Studio](/monocle/building-apps/#getting-started-with-ar-studio-for-vscode), firmware updates are automatic, and you will be prompted whenever a new update is available. 17 | 18 | If you run into problems, you can update manually using the following steps: 19 | 20 | 1. Firstly, ensure that you're using a browser that supports Web Bluetooth such as Google Chrome for Desktop, Android Chrome, or [Bluefy](https://apps.apple.com/us/app/bluefy-web-ble-browser/id1492822055) on iOS. 21 | 22 | 1. Next, navigate to the WebREPL at [https://repl.brilliant.xyz](https://repl.brilliant.xyz) 23 | 24 | 1. Press any key to open the connection dialog box. 25 | 26 | 1. Select **monocle** from the list and hit connect. (Note, if a previous update had failed, you may see **DFUTarg** in the list. In this case, connect to that instead). 27 | 28 | 1. Once connected, check the message at the bottom of the screen. If it says `Connected.` that means that you're already on the latest firmware, if not, you will be prompted that an update is available and you can click the update button to start the update. 29 | 30 | 1. If you see a message that the current firmware could not be detected, then you can use the following commands to start the update manually: 31 | 32 | ```python 33 | import update 34 | update.micropython() 35 | ``` 36 | 37 | 1. Monocle will then restart into the update mode, and you'll be prompted to reconnect. Press any key to open the connection dialog box again. 38 | 39 | 1. Select **DFUTarg** from the list, and connect. 40 | 41 | 1. The firmware update will start and Monocle will reboot. You may need to place your Monocle into the case once the update is complete to reboot it. 42 | 43 | 1. That's it! You're Monocle should now be up to date. Check it by using the commands below once connected again. The version number should match the latest release on the Monocle MicroPython [releases page](https://github.com/brilliantlabsAR/monocle-micropython/releases). 44 | 45 | ```python 46 | import device 47 | device.VERSION 48 | ``` 49 | -------------------------------------------------------------------------------- /noa/delete-account.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Noa 3 | description: Instructions for how to delete your Noa account 4 | image: /images/frame/frame-splash.png 5 | nav_order: 10 6 | has_children: false 7 | nav_exclude: true 8 | --- 9 | 10 | # Deleting your Noa account 11 | 12 | To delete your Noa account. Navigate to the account page inside the Noa app, and then tap the **Delete Account** button. Your account will then be deleted after 30 days. If you log back within 30 days, your account will be restored. 13 | 14 |
15 | How to delete the Noa user account 16 |
-------------------------------------------------------------------------------- /noa/images/delete-account.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brilliantlabsAR/docs/95cd039effa54f63ab31b2867581f50c774b7f52/noa/images/delete-account.png -------------------------------------------------------------------------------- /robots.txt: -------------------------------------------------------------------------------- 1 | --- 2 | layout: none 3 | --- 4 | User-agent: * 5 | Sitemap: {{ site.url }}/sitemap.xml --------------------------------------------------------------------------------