├── vst2-gui-requirements.md ├── already-existing-crates.md ├── additional-resources.md ├── README.md ├── rutabaga-notes.md ├── vst2-gui-background.md └── gui-from-scratch.md /vst2-gui-requirements.md: -------------------------------------------------------------------------------- 1 | # VST2 GUI requirements 2 | 3 | **TODO.** This will likely be a "living document" as we discover more about the problem space. 4 | -------------------------------------------------------------------------------- /already-existing-crates.md: -------------------------------------------------------------------------------- 1 | # Notes on using already-existing crates 2 | 3 | Document anything you've found by trying to use crates from the greater Rust ecosystem here. 4 | 5 | What works already? What's preventing us from using it with `vst-rs` (parent window handle / child window creation, event loop, etc.)? Are there any GitHub issues tracking those things? How hard would it be to hack it to get it to work for our needs? How hard would it be for us to maintain that fork? 6 | 7 | Also, put a date next to your findings so we can judge how fresh the information is. 8 | 9 | ## Iced 10 | 11 | Repo: https://github.com/hecrj/iced 12 | 13 | [12/12/2019] 14 | 15 | There is a [very early vst-rs gui example](https://github.com/hatoo/vst-rs-example-iced) with an associated [iced issue](https://github.com/hecrj/iced/issues/118) tracking the ability to pass state into the GUI application. 16 | 17 | Note that Iced uses Winit as a backend, which currently [doesn't support parent windows on Linux and Mac OS](https://github.com/rust-windowing/winit/issues/159#issuecomment-511131729) yet. Getting cross-platform VST2 GUIs working with Iced will require getting this working in Winit first. 18 | 19 | ## Conrod 20 | 21 | *(TODO)* 22 | 23 | ## Piston 24 | 25 | *(TODO)* 26 | 27 | ## Druid 28 | 29 | *(TODO)* 30 | 31 | ## gfx-rs 32 | 33 | *(TODO)* 34 | -------------------------------------------------------------------------------- /additional-resources.md: -------------------------------------------------------------------------------- 1 | # Other resources that might be worth looking into 2 | 3 | Some of these links might also be scattered around this repo, but it's probably useful to collect them on this page too. 4 | 5 | - [**rutabaga**](https://github.com/wrl/rutabaga/) - [@wrl](https://github.com/wrl) made his own OpenGL GUI crate in C called `rutabaga`, which he used for his commercial VST [cadmium](https://lhiaudio.com/cadmium/). He's interested in eventually porting some of the ideas over to Rust, but in the meantime he's open to people looking into it and getting ideas for their own Rust toolkits. Relicencing from 2-clause BSD is possible if needed; that and other questions about rutabaga can be directed to him (he's pretty active on [the RustAudio Discord](https://discord.gg/QPdhk2u)). 6 | - I made a page for [Rutabaga-specific implementation detail notes](./rutabaga-notes.md). 7 | - [**rtb-rs**](https://github.com/rust-dsp/rtb-rs) is one such attempt at (very vaguely) "porting some ideas" of `rutabaga` over to rust. `rtb-rs` is a joint-effort by a few members of the [RustDSP GitHub organization](https://github.com/rust-dsp). Unfortunately, development has ultimately stalled for the time being, but some of the code might still be useful to reference in the future (particularly, how to structure a cross-platform crate composed of platform-specific modules). I wrote a little bit more about the project [on this page](./gui-from-scratch.md#rtb-rs). 8 | - [**vst2-window**](https://github.com/crsaracco/vst2-window) is another (very similar to `rtb-rs`) attempt at creating a cross-platform crate "from scratch", by [@crsaracco](https://github.com/crsaracco). The main idea that's different from `rtb-rs` is to let the platform-agnostic API part grow more organically than it did in `rtb-rs`. Some [platform-specific VST prototypes were made](https://github.com/crsaracco/vst2-gui-prototypes) for some pre-development research, which might be useful to reference in the future. I wrote a little bit more about the project [on this page](./gui-from-scratch.md#vst2-window). 9 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # VST2 GUI research 2 | 3 | This repo is intended to be a brain-dump for anyone interested in working on GUIs for [vst-rs](https://github.com/RustAudio/vst-rs). 4 | 5 | There have been a few efforts in the past for trying to get this working, but what always ends up happening is that someone dumps a considerable amount of time working on it, learns a bunch of new information, then drops the project for whatever reason, slowly losing the information they've gained. My goal for this repo is to instead retain that information, so that we can eventually converge on a solution. 6 | 7 | If you're interested in doing some GUI work in this space, read through this repo to get a feeling for what the requirements are, what has worked, and what hasn't. 8 | 9 | If you've been doing some GUI work and you have *any* bit of information that isn't in this repo, please file a PR or let us know what you've found on [the RustAudio discord server](https://discord.gg/QPdhk2u) (channel #vst2-gui)! 10 | 11 | ## Goals 12 | 13 | Ideally, we would like to have a GUI system that works on all three of the "major" platforms that might use `vst-rs`: Windows, Mac OS X, and Linux (X11). A programmer who uses the solution we come up with should be able to just *use it* to create a VST GUI: they shouldn't have to worry about coding platform-specific stuff, and any given feature we have should work equally well on all three platforms. 14 | 15 | Information that only applies to one of those specific platforms is still appreciated, because it still helps us scope out the overall requirements. Other platforms are considered "out of scope" for this project, but I will happily collect that information as well. 16 | 17 | I will list out "harder" requirements on a separate page. 18 | 19 | ## Approaches 20 | 21 | There are two general approaches to getting GUIs working for Rust VSTs: 22 | 23 | 1. Try to use a cross-platform crate that already exists, figure out it isn't *quite* what you need, and decide to hack around with it trying to get something working. 24 | 2. Build something new from the ground up that fits our exact requirements, and realize there's quite a bit of work ahead of you. 25 | 26 | Each of these approaches will have their own page to document findings. 27 | 28 | ## Information 29 | 30 | - [Background on how GUIs work in VST2](./vst2-gui-background.md) 31 | - [Requirements specification for a Rust VST2 GUI crate](./vst2-gui-requirements.md) **(TODO)** 32 | - [Notes on using an already-existing Rust GUI crate](./already-existing-crates.md) 33 | - [Notes on creating a new GUI crate from the ground up](./gui-from-scratch.md) 34 | - [Additional resources that might be worth looking into](./additional-resources.md) 35 | - [Notes on Rutabaga implementation details, specifically](./rutabaga-notes.md) 36 | -------------------------------------------------------------------------------- /rutabaga-notes.md: -------------------------------------------------------------------------------- 1 | # Rutabaga notes 2 | 3 | Information about Rutabaga-specific implementation details, from @wrl himself. Light editing for formatting reasons. 4 | 5 | Editor's note: I'd like to eventually take this information and generate some requirements that each specific platform has to implement, but for now I'm just leaving it in "brain dump" format. Also, "implementation details" probably doesn't describe this page too accurately, since a lot of this probably applies to *any* VST-specific GUI crate you'll make. Regardless, here's the deets: 6 | 7 | ## Brain dump 8 | 9 | - On linux, since the connection to the window server (under x11) is just a file descriptor, you're free to spin up a new thread and run your own event loop from there, and this is what I do in rtb. Sort of. rtb assumes that it owns the event loop on linux, so in the plugin I spin up a new thread and run rtb from it. 10 | - On linux, you must spin up a thread to own the event loop, for there is no other event loop to which you can subscribe. On windows and mac, you *absolutely cannot* create a new thread. Everything will break in very sludgy unfortunate ways. 11 | - The reason why it breaks things is generally deep inside the platform library and "I'm not really sure". 12 | - I know on windows that the event dispatching that goes from parent window to child window can't cross threads, or the events just... don't show up. 13 | - On mac, NSView is marked "main thread only". So I assume there are assumptions in cocoa related to that -- such as some data structures being insufficiently (or not at all) protected by mutexes or whatever. 14 | - Nobody has decided how plugin UIs should work natively on wayland yet, so that's all WIP and rtb has no support for it presently. 15 | - On windows and macos, the connection to the window system is a platform-specific abstraction and is not exposed to client code in a reliable and accessible fashion. 16 | - On windows it's `HANDLE hEventQueueClient` in `THREADINFO` 17 | - On macOS it's a mach port but i don't know where to find it. this way lies dragons, you will drive yourself mad, your code will break. perish the thought. 18 | - Roughly: in `open_editor()`, you create the platform window, hook the events you want (on windows you implement this in your `windowproc`, on macOS you subclass `NSView` and override the handler methods) and then return from `open_editor()`. this is the hard part – UI toolkits, as we've established, aren't used to working like this. 19 | - You'll also want to create some way of doing redraws. VST2 has an "editor idle" opcode but it's up to the host how often it gets called and other plugin APIs (AUv2 for one) don't provide the facility. 20 | - Your easiest route here is to create a timer (`WM_TIMER` on windows, `CFRunLoopTimer` on mac) and hook it into the event loop. This is your redraw clock. Life gets more difficult while handling resize (especially on windows) but in a plugin UI you generally don't have to deal with that. 21 | - I am also interested in using `CVDisplayLink` on mac for better vblank and fps sync but it complicates locking and also you need to catch notifications for when the view moves to a different monitor... blegh. WIP. Timer method works fine. 22 | - **You will absolutely want a communication channel of some sort for the UI.** 23 | - At the bare minimum you will want DSP -> UI for things like knowing to redraw widgets if the parameter changes. `set_parameter()` *will* get called on the DSP thread and you cannot mess with the UI at all from it. This is **vital**, you **will** cause underruns [if you do]. 24 | - Have a little SPSC message queue. Bounded, so you're not allocating. In my plugin I have my own circular buffer for the message storage and then I use a tiny signaling primitive (pipe on mac, eventfd on linux, I think EVENT on windows, but it's a libuv thing regardless) for notifying the UI event loop. Strictly speaking you could get away without a signaling primitive -- just use the buffer and check it in your redraw timer. 25 | - I really want to get multithreaded drawing working on windows and mac because openGL vblank usually works by blocking in `glSwapBuffers()` until the vblank is complete. Which means that the buffer swap is holding up the whole event loop, and that's just no bueno. 26 | - The name of the game with all of this deep platform-specific work is "sometimes what you're trying to do just won't work and you have to hack around it". 27 | -------------------------------------------------------------------------------- /vst2-gui-background.md: -------------------------------------------------------------------------------- 1 | # Background on how GUIs work in VST2 2 | 3 | *(**TODO:** might break this up so that each platform gets its own section, depending on how this page evolves.)* 4 | 5 | *(**TODO:** at the time of writing, I haven't worked on GUIs in a while, so certain things are a bit generalized and hand-wavey. More precise information is always good!)* 6 | 7 | ## Host window creation 8 | 9 | The DAW handles the lifecycle of the actual GUI window: creating it, opening it, closing it, etc. 10 | 11 | When a VST plugin wants to have a GUI, the DAW creates this window and passes a handle to it to the plugin, so the plugin has access to it. 12 | 13 | The type of window that is created is **platform-dependent**, which means that the API you use to interact with the window is also platform-dependent. 14 | 15 | Your job as a VST2 GUI developer is always the same, though: "connect" your GUI to this parent window, using the handle that the DAW gave you, so that the DAW can manage the window (open, close, get events, etc). The "connecting" part is usually done by passing the handle as an argument to your platform's "native" window creation API (Windows' `CreateWindowEx`, X11's `xcb::create_window`, etc). 16 | 17 | #### Specifics for Rust and `vst-rs` 18 | 19 | When using `vst-rs`, the handle will always be a `*mut std::os::raw::c_void`, which is given to you through [`Editor::open`](https://github.com/RustAudio/vst-rs/blob/c1e29953a80946c47987180bec8b3c26c494941a/src/editor.rs#L32): 20 | 21 | ```rust 22 | fn open(&mut self, parent: *mut c_void) -> bool; 23 | ``` 24 | 25 | - In Windows, that parent pointer is really a `winapi::HWND__`. You can convert the parent pointer by doing `parent as *mut winapi::HWND__`, and connecting to the window through [`CreateWindowExW`](https://github.com/crsaracco/vst2-gui-prototypes/blob/master/windows-opengl-vst/src/editor/window/win32_window.rs#L23-L36). 26 | - In Linux, that parent pointer is actually just an ID number to the parent window. You can convert the parent pointer by doing `parent as u32` and then connecting to the window through [`xcb::create_window`](https://github.com/crsaracco/vst2-gui-prototypes/blob/master/linux-opengl-vst/src/editor/window/mod.rs#L112-L114). 27 | - On MacOS, the parent pointer type depends on a few factors: 28 | - If the plugin is running in 32bit and the plugin has *not* received a `canDo` request for `"hasCockosViewAsConfig"`, the parent pointer will be a Carbon `WindowRef`. 29 | - If the plugin is running in 64bit, or if it is running in 32bit and *has* received a `canDo` request for `"hasCockosViewAsConfig"`, the parent pointer will be a Cocoa `NSView *`. 30 | - When receiving the `"hasCockosViewAsConfig"` `canDo` request, the plugin should return `0xbeef0000` to the host. 31 | - Honestly, just don't ship 32bit anymore. It's dead completely as of 10.15, and the additional complexity of wrapping the Carbon window is best avoided. 32 | 33 | - AudioUnits have a slightly different model. Rather than asking the plugin to parent the `NSView` itself, instead the plugin returns an `NSView *` to the host, and the host can then handle the reparenting itself. This, however, means that you need to override `[NSView dealloc]` or `[NSView release]` in order to figure out when the host has closed the plugin editor so the plugin can release resources. 34 | 35 | Most crates assume you just want to get up-and-running with a cross-platform standalone GUI window, so they hide all the platform-specific stuff from you. They have no functionality to "connect" to a platform-specific window, like we need to do here, so there's no way to actually get a `*mut c_void` handle out of them. **This is one of the main reasons why most UI toolkits won't work for VST.** *(Check out the ["Notes on using an already-existing Rust GUI crate"](already-existing-crates.md) page for more specific information.)* 36 | 37 | ## Events and threads 38 | 39 | VST2 expects the DAW to have complete control over the application, including its threads. Generally, the DAW will have one or more threads dedicated for each running plugin, and will send calls over FFI when an event is ready to be dealt with (window closing, audio buffer ready to be processed, etc). VST2 expects that function call to give control back to the DAW, so it shouldn't run forever. 40 | 41 | On the other hand, most UI toolkits (including most Rust ones) assume that they own the top-level event loop -- that is, they own `main()` and never return from it. They almost always assume that you're building a standalone GUI application, so they assume that you want your own event loop and processing thread, so they make it for you and never return from it until your program is done. 42 | 43 | **These two assumptions are directly contradictory, and is the other main reason why most UI toolkits won't work for VST.** *(Check out the ["Notes on using an already-existing Rust GUI crate"](already-existing-crates.md) page for more specific information.)* 44 | -------------------------------------------------------------------------------- /gui-from-scratch.md: -------------------------------------------------------------------------------- 1 | # Notes on creating a new GUI crate from the ground up 2 | 3 | If no already-existing GUI crates work well out-of-the-box for our purposes, what about making our own? We can get very fine-grained control over the [two main problems in VST GUI creation](vst2-gui-background.md), and we can tailor it to [our exact requirements](vst2-gui-requirements.md); no more, no less. 4 | 5 | Unfortunately, "our requirements" actually encompass a *lot* of work: 6 | 7 | - Window creation / attaching to a parent window -- *for each platform* 8 | - Cross-platform GUI context creation (OpenGL, Vulkan/Metal, etc) -- *for each platform* 9 | - Event handling -- *for each platform* 10 | - Wrapping all of that up in a platform-agnostic crate for others to use 11 | 12 | And that's before we try to create *another* crate for drawing on the GUI context -- circles, rectangles, menu systems, etc. 13 | 14 | **TL;DR: Creating our own GUI crate from scratch is a lot of work**, and the tradeoffs between it and trying to use an existing GUI crate should definitely be considered. Even if we get the basics down, there's still a lot of hidden issues like "how does DPI scaling work" and many others, that an existing GUI crate would already have solved out-of-the-box. Even if all of the other crates might not work for our use-case, we're still duplicating *a lot* of effort and fragmenting the already-thin Rust GUI ecosystem further. Definitely consider if it's worth it to contribute to some other upstream crate to try to fulfill our use-case, or at least fork one and mold it to suit our needs. 15 | 16 | ## The basic idea 17 | 18 | The basic idea is to [define a set of requirements](vst2-gui-requirements.md), get each requirement working in a platform-specific module, and then wrap everything together in a cross-platform interface module/trait/whatever so that people can use the crate without worrying about the platform-specific implementation details. 19 | 20 | - Working with the platform-specific APIs for managing windows is tedious, but doable. 21 | - Setting up a cross-platform GUI context (OpenGL, etc.) is possible, with some googling. 22 | - The harder part is figuring out the intricacies of each platform's event system, and trying to design an umbrella API that can handle each one in a platform-agnostic way. 23 | 24 | Once we have those things sorted, we should be able to create a window that connects to the window that the DAW gives us, and handle events like mouse clicks and keyboard presses in a way that our VST plugin can understand. Additionally, the user should be able to target one GUI paradigm (OpenGL, etc.) and have it work on all three platforms. 25 | 26 | As mentioned before, it would be nice to also have another crate on top of that for GUI primitives (creating circles/rectangles, etc) in the chosen system, so that it's easier to draw potentiometers, levels, menus, and whatever else people would like to create. 27 | 28 | ## Past attempts 29 | 30 | #### [rtb-rs](https://github.com/rust-dsp/rtb-rs) 31 | 32 | `rtb-rs` is (was?) an attempt to create a library built around the "basic idea" above. The project got to the point where you could spawn a window that connects to the DAW on all three platforms through a platform-agnostic API; OpenGL and event support is a little more sparse. 33 | 34 | The project suffered from two issues: 35 | 36 | 1. A lot of the "platform-agnostic" API was built up before anyone knew too much about the platforms themselves, which led to a lot of churn as developers discovered that parts of the API wouldn't work on all three platforms and resulted in some platform-specific code having to undergo major reworks or being thrown out entirely. 37 | 2. There wasn't a concentrated effort in the development of each of the individual platforms (and Windows didn't have a dedicated developer at all), so when churn did happen, it took *quite a long time* to resolve. 38 | 39 | These are mostly project management issues; no technical/software issues were uncovered that stopped us from continuing this project to completion. I think if we had more devs actively working on the project (in particular, at least one active dev for each platform), this project could actually work with enough elbow grease. 40 | 41 | #### [vst2-window](https://github.com/crsaracco/vst2-window) 42 | 43 | Taking note of problem #1 from `rtb-rs`, I decided to try my hand at creating my own crate from the ground up. I didn't define a platform-agnostic API to start with; instead, I opted to let the platform-agnostic API grow organically as I observed what each platform needed to do to fill the requirements of a VST GUI. 44 | 45 | In my research phase, I decided to [create some prototype VSTs that each only support one platform](https://github.com/crsaracco/vst2-gui-prototypes). 46 | 47 | - I got a [Linux (X11)-specific VST](https://github.com/crsaracco/vst2-gui-prototypes/tree/master/linux-opengl-vst) working with OpenGL and simple events. 48 | - I got a [Windows-specific VST](https://github.com/crsaracco/vst2-gui-prototypes/tree/master/windows-opengl-vst) working with OpenGL and simple events. 49 | - I even purchased a Mac Mini and [did some prototype OpenGL work](https://github.com/crsaracco/vst2-gui-prototypes/tree/master/osx-opengl), without the VST part. 50 | 51 | Once I did all three of those, I felt reasonably confident that I could get everything working and started to create [`vst2-window`](https://github.com/crsaracco/vst2-window). My idea was to define one chunk of the platform-agnostic interface, then implement it for all three platforms, then define one more platform-agnostic chunk, implement, and iterate like that. 52 | 53 | Around here, I started to burn out -- I don't really like GUI programming in the first place, and I felt like I'd rather be coding other things, so I shelved the project for the time being. 54 | 55 | This project ultimately suffered from one issue: 56 | 57 | 1. The project had one programmer, who doesn't really like GUI programming in the first place :). Since I'm not really familiar with GUI programming in general, I had to go research how to do the same thing three different times, once for each platform. It's mind-numbing. 58 | 59 | With more (active) programmers, each focused on one particular platform, I think this method is the best way to do it and is very likely to succeed. 60 | 61 | ## Ideas for future attempts 62 | 63 | Given the successes and failures of the previous two projects, here's how I would go about trying to implement our own cross-platform GUI crate from scratch: 64 | 65 | 1. Conjure up some developers who are willing to dedicate the good part of the next few months' worth of weekends to programming such a GUI crate, while being active on Discord to help keep the project going smoothly. 66 | - Ideally, each dev working on their own favorite platform, so that dumping some time into research doesn't seem like such a waste of time to them. 67 | - At least one dev per platform, ideally two or more. 68 | 2. Brainstorm what an ideal end-goal would be like. 69 | - What's in scope, and what's out-of-scope? 70 | - Which cross-platform GUI context to use? OpenGL? Vulkan + Metal? 71 | - Which events to handle? 72 | - What should the graphics and events API look like to a VST programmer using this crate? 73 | 3. Go out and develop platform-specific VSTs (like the [prototypes](https://github.com/crsaracco/vst2-gui-prototypes) repo I made) to get a feel for the problem space in that particular platform. If there's more than one developer for a given platform, they can either work together to do this, or split up and each make their own. Either way, try to implement a majority of the end-goal requirements for each platform before reconvening and working on the next step. 74 | 4. Make a new shared repo that defines a small chunk of the platform-agnostic API, and have each platform-specific team implement it completely before moving on. Then, define another small chunk of the platform-agnostic API, and implement that. Iterate until complete. 75 | - You'll probably want to start with window creation / connecting to the DAW window as the first major step. Then, maybe getting an OpenGL/Vulkan+Metal context and proving that you can draw to it with platform-agnostic APIs? Then event handling? 76 | - Ideally, you constantly figure out which part has the biggest risk of not fitting into the platform-agnostic API and therefore messing everything up, and work on that next. The big goal here is to minimize the amount of churn that might happen if/when something does mess everything up. Try to make as few assumptions as possible. 77 | - Since each team already made their own platform-specific VST, this phase should *hopefully* be relatively easy; implementing each iteration should be only slightly harder than copy-pasting your previous work into this new repo. 78 | --------------------------------------------------------------------------------