Design an API-Independent plugin interface in Rust

[crsaracco]

Okay, gathered together some thoughts finally, now that I’m home from work.

TL;DR:

• I’m excited about the idea of an API-independent plugin suite, sounds like an awesome idea to me. However, I’d still like to be able to target specific plugin formats alone if I chose to do so.
  • VST2 isn’t the only one I feel this way about; I’d like standalone VST3 and LV2 crates as well.
• I’m totally game to split the current VST2 crate into vst and vst-sys; that sounds useful to me regardless of if we have an API-independent suite or not.
• I’m still game to try to support the GUI story of any plugin crates we decide to support regardless; I think it’s important to support GUI features if the plugin format allows for it.
• If we decide to straight up delete the VST-specific crate, let me know beforehand so I can fork it

Disclaimers:

1. I basically have no idea what I’m talking about when it comes to plugin formats other than VST2.
2. I haven’t really worked with JUCE, haven’t looked at GMPI yet, and haven’t played around with any other multi-plugin-supporting suite ideas at all.
3. I haven’t even “shipped” a plugin, nor even made a non-trivial VST yet, so some people (@wrl for sure) probably have a lot more knowledge on the topic of full-scale plugins than I do regardless.

My main concern:

Ok, I admit I might just be Afraid Of Change™ on this topic, but my gut reaction is that it’s going to be hard to support 4+ different plugins at the same while still being able to pay attention to plugin-specific details without either:

1. forcing the user to write some per-plugin-format logic (eww), or
2. degrading to some form of “lowest common denominator” API specification to handle all of the different formats.

Like I said earlier, I basically have no idea what I’m talking about when it comes to other plugin formats, but I imagine the differences are not too easy among the different plugin formats, like:

• differences in how the threading/concurrency works
• differences in how the GUIs work
• differences in how parameters work
• differences in how events work (MIDI, etc)
• differences in how channel setup works
• maybe even differences in how process() equivalent functions work (?)
• 32 bit vs. 64 bit for parameters, process(), events, and arguments to functions

I wasn’t able to find any actual numbers regarding plugin format market share, but I’m going to take a guess and say that VST2 is well in the lead at the current moment. Just from a pragmatic, “I just want to make a plugin right now” standpoint (and from the “I want to support the maximum number of people with the least amount of effort” standpoint), I’m going to argue that supporting VST2 as a standalone crate is still worth it, at least for now.

• If I find a bug in a VST2 plugin, it will be easier to identify it in a standalone crate vs. figuring out if it’s just the VST2 module of an API-independent crate, or if it’s a bug in the API wrapper portion, etc. And if it is in the API wrapper portion, how said bug would affect all of the other plugin types, how to fix them all, etc.
• If I want to experiment with my own concurrency model, I feel like it’s going to be difficult to work with 4+ plugin formats rather than just one. If I’m targeting just VST2, I can constrain my problem to just that domain and figure out a solution.
• Same thing as the previous bullet point, but for parameters and process().
• I’m going to guess that the API-independent crate will introduce a tiny bit of processing overhead, which I’d like to have the option of avoiding if I wanted to. (This isn’t a major concern, but options are always good.)

Basically, I guess what I’m saying is I’d like to maintain the option of trading off: 1) supporting all of the various plugin formats at the same time, vs. 2) supporting one plugin format with the ability to focus on details of that format. And, of the various different formats, VST2 is (unfortunately) de-facto the most “effort-effective” one to support if I wanted to make that tradeoff. If we do decide as the Rust DSP group to drop support for the higher-level VST2 crate, I’ll probably just fork it and maintain it myself in case I want to make that tradeoff in the future.

(I’d actually like to have this option for ALL of the different plugin formats, but realistically the only ones that affect me are VST2, VST3, and LV2.)

All that having been said, separating the crate into vst and vst-sys is something I can get behind regardless, so I’m on board to help work on that if needed.

Enough pessimism, here’s support for the idea:

Okay, putting the pragmatic side of things aside and speaking idealistically though, I agree with the sentiment of “screw Steinberg, VST2 isn’t even that great anyway, let’s garner support for the other plugin types and break the monopoly” – the audio community desperately needs more choices for plugin formats, and I personally don’t want to work with C++, so I think this is an awesome initiative.

I’m all on-board for the idea (minus the “throwing away support for targeting a single plugin” part).

Some thoughts/ideas:

• How do we decide which plugin formats to target? (I’m going to make another post for this)
• What happens if we decide we want to target a new one that we didn’t initially want to support? (IE is it easy to abstract it far enough that adding new plugin formats is easy?)
• For the set of plugin formats that we decide to support, I’d still like to have separate crates (not just modules inside the format-independent crate) of the x-sys format.
• We definitely need to figure out the “project management” aspect of this group pretty soon to tackle something of this size.

One specific response on this topic:

Even though I’m very unhappy with Steinberg about how they’ve handled VST2, VST2 is nothing special and we don’t lose much if we deprecate it. Honestly, it’d be better to do so sooner rather than later and move to something API-independent. We certainly shouldn’t be championing it as our standard API.

We lose a focused effort towards Rust support of the largest plugin format by market share. It sucks that Steinberg is grossly mismanaging the standard, but it is what it is. To be blunt myself, we don’t really move too quickly with one plugin format, let alone 4+ at the same time.

[crsaracco]

Okay, now that that’s out of the way…

Plugin formats that I think are worth supporting:

• VST2: We pretty much have this done already, sans GUI. Biggest market share. Etc.
• VST3: I’m going to guess this is where most people that create VST2 plugins are going to go, if they want to be in compliance with Steinberg’s license. NOTE: I don’t like the GPL on this, though, because that would necessitate you open-sourcing your entire plugin.
• LV2: We need to support this. It’s an open standard, not in the grips of some company. Also, as a Linux user, it would be nice to have this either way.
• AUv2: For reasons posted elsewhere in this thread.
• AAX: ProTools is pretty important. I agree that we should at least keep this in mind, but not necessarily support it day-one.
• AUv3: For reasons posted elsewhere in this thread. Should probably keep it in mind, but not necessarily support it day-one.

[Prokopyl]

It would be cool to see if an open plugin API could be extracted from the project similar to LV2, or maybe taking LV2 and making it cross-platform (I know it’s an idealistic pie-in-the-sky dream but it’s something I’m keeping in the back of my mind).

Actually, LV2 is already cross-platform (despite its name)! In fact, with the core specification being a single, small C header file (like all other specs actually), it is completely platform-agnostic, and I’m pretty sure it can even be made to work in #![no_std] environments.

The main difference between LV2 and other plugin APIs is that it is designed to be completely extensible, i.e. you only have to support the core spec to support LV2, and everything else comes from additional specifications, which can be the official ones or not, so there is no set of features LV2 does or does not support.

Both hosts and plugins are free to only implement a tiny subset of the LV2 specifications (either because they don’t support the rest or because they don’t need it), and can be used interchangeably with full-featured ones.

I think this kind of architecture would be interesting to use for an API-independent Rust audio plugin interface.
First, because it plays really well with the whole crates.io ecosystem: if you want to use or implement an extra spec for cool new features, just put it in your Cargo.toml and you’re done.
Second, because it allows plugin developers to reason in terms of features (they either need or can optionally use), and not in terms of what audio API backend they can or cannot target. In this configuration, backends can be seen as just bundled-in thin hosts that offer to the plugin the features they implement, depending of the audio API they’re implemented onto.

This way, we have none of the two issues @crsaracco talked about: no plugin-format-specific logic (although we have feature-specific logic instead), and no “lowest common denominator” (plugin authors implicitly opt-out of some backends if they require features some APIs don’t support).

This has some downsides of course: it makes the whole API a bit harder to design since pretty much everything may come from external code bases we can’t control, and it adds the overhead of feature negotiation and resolution at plugin instantiation time.

However, from my small rustic LV2 experiments I can say that extremely modular APIs like this aren’t bad at all to use, and from my heavy usage of LV2 plugins I can’t say the extra instantiation time really is an issue at all.
LV2 handles this by passing an array of URIs and their associated function pointers for the spec they implement, which plugins can search through to find what they need, and they can simply refuse to instantiate if they don’t find a feature they really require (some features may be optional for a plugin to work).

(Mandatory disclaimer: LV2 is the only audio API I really know about, so I may be quite a bit biased, take my opinions with a grain of salt!)

[Janonard]

That’s pretty much what I wanted to write too!
Although, I think that integrating LV2 into such a library will be hard to impossible because of it’s way of storing plugin information:

A LV2 plugin stores no information like port-count and type in the shared object, instead it stores such data in configuration files that are passed by host before loading the shared object. VST on the other hand stores such in information in the shared object, there are special methods that return them.

However, I don’t think that integrating LV2 into such a library is even desirable, because it does not try to be “VST, but from my company”. It has many features (like Atoms, which are awesome!) that can’t be integrated into the design of other standards. Therefore, many things that make LV2 good have to be cut.

[Prokopyl]

A LV2 plugin stores no information like port-count and type in the shared object, instead it stores such data in configuration files that are passed by host before loading the shared object. VST on the other hand stores such in information in the shared object, there are special methods that return them.

To be fair, this is the only part of LV2 that grinds my gears a bit, as far as the Rustic LV2 implementation is concerned.
Having all of the plugin’s information/metadata stored in a separate text file is awesome for hosts: this makes the plugin discovery blazingly fast™, as they don’t have to load each and every shared library just to extract the metadata.
So not only the host can quickly list all of the options and features of the plugin to the end-user, but it can also easily determine if a given plugin cannot be instantiated because the host lacks features the plugin requires. (Ardour, for instance, does it by graying out the plugin in its plugin browser).

The story is a bit different on the plugin side however. They have to distribute some additional .ttl files alongside the plugin to describe it (example here), and if the plugin’s description file has mistakes in it (such as wrong indexes or types for ports), this leads straight to undefined behavior, whatever safety mechanisms may be implemented on the Rust-side.

While this makes total sense in C, the fact that Rust’s type-system is much more expressive (which is something I use and abuse in my prototype) tends to duplicate that information, as it is present both in source and in metadata.

For this reason, I thought about making the Rust plugin declarations (ports, features, etc.) the source of truth, and generate the metadata files from it (should it be done using a build.rs script, a custom cargo command, is something we can discuss further another time I think).
Using this design, it allows to keep the metadata for both LV2, VST and others in a single place, and each backend would be free to interpret it however they’d like, so I think in the end it wouldn’t be an issue.

However, I don’t think that integrating LV2 into such a library is even desirable, because it does not try to be “VST, but from my company”. It has many features (like Atoms, which are awesome!) that can’t be integrated into the design of other standards. Therefore, many things that make LV2 good have to be cut.

I think you’re getting my idea backwards: I had in mind that each plugin would “declare” what features it needs (just like an LV2 plugin does, actually), and if it requires features that the API target can’t support, then the plugin simply cannot be used on that API.
On that architecture, LV2 would not be cut off from its features, but instead be the “I-can-support-literally-everything” target.

The features also don’t have to map one-to-one in an high-level API. You take the example of LV2’s Atoms, which are a (pretty cool indeed!) extensible serialization format, used for event passing (for e.g. MIDI events, but also probably others), inter-thread/inter-process/inter-machine communication (plugin <-> GUI communication), and the plugin’s state serialization (to be saved in project files or presets).

To me this isn’t a feature in itself, but rather a way to build many other features on top of it (like the ones I presented above). And while it is very nice to have as-is in C, in Rust-land it just asks to be a serde backend, so everything can be build on top of it, whether or not you’ll be serializing it to LV2 Atoms in the end.

[m-hilgendorf]

One of the things we talked about a few weeks ago was using configuration files (toml/JSON, any structured schema really) to contain the plugin metadata anyway, which would be expanded as a part of the build process using macros.

Since we need to use a post build steps no matter what for VST3/AU, we can just move a translation step from the config file to whatever LV2 needs.

Also I’m strongly against using anything from the LV2 SDK that isn’t the API used by hosts. One of the things I really don’t want to see is a wrapper crate that makes design decisions about anything unrelated to interfacing with the host, except where absolutely necessary (like parameter/event abstractions). IPC/serialization is well outside that scope imo.

[Engid]

For this reason, I thought about making the Rust plugin declarations (ports, features, etc.) the source of truth, and generate the metadata files from it (should it be done using a build.rs script, a custom cargo command, is something we can discuss further another time I think).
Using this design, it allows to keep the metadata for both LV2, VST and others in a single place, and each backend would be free to interpret it however they’d like, so I think in the end it wouldn’t be an issue.

That is a neat idea, and feels better than maintaining a separate file in my opinion.

One of the things we talked about a few weeks ago was using configuration files (toml/JSON, any structured schema really) to contain the plugin metadata anyway, which would be expanded as a part of the build process using macros.

Do you think this metadata could be gleamed from the traits/impls in a way that Prokopyl is describing for Lv2?

[Prokopyl]

Also I’m strongly against using anything from the LV2 SDK that isn’t the API used by hosts. One of the things I really don’t want to see is a wrapper crate that makes design decisions about anything unrelated to interfacing with the host, except where absolutely necessary (like parameter/event abstractions). IPC/serialization is well outside that scope imo.

While I agree about the idea, IPC and serialization are features used by the host: serialization for saving the plugin’s state for project files and presets, and IPC+sezialization is needed for communication between the plugin and the UI.
Although I definitely agree those should not be exposed and only kept as implementation details for the LV2 backend (in favor of high-level stuff like serde for serialization, and something else for IPC).

One of the things we talked about a few weeks ago was using configuration files (toml/JSON, any structured schema really) to contain the plugin metadata anyway, which would be expanded as a part of the build process using macros.

It depends what kind of metadata we’re talking about?

If we’re talking about stuff like the number of ports and their type, I believe this should be contained in the source code, so it can be checked by the type system, so that invalid metadata cannot cause undefined behavior at runtime. (Stuff like InputPort<Audio>, Option<OutputPort<Midi>>, etc. are better declared in code imo.)

However if it is more like the label of the ports, their translations, how they are displayed in the host’s UI (i.e. stuff that does not affect the plugin directly), etc., then yeah, I believe those should be in a separate structured file, so we don’t have to have too many declarative attributes in the source code.

[m-hilgendorf]

While I agree about the idea, IPC and serialization are features used by the host: serialization for saving the plugin’s state for project files and presets,

AU/VST2/VST3/AAX all handle this with binary blobs, can you not do that with LV2? Do all LV2 require the same serialization primitives?

and IPC+sezialization is needed for communication between the plugin and the UI.

It’s not funneled through the host? And again, does it need to be done using the LV2 methods?

If we’re talking about stuff like the number of ports and their type, I believe this should be contained in the source code, so it can be checked by the type system

Using procedural macros we can be far more explicit with compile time error messages than just relying on the type system. The metadata needs to expand to code for the other APIs as well, it’s just a matter of controlling how it is expanded based on the build configuration.

Channel set configurations (I guess the analog would be an audio “port” in LV2) and parameters are a big reason why we want to do this, as dealing with it in code is extraordinarily messy when you start dealing with more than one API.

[PieterPenninckx]

Interesting discussion!

I especially like the idea of “per-feature” traits instead of “per-API” traits.

I would like to draw your attention to the rsynth crate, which already has an abstraction layer for two API’s: rust-vst and Jack.
rsynth contains some interesting techniques (like the VecStorage and the VecStorageMut structs), so I think it is worth having a look at it. I have written some comments in the source code about some of the design decisions, so you can build on the experience that was gained.

I would also like to mention that many of the design decisions were driven by the “higher level utilities” that are also in rsynth (like support for polyphony). I think there’s a danger with first building an API abstraction layer and only after that on top an ergonomics layer and only after that a plugin. The danger is that you risk to discover only late in the development process that the underlying layer is not usable in practice. The most important aspect – in my opinion – of software development methodologies like Agile is working with “vertical slices”, i.e.: working feature by feature instead of layer by layer. In that way, you know earlier in the development process whether something works or not. So for this reason, I propose to start working on the API abstraction layer in tandem with an “improved ergonomics layer” and also start using it for plugins from the very start.

[doomy]

I had in mind that each plugin would “declare” what features it needs (just like an LV2 plugin does, actually), and if it requires features that the API target can’t support, then the plugin simply cannot be used on that API.

I especially like the idea of “per-feature” traits instead of “per-API” traits.

I like this a lot too. However we would need to somehow make it clear that certain features are available only to certain backends (lv2, vst, etc).

If I want to impl SomeXFeature, how can I ascertain what backend supports that SomeXFeature?

[Prokopyl]

AU/VST2/VST3/AAX all handle this with binary blobs, can you not do that with LV2? Do all LV2 require the same serialization primitives?

Although using LV2 Atoms for serialization is preferred within the LV2 ecosystem (there are some arguments for it in the LV2 State spec, and the Patch spec provides a nice use-case allowing to query and update any serializable state from the UI), we can use whatever we’d like as a serialization format (which may or may not differ depending on the target API), including binary blobs, so no real restriction there.

But whatever serialization backend/format(s) we end up choosing, it does not change the fact that state serialization is needed for plugins to let hosts save their state (and is therefore not outside the scope for a plugin API).
We cannot just use a binary blob straight from the plugin’s struct (nor use something like bincode), because the state needs to be reusable across plugin builds, host machines and plugin versions. For instance, if I use v1.x (assuming semver) in my DAW’s project, and reopen the project two years later, the plugin should be able to load its state back even if it was updated to v1.y in the meantime.

However, I am strongly against exposing the serialization format to the plugin developers, and to the common/API-independent plugin interface even. If (for instance) the VST2 backend wants JSON, VST3 wants BSON, AU wants YML, or LV2 wants its Atoms, it’s their own business. The plugin author should just have to derive/implement Serialize and Deserialize and that’s it.

It’s not funneled through the host? And again, does it need to be done using the LV2 methods?

IPC in LV2 is indeed funneled through the host (in the form of messages), though whatever method LV2 chooses to use for its IPC should not affect the plugin author. But we should expose something to allow communication between the plugin and its UI, so I do not believe it is outside of the scope for an API-independent plugin interface either.

Now that I think about it, i think may not have been completely clear in my previous messages: I was not talking about doing stuff the LV2-specific way for the implementation, but talking about which kind of features we should expose to the plugin authors to help them write their plugins. Sorry if that was unclear.

Using procedural macros we can be far more explicit with compile time error messages than just relying on the type system. The metadata needs to expand to code for the other APIs as well, it’s just a matter of controlling how it is expanded based on the build configuration.

Channel set configurations (I guess the analog would be an audio “port” in LV2) and parameters are a big reason why we want to do this, as dealing with it in code is extraordinarily messy when you start dealing with more than one API.

Oh, of course there would be quite a bit of procedural macros running around, but in terms of usability and readability I personally despise codegen that generates structs that I then have to use in my code, as it makes the contents of that struct to not only me (as a developer/user of the API), but also to my tooling and IDEs that often have no idea what type of structs I’m even interacting with (error-chain vs. failure is a good example of this issue).

For the specific case of channel set configurations and (automatable) parameters (the LV2 analogs for those would indeed be audio ports and control ports, respectively), I’m not sure this is an issue to express them in code?
For instance, in the lv2 crate this is how things are done right now (this is a simplified example for a stereo amplifier with a gain parameter):

#[derive(Lv2Ports)] // This custom derive does all of the heavy lifting
struct MyAmpPorts {
    inputLeft: InputPort<Audio>, // this derefs to &[f32]
    inputRight: Option<InputPort<Audio>>, // this port can be disconnected (i.e. no buffer behind it), if we want

    outputLeft: OutputPort<Audio>, // this derefs to &mut [f32]
    outputRight: OutputPort<Audio>,

    gain: InputPort<Control> // this derefs to &f32
}

impl Plugin for MyAmp {
    type Ports = MyAmpPorts; // If you want multiple port/channel configurations (like mono vs. stereo),
                             // you can make your plugin struct generic over the ports struct

    fn run(&mut self, ports: &Self::Ports) { // Ports are passed here
        let some_audio_data = ports.inputLeft; // Data is accessible directly
        // (Process the audio...)
    }
}

But maybe this kind of design has some limitations I’m not seeing?

I like this a lot too. However we would need to somehow make it clear that certain features are available only to certain backends (lv2, vst, etc).

If I want to impl SomeXFeature , how can I ascertain what backend supports that SomeXFeature ?

The first thing to make it clear would be obviously be documentation for the most common plugin formats (i.e. the ones this group decides to maintain), but that alone would not be sufficient I think.

I’m assuming the plugin author would need to opt into the backends they want to support, so I think the ideal way would be to get a hard build error if some backend doesn’t support a required feature (optional ones will just be ignored), so it can be easily caught by CI.

In my mind (suppose we call this plugin interface fooapi), the developer would only develop against fooapi, but would also pull some backend crates for the audio APIs they’d want to target: just like serde has serde-json and serde-yaml for instance, there would be a fooapi-vst (built on top of the rust-vst crate), a fooapi-lv2 (built on the lv2 crate), and so on.

Obviously these would need to be enabled and disabled depending on the OSes the various plugin formats can work on (for instance I believe you can’t build AU plugins in OSes other than Apple’s?), but I think a few simple cfg attributes should to the trick here.

Also, just a nitpick: I think the impl SomeFeature {} syntax would be used for plugin features (not for host features), so by the very nature of both plugin features and traits they would be automatically optional (just like Vec implements IntoIter, but you don’t have to iterate over it to use a Vec).
Host features would be instead something resolved at the plugin’s instantiation time (like MIDI support for instance) and can be either required or optional, so they can be used in the plugin’s process()/run() method for instance.