Rust Audio

Using the rimd crate in plugins to handle MIDI messages?

Despite our rather packed and erratic schedules, @Janonard and I have advanced on the development of the lv2 quite a bit now, and we have almost all of the prerequisites in place to implement MIDI messages handling.

Since MIDI Messages are such an inevitable standard in audio, every plugin format will end up having to deal with it at some point, and therefore I was looking at the possibility of using the rimd crate as a public dependency of both the lv2 and vst crates, instead of re-implementing it each time.
This would allow better code sharing across the RustAudio ecosystem, and it will probably help quite a bit when we’ll end up making our API-independent plugin framework (since MIDI messages generated by all backends would be the same struct).

There are a few issues with rimd right now that would prevent us from, like the fact that it seems quite tied to the SMF format (which we likely won’t use), and that it is not realtime-safe for now (right now MIDI messages in rimd are backed by a Vec<u8> when I believe they could use a [u8; 3] instead).

These aren’t too hard to fix, but the real issue is the fact that the rimd crate seems to be unmaintained right now. If there is interest in using it for plugin APIs I would be up to help fix those issues as well as clear out the current backlog of issues and PRs, however I would also like to find a co-maintainer around here, just to help with the bus factor. :slightly_smiling_face:

What do you all think about this idea?

That’s good to hear :slight_smile:

That is true for “normal” midi messages, but not for SysEx (system exclusive) messages, which are “variable-length”.

I’m starting to use rimd as a “backend” in rsynth (the API-independent plugin framework) for reading/writing SMF files so that rsynth can be used for offline processing (not inside a DAW) as well, mainly to make it easier to test the plugin. I chose for rimd because at the moment, it’s the best tool for what I need in rsynth, but I noticed as well that there are some things that can be improved.

Guess what? I’m re-implementing it in rsynth. On the other hand, it’s not that hard:

#[derive(Clone, Copy)]
pub struct RawMidiEvent {
    data: [u8; 3],
}

and

pub struct SysExEvent<'a> {
    data: &'a [u8],
}

(source)

It’s quite straightforward to map between this and the events defined by the jack crate and the vst crate.

I think the value of rimd does not lie in defining data types for raw midi events, but in dealing with SMF files and all the stuff you can have in SMF (like meta-events, copyright notice, …). If you want to parse “just” the midi-events that can be transmitted over the wire, wmidi does a better job, in my eyes.

1 Like

I guess the main point in having an external MIDI message crate is that one could create some sort of MIDI processing library with it, which, in effect, can be used by multiple plugins with different standards. I really dig that idea and I also think that this crate may also contain means to read/write SMF files, which makes rimd a good candidate to be this crate.

However, I don’t quite like the way it represents/reads/stores MIDI messages: Internally, MIDI messages should be enum variants, not just a slice of bytes, since this makes handling messages in code easier and faster. Also, there should be a way to read and write MIDI messages directly from and to a slice of bytes, since this is the way it is done with LV2 atoms.

I’m really looking forward to that, but I probably won’t have the time to maintain it. Best of luck with that! :+1:

1 Like

Right, I forgot about this one, my bad!

It’s not hard, but I think it is still a little cumbersome for to have to map, say, jack Midi to raw bytes and then tovst Midi if they want some interoperability between the two.

And while re-implementing those structs is indeed easy, if those are user-facing then you’d have to also add all of the accessors/enum variants that cover the MIDI standard. Which isn’t too hard in itself, just cumbersome, and I feel the job has been done many times already. :slightly_smiling_face:

I did not see this one when I looked up MIDI crates a bit ago, thanks!

Indeed it seems more suited to be used as a common base, and as @Janonard points out I also think it’s nicer to have the user API to be an enum instead of just raw bytes, I wouldn’t mind trying it out personally.

Perhaps we could also add wmidi into the RustAudio organization? If such a dependency is to be made with it, I’d rather have it into the org as well. :slightly_smiling_face:

It’s nicer indeed: it saves about four lines of code in the application/plugin: one in Cargo.toml, one to use extern crate wmidi; one to use wmidi::* (or similar) and one to do the actual conversion. (Edit: + error handling, of course, so it will be more than four lines.)

It has its drawbacks, however:

  • It’s an extra dependency if you don’t need it.
  • It creates extra trouble if you want to use a different MIDI library or a different version of the MIDI library
  • It renders some use cases impossible, such as counting malformed midi messages from the host.

Also, from the point of view of an API-independent plugin framework it’s not ideal if not all underlying libraries (vst, jack, lv2, …) use the same midi library … and the same major version. Especially the version issue troubles me: you would either have to synchronize the release of the libraries, or the API-independent plugin framework would have to map enums from one version of the library to another, which is something I would like to avoid.

Putting it all together, I think the disadvantages outweigh the advantages and I would simply kindly suggest the plugin authors to do the conversion in the plugin.

When looking at other standards, I see your points @PieterPenninckx, but from a LV2 perspective, I have to disagree with some of them:

While developing rust-lv2, we try to have exactly one crate per specification. Since MIDI-IO has it’s own specification in LV2, it would also have it’s own crate. If the user isn’t happy with it, they could simply develop their own MIDI-IO crate and circumvent the dependency to wmidi (or something like it). Since no other specification directly depends on MIDI-IO, all of the other stock LV2 crates could be used too!

MIDI-IO is quite optional in LV2: Its “only” a special case of a feature that’s in theory optional; Unlike VST, where it is a core part of the specification. If the user doesn’t need MIDI, they simply don’t use MIDI and have no extra dependencies.

I think the version issue is the smallest trouble of them all and also, at last, independently simple for all standards; Thanks to cargo and Semantic Versioning. I know, it’s going to be obsolete, but please take a look at the futures crate: It’s one crate that is used by many other crates to create a big, interoperable ecosystem; It’s exactly what we’re thinking of too! Maybe the integration of the external MIDI crate into the different standard implementation is not possible or desirable, but if there is one thing we don’t have to worry about, it’s version control!

Maybe it’s more complicated for other standards, but I think that at least rust-lv2 should use an external MIDI message crate. It would at least leave space for some interoperability.

Let me sketch some scenario’s that illustrate what the impact of a wmidi dependency in rust-lv2-midi (lacking a specific name of the crate, let’s call it like that in this discussion) is on an API-independent plugin framework (e.g. rsynth).

Scenario 1: rust-lv2-midi supports midi using wmidi; the other API’s use just raw bytes.
An API-independent plugin framework (e.g. rsynth) would then have the choice between the following options:
Option a) In the traits defined by rsynth do not use raw bytes to describe midi, but the wmidi-defined data type. Because MIDI is not optional for other API’s (e.g. vst), this would add wmidi as a mandatory dependency, also for plugins that do not use MIDI.

Option b) Keep the raw bytes to describe midi in the traits defined by rsynth. This would mean that for lv2 support, rsynth would have to reconstruct the raw bytes from the wmidi-defined data type. None of the fun, all of the problems.

Option c) Fork rust-lv2-midi for the sake of removing a dependency as you suggest. I’d rather avoid that.

Scenario 2: rust-lv2-midiuses wmidi, some other API uses another crate, e.g. vst-rs uses rimd.
This makes it worse, because rsynth would have to pick wmidi or rimd and either for lv2 support or for vst support , It would have to convert the rimd data structure to wmidi data structure or vica versa. Overhead.

Scenario 3: rust-lv2 uses wmidi and another API uses a compatible version of wmidi, e.g. rust-lv2 uses wimidi 2.1.1 and vst uses wmidi 2.2.0.
Thanks to cargo and semantic versioning, this does indeed not add extra problems, so it’s similar to Scnario 1.

Scenario 4: rust-lv2 uses wmidi and another API uses an incompatible version of wmidi, e.g. rust-lv2 uses wmidi 2.2.1 and vst uses wmidi 3.0.1.
In this scenario, cargo cannot help us and we end up in the same situation as Scenario 2.

Does this sound right to you? Maybe you see other options how rsynth (or another API-independent plugin framework) could handle these scenario’s or you have ideas to avoid ending up in these?

My “vision” for the ecosystem is as follows.
At the bottom layer, there are all the API-crates, e.g. rust-vst, rust-lv2 etc. Then on top of that, you can build an API-independent “framework”, e.g. rsynth. On top of that, you can either build a convenience framework that does midi parsinng etc. for the plugin author, or you can build the plugin directly and “mix and match” (combine) crates such as wmidi as needed.

I would put the MIDI parsing in the “convenience layer” or in the plugin.

If you still want wmidi support in rust-lv2-midi, then how about putting the wmidi dependency in rust-lv2-midi behind a cargo feature that is enabled by default? Then plugins and framework who do not want to use this particular version of wmidi or who want to do their own error handling, have the option to do so without forking rust-lv2-rs.

Can you let me know what you think about this idea?

1 Like

Ok, you got me. Having a common MIDI crate only works if everyone uses it and the only true common ground between all libraries are raw MIDI messages, represented as byte slices. I’m with you that at default, plugin libraries should provide all read or written messages as byte slices. This solution eliminates all scenarios and provides maximum interoperability.

However, I hope that you agree with me that having a common, interoperable and enum-based MIDI message crate would be nice, wouldn’t it? Maybe we could settle choosing a common crate/version and making it “optional, but recommended”, for example by placing it inside a feature and enabling that feature at default. This would still introduce a high-level ecosystem while providing the option to opt-out and deal with the raw messages instead.

Is this a better deal?

1 Like

Yes, I like that. :+1:

I was going to suggest just that, so of course I agree there too. :slightly_smiling_face: