Multi-stream instrument rework

[messmerd]

Enhancement Summary

The design of multi-stream instruments prevents them from working with the pin connector (#7459) and complicates the plugin API, which is a big problem.

To explain briefly, "multi-stream" instruments are those that receive NotePlayHandles with buffers that they use to render an individual note to. Most native LMMS plugins are multi-stream plugins. So if 10 notes are playing at once, there are 10 NotePlayHandles - each with a buffer - and each note's audio is rendered to their respective buffer during the Instrument::playNote() method.

The purpose of all these buffers is to allow ADSR to be applied to each individual note, which is why only multi-streamed instruments have an "Envelope, filter & LFO" tab. If the audio for all notes was mixed together into a single buffer like it is for single-streamed instruments, the host would not be able to apply ADSR.

The problem with the current multi-streamed plugin API is that no other audio plugin API chooses to implement support for ADSR this way. It requires allocating a buffer for each note and creates a weird discrepancy with how the other plugin types function. This discrepancy complicates how multi-streamed plugins are used on both the plugin and host sides.

Implementation Details / Mockup

I believe today's multi-streamed instruments could use a single buffer like all other plugins, but the host should communicate the ADSR information to them so they can apply ADSR themselves as they mix each note into the single buffer. That way there is no craziness of buffers being allocated and deallocated all of the time and a lot of the weirdness in the plugin API can be removed, but the same ADSR functionality should remain.

In other words, instead of having the plugin pass each individual note's buffer back to the host for the host to apply ADSR, the host should pass the ADSR settings to the plugin for the plugin to apply ADSR.

I haven't seriously looked into implementing this yet, so there may be complications, but I'm sure we can solve the problem somehow since no other plugin API (VST, CLAP, etc.) does what we do with a buffer for each note.

[regulus79]

This sounds quite appealing to me. Passing envelope/lfo info to the plugin before it generates the audio would make it much easier to implement pitch envelopes/lfos, which is something I've wanted to do for a while. Trying to implement pitch changes after the audio has been generated (like how vol/filter is applied by the host on the output audio) is not really possible without using weird resampling tricks.

I could envision that there would be a lot of code duplication if each plugin has to reimplement env/lfo handling, unless we want to have a helper function for it. That could be beneficial if we add more custom env/lfo types, if plugins want to have some of their parameters modulated per-note. I'm not sure.

[messmerd]

I could envision that there would be a lot of code duplication if each plugin has to reimplement env/lfo handling, unless we want to have a helper function for it.

Yeah, avoiding code duplication is one of the hurdles here.

Also, the current implementation has no real limitation on the number of voices (as far as I know) since data is dynamically allocated (bad!) for each new voice and assigned to the NotePlayHandle's m_pluginData member. But we want to remove all dynamic allocations here, so that may mean providing some sort of memory pool (maybe a thread_local std::pmr::unsynchronized_pool_resource on each audio thread) for plugins to allocate voice data from.

And we also should try to maintain the same level of parallelism when processing voices, otherwise we'd have a significant loss of performance. Currently, if 4 notes are playing, they may all be rendered concurrently in Instrument::playNote(), which is really nice. I'd like to create something similar to CLAP's thread-pool extension to handle this. We could hopefully hook it into our current AudioEngineWorkerThread system, and if not, maybe we should consider using something like libfork. It would work like this: The plugin schedules N jobs for N active voices, then waits until each job completes (and while waiting, the calling thread should also render voices itself to avoid waste). Each voice would use something like CLAP's scratch-memory extension to obtain a temporary buffer for writing to. After all notes have been rendered, the calling thread would call a utility function that mixes all the voice buffers together while applying ADSR. Then the scratch buffers could be returned to where they came from.

That could be beneficial if we add more custom env/lfo types, if plugins want to have some of their parameters modulated per-note. I'm not sure.

This problem also intersects with the goal of adding a single event queue for all note events (MIDI or native LMMS note events), parameter updates, etc. For now I think passing a single plugin-wide ADSR settings object to the plugin is fine, but if we add ADSR on a per-note basis (or any other per-note parameter automations), we should consider switching to an event queue. I believe NotePlayHandles currently handles per-note volume and panning, but if we switch over to an event queue, those would become parameter events and the rest of the NotePlayHandle functionality would become note events.