Whisper.cpp backend: support audio tagging (Whisper-AT or parallel classifier) for music / non-speech annotation

[jorboyd]

Summary

whisper_at is currently a config option that only takes effect on the Python backend (it swaps whisper for whisper-at in the CLI invocation). The cpp backend has no equivalent — meaning users who choose cpp for speed and offline operation lose the ability to detect and label music, applause, ambient sound, and other AudioSet events.

Motivation

I'm building a film shotlist annotation corpus where one of the per-scene dimensions is score presence / pace / register. Whisper-AT's AudioSet labels (Music, Orchestra, Synthesizer, Acoustic guitar, Singing, etc.) would let me detect score boundaries and classify music character automatically — currently I'd have to switch from cpp to Python to get this, sacrificing the GPU-accelerated speed advantage that makes corpus-scale work practical.

This matters for any workflow analyzing film, sports doc, or branded video at scale.

Suggested approaches

1. Easiest: invoke a separate audio classifier in parallel with whisper-cli. Options include PANNs, YamNet, or a CLAP-based classifier. Outputs would populate the existing audio_tags field already in the AudioResult type.

2. More integrated: if whisper.cpp adds AudioSet tagging in a future release, route to it. (Doesn't exist today.)

3. Best fidelity: integrate with a small audio-understanding LLM (Audio-Flamingo, Pengi, GAMA) for natural-language music descriptions. Larger lift; might be a separate "backend".

Note

This is a feature request rather than a bug — current behavior (cpp returns empty audio_tags) is reasonable as a baseline. Filing because the related bug report references this gap as the workaround constraint.

[jordanrendric]

Thanks for the detailed motivation — the score-presence dimension in a shotlist corpus is exactly the kind of workflow where cpp's speed and offline operation matter, and losing that to fall back to Python isn't a good tradeoff.

I want to do this, but it's a larger lift than the other fixes you've sent. Some open questions before I commit to a shape:

1. Approach. Of your three options, (1) parallel classifier is the most realistic. PANNs and YamNet are well-supported, audioset-trained, and have small models. CLAP is also good but cross-modal — overkill for tag emission. Option (2) is gated on upstream whisper.cpp work that may never land. Option (3) is a separate backend, not a cpp-backend enhancement.

So: parallel AudioSet classifier alongside whisper-cli, both writing into AudioResult.

2. Implementation surface.

• Run classifier in parallel with whisper-cli inside transcribeWithWhisperCpp (they're independent — both consume the wav)
• Populate audio_tags with the resulting events. Existing type already supports this
• Auto-download the model on first use (~50 MB for PANNs CNN14), mirroring the existing whisper model pattern with SHA-256 verification
• Make it gated on a config flag (whisper_at is the natural extension, even though it's a name lie at that point — or introduce cpp_audio_tagging: boolean)

3. Runtime requirement. PANNs / YamNet need Python + onnxruntime. That's a heavier dependency than the cpp backend currently has (just whisper-cli + ffmpeg). Worth considering whether this defeats the purpose of "cpp = no Python" for some users. Possible answers:

• Ship as an additional backend (local-with-tagging) rather than baked into cpp
• Use a compiled inference path (ONNX → tract / candle / wasm) to keep zero-Python promise
• Document the dependency clearly and let users opt in

The third is the most honest tradeoff but it does muddy "pure cpp."

I'm parking this issue as "in flight, will implement after #46/#45 land" — your annotation corpus work clearly benefits, and I want to design the dependency story carefully. If you have strong preferences on (2) the gating flag and (3) the runtime tradeoff, weigh in here so I'm not guessing.