Unable to reproduce WER results with streaming models

[ben-w-conrad]

I'm currently working on adding Moonshine Streaming models into this Speech-to-text benchmark repo, however I'm getting different results than what is reported in the Huggingface readme.

Using ModelArch.TINY_STREAMING, I'm getting 11.8% WER on LibriSpeech test-clean which is much higher than the reported 4.5%. My code is essentially identical to transcribe_with_streaming from the example code, with chunk_duration=0.1 and leaving the update_interval at its default of 0.5. I'm just wondering if these numbers are to expected when doing streaming inference or if there's some options that I should be using to match the reported numbers.

Similarly when I measure the average per-word latency I'm getting 760ms with the same model configuration. I'm measuring latency differently than how this repo's benchmark does so I'm just also wondering if the number I'm getting any makes sense.

[petewarden]

Thanks for the heads up! I've pinged @keveman and @evmaki since they know this area better than I do.

[petewarden]

@keveman shared:

The repro script is here: https://github.com/huggingface/open_asr_leaderboard/blob/main/moonshine/run_eval.py

I think you're probably looking for something that doesn't use HF Transformers though. The closest in-framework example we have uses the Fleurs dataset, and is aimed more at multilingual comparisons: https://github.com/moonshine-ai/moonshine/blob/main/scripts/eval-model-accuracy.py

I'm happy to work with you on this, it will be good to get some more accuracy evaluation scripts in the repo.

[kylehowells]

@ben-w-conrad I've been benchmarking the WER on some synthetic text to speech generated audio and some real audio from random files I have on my computer and am seeing very similar 17% WER for the tiny models on my test files and 7.5% on the medium model.

However, playing around with the batching and segmentation I've been able to get it up to an 8% error rate for the tiny model on my test files. Going to play around with it a bit more and test more thoroughly to see if it holds reliably, but in the mean time if you aren't seeing results as accurate as you need, it might be worth playing around with increasing the buffer either side before and after the audio you are feeding into the model it might help boost the accuracy for you.

[petewarden]

One issue to watch out for is the VAD. Since diarization is a separate metric, and a lot of models don't include it by default, I tend to disable voice activity detection for pure word error rate calculations. You can see an example of how to disable it by setting the vad_threshold to zero here: https://github.com/moonshine-ai/moonshine/blob/main/scripts/eval-model-accuracy.py#L138

[petewarden]

This is an important question, and I've put some time into resolving this. I've put together a script to help reproduce our word error rates for the streaming models, and added a section to the documentation covering the methodology and results. I've also added a note to the comparison table in the models section clarifying that the WER numbers are from OpenASR.

Since we now have a reproducible source for the WERs in the documentation and I hope the note helps make things clearer, I'm closing this as fixed. Please do reopen if this doesn't answer what you wanted to know.