concurrent model execution

[leeseungchae]

Hello,
I'm Steve Lee, an AI developer.
The performance improvement of Faster-Whisper is quite surprising.
However, while it's very fast when running a single script, the speed becomes significantly slower
when running multiple requests at the same time. Is there any solution for this concurrency issue?

Additionally, despite adjusting the num_workers and threads,
there's no performance improvement. What could be the reason for this?

[guillaumekln]

Hi,

Can you post a code snippet demonstrating your usage and the performance issue?

[leeseungchae]

For example, the speed of the result from a single request to a single container is very different from the speed of the results from three requests to three separate containers. I would like to show you the logs with the above example, but it is difficult due to the project structure. So, I will show you the speed results of the implementation with multiprocessing.

[guillaumekln]

You can try using multiple threads instead of multiple processes. In this case num_workers should make a difference, especially when the model is running on CPU.

[vedanshuFonada]

Same problem. I think we need continuous batching.