Hi,
I've been trying to use pyrate-limiter to limit calls to the OpenAI API according to two rate-limited resources: tokens per minute, and requests per minute.
As far as I can tell, after reading the documentation, I need two separate limiters. Something like:
self._rpm_limiter.try_acquire("api_calls", weight=1)
self._tpm_limiter.try_acquire("tokens", weight=token_count)
However, I think acquiring each resource sequentially can create a race condition with other threads: We may acquire the 1st resource and use it up, and then block when acquiring the 2nd resource.
It would be great if it was possible to either:
- Acquire different resources at the same time;
- Or release a resource early after a
try_acquire: in this case, we would acquire the 1st resource, and if we can't acquire the 2nd resource we would then release the 1st resource and retry;
Is there a way to do this?