APIM returns 500 Error when one of backend azure openai returns 429 at the first time.

[nowjean]

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

429 Retry test

Any log messages given by the failure

APIM returns 500 Error when one of backend azure openai returns 429 at the first time.
Second try is ok to retry another azure openai backend.

Expected/desired behavior

There is no 500 error when one of azure openai returns 429.

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Versions

Using Azure portal APIM policy with apim-policy.xml

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

[vandriot]

Hello,

I can confirm that I encountered the same issue as you.

I was able to reproduce the issue by using the Trace feature in APIM to identify the source of the problem:

{
    "messages": [
        {
            "message": "Expression evaluation failed.",
            "expression": "context.Response != null && (context.Response.StatusCode == 401 || context.Response.StatusCode == 429 || context.Response.StatusCode >= 500) && (int.Parse((string)context.Variables[\"remainingBackends\"])) > 0",
            "details": "Unable to cast object of type 'System.Int32' to type 'System.String'."
        },
        "Expression evaluation failed. Unable to cast object of type 'System.Int32' to type 'System.String'.",
        "Unable to cast object of type 'System.Int32' to type 'System.String'."
    ]
}

The issue originates from this line in the policy:

openai-apim-lb/infra/modules/apim/policies/api_policy.xml

Line 80 in 039195e

<retry condition="@(context.Response != null && (context.Response.StatusCode == 401 || context.Response.StatusCode == 429 || context.Response.StatusCode >= 500) && (int.Parse((string)context.Variables["remainingBackends"])) > 0)" count="50" interval="0">

From what the error is telling us, the variable remainingBackends is actually not a string but an integer. It's surprising considering how it is initialized but why not 😄

To solve the issue, replace the problematic line by:
<retry condition="@(context.Response != null && (context.Response.StatusCode == 401 || context.Response.StatusCode == 429 || context.Response.StatusCode >= 500) && ((int)context.Variables["remainingBackends"]) > 0)" count="50" interval="0">

To be clear, we are just updating the last condition
From: (int.Parse((string)context.Variables[\"remainingBackends\"])
To: ((int)context.Variables["remainingBackends"])

This solution worked for me—I no longer receive HTTP 500 errors when handling a 429 or 5xx response from the backend.

Let me know if it resolves the issue for you as well, and I’ll submit a PR to implement this fix in the codebase permanently.

[vandriot]

I just took a closer look to the open PR and actually there is the PR #18 which will also solve your issue.
It's using a different way but achiving the same result.

[nowjean]

Victor (@vandriot) Awsome! It works fine to me 💯 Thanks a lot!