update FilesExt retry logic

[yuanjieding-db]

What changes are proposed in this pull request?

WHAT

• Extending retry function with a new parameter max_attempt to allow client to retry and fail after certain amount
• Remove 500 from FilesExt retry status code
• Add new config to set the retry attempts for FilesExt
• Update the retry logic of FilesExt to fail after certain attempts.

WHY

• 500 errors shouldn't be retried
• The FilesExt should always prioritize fallback over retry to avoid regression

How is this tested?

Unit tests were updated to reflect the change.

[parthban-db]

Do we have a better error to represent this error? TimeoutError feels a bit odd for this case, as the function is not actually timed out.

[yuanjieding-db]

Ideally we should use a custom type RetryError, with TimeoutError and MaxRetryExceededError as it's derived types, so that the user can catch the RetryError if they don't care why the retry exhausted, while keeping the information.
However since we have been using built-in TimeoutError and users may already be depending on this behavior, it is risky to change it to a different Error.
If we were to introduce a new type of error for max retry exceeded scenario, it would be more difficult for the upper layer to handle the retry error: it needs to catch both Errors manually.

I don't see a better solution here, unless we can rewrite the retry logic completely, or make FilesExt using a different retry library.

[parthban-db]

So, you finally changed this to RuntimeError, right? I think it is fine to throw that, as I don't want any other function to depend on that error.

[github-actions]

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/sdk-py

Inputs:

• PR number: 1211
• Commit SHA: 04305fbf1709fd13f5652c8249683b15c5d945c8

Checks will be approved automatically on success.