Use custom exponential backoff: Wait longer to avoid rate-limit. #1757
Description
/kind feature
Currently, when Reconcile returns an error, the controller-runtime retries immediately.
If the error does not disappear, then Reconcile gets called again and again in short time.
This can lead to hitting rate-limits of third party services.
Example: Reconcile() of a hcloudmachine will contact the Hcloud API of Hetzner.
We should use a custom (instead of the default) backoff. Example:
import (
"time"
"k8s.io/client-go/util/workqueue"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/controller"
)
func SetupWithManager(mgr ctrl.Manager, r *MyReconciler) error {
return ctrl.NewControllerManagedBy(mgr).
For(&myv1.MyResource{}).
WithOptions(controller.Options{
// Exponential backoff between requeues for errors:
RateLimiter: workqueue.NewItemExponentialFailureRateLimiter(
5*time.Second, // base delay
5*time.Minute, // max delay
),
}).
Complete(r)
}