What's the best way to proceed without caching if an exception is hit in a Cache Annotation

[scprek]

Cache Interceptor Usage for proceeding when exception occurs

Unanswered in Discord https://discord.com/channels/1121511613250412714/1174821270785900666/1174821270785900666

We hit a Production issue during an AWS elasticache cluster maintenance window where micronaut redis annotated operations would start crashing the requests when we really just wanted it to continue on and hit the database again. The only way we found to support our needs was to subclass the CacheInterceptor and toss a try/catch saying context.proceed(). This seems not ideal since it's global? but maybe there's some way to limit it's use?

Now ideally, maintainence window finishes at a time we are ok with (this is true today) but for some reason some app
instances never fully recovered and don't know why since we cannot recreate the issue of not reconnecting after the maintainence is done.

We CAN re-create the issue during maintenance of the cluster tho consistently and the app instance just constantly fails requests due to the cache exception.

We can leave the Redis part out of this discussion, but included for background.
I'm not expert on Redis clusters and redis-lettuce but it did seem like if we only have 1 instance of our app and we do a failover the app is completely broken until resolved (failover complete). With multiple instances we see some subset stuck constantly throwing these errors and understand that cluster mode is maybe queueing those things up until the topology is correct?

Workaround

class CustomCacheInterceptor(
    cacheManager: CacheManager<Any>,  // Adjust the type parameter as needed
    errorHandler: CacheErrorHandler,
    asyncCacheErrorHandler: AsyncCacheErrorHandler,
    @Named(TaskExecutors.IO) ioExecutor: ExecutorService,
    beanContext: BeanContext
) : CacheInterceptor(cacheManager, errorHandler, asyncCacheErrorHandler, ioExecutor, beanContext) {

    /**
     * Allows Redis errors to continue
     */
    override fun intercept(context: MethodInvocationContext<Any, Any>): Any? {
        return try {
            super.intercept(context)
        } catch (e: Exception) {
            // Maybe add a metric
            // Handle the exception, e.g., log it and proceed without caching
            context.proceed()
        }
    }
}

Example Errors

Errors we hit in production

i.m.m.h.indicator.HealthResult Health indicator [redis(Primary)] reported exception: io.lettuce.core.RedisConnectionException: Unable to connect

io.lettuce.core.RedisException: java.io.IOException: Connection reset by peer
at i.l.core.internal.Exceptions.bubble(Exceptions.java:83)
at io.lettuce.core.internal.Futures.awaitOrCancel(Futures.java:250)

i.l.c.RedisConnectionException: Unable to connect
at i.l.c.RedisConnectionException.create(RedisConnectionException.java:94)
at i.l.core.AbstractRedisClient.getConnection(AbstractRedisClient.java:372)
at i.l.c.cluster.RedisClusterClient.connect(RedisClusterClient.java:403)
at i.l.c.cluster.RedisClusterClient.connect(RedisClusterClient.java:378)

Existing Discussion

Only PR I could find related to someone wanted to subclass cacheInterceptor

#279

[graemerocher]

multicode has started work on this issue in workspace micronaut-cache-live.