Message listener stops pulling more messages - Ktor

[SalyczeQ]

Hi,
I discover some issues with pulling messages. It seems that after some time of application running, the message listener stops pulling more messages from SQS. (Other parts of the app runs normally)

I am using prefetchingMessageListener in version 4.4.0 with this conf in Ktor 1.6.7.

concurrencyLevel = { 2 }
messageVisibility = { Duration.ofSeconds(60) }
desiredPrefetchedMessages = 5
maxPrefetchedMessages = 10

I understand that this is not easy to figure out what is exactly happening and what can be wrong.
Would you have even advice on how to possibly trace this issue? Or debug it?

thanks,
Saly

[JaidenAshmore]

Hey, sorry for the delay!

yeah we publish a bunch of DEBUG logs that could help you nail it down so make sure that the com.jashmore namespace is in DEBUG.

The most likely problems that could be causing this are:

• You have some sort of AWS problem causing no messages to be received. E.g. the PrefetchingMessageRetriever is polling for messages from the queue but none are being returned. You can see on this line it would tell you whether any messages were retrieved: PrefetchingMessageRetriever.java#L114. E.g. it might continuously be saying received 0 messages which could indicate a problem with your queue.
• Maybe you have a set of messages that have been downloaded but you aren't picking them up to be processed. We use a semaphore for this so if you are able to put a debugger in a line like this: ConcurrentMessageBroker.java#L63 it could indicate that it wasn't able to obtain a permit to process another message. This fuels the concurrency aspect in that it will allow processing of N messages at the same time. There could be a bug in there that I don't know about.
• Another problem could be that you aren't resolving the message. You could try and look for log lines here to make sure that it is actually deleting the message from the queue: BatchingMessageResolver.java#L110.

If you were able to reproduce the problem with this example I could help look into debugging but it is hard to know without being able to reproducing it myself.

Let me know how you go!