fix: use standard IMAP SEARCH instead of ESEARCH to avoid UID/seqnum mismatch#282
Open
vcazan wants to merge 2 commits intoabhinavxd:mainfrom
Open
fix: use standard IMAP SEARCH instead of ESEARCH to avoid UID/seqnum mismatch#282vcazan wants to merge 2 commits intoabhinavxd:mainfrom
vcazan wants to merge 2 commits intoabhinavxd:mainfrom
Conversation
…mismatch Some IMAP servers (e.g. Purelymail) return UIDs in ESEARCH responses without setting the UID tag, causing fetchAndProcessMessages to treat them as sequence numbers. When the UID values exceed the mailbox message count (e.g. UID 590 in a 444-message mailbox), the FETCH command finds nothing and all incoming emails are silently dropped. This removes the ESEARCH code path entirely and always uses standard SEARCH, which reliably returns sequence numbers. The fetchAndProcess function is simplified to only use AllSeqNums(). Fixes silent email import failure with IMAP providers whose ESEARCH implementation does not correctly tag results as UIDs. Made-with: Cursor
Connect a Shopify store via OAuth to display customer profiles, order history, and lifetime spend in the conversation sidebar. Includes an admin UI for configuring credentials, an OAuth callback handler for token exchange, encrypted secret storage, and a v1.0.4 DB migration. Made-with: Cursor
Owner
|
Scope of this PR looks off - the description is about an IMAP ESEARCH fix, but the diff also adds a shopify integration. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Some IMAP servers (confirmed with Purelymail) return UIDs in their ESEARCH responses without setting the
UIDtag in the response. This causesfetchAndProcessMessagesto interpret UID values (e.g. 590–597) as sequence numbers and pass them toclient.Fetch()via aSeqSet. When those values exceed the actual mailbox message count (e.g. 444 messages), the FETCH command returns zero results and all incoming emails are silently dropped.Root cause
In
searchMessages, when the server advertisesESEARCHvia its capability flag, the code sends aSEARCHcommand withRETURN (MIN MAX ALL COUNT)options. The server responds with an ESEARCH result containingMIN,MAX, andALLvalues. Per RFC 4731, these should be sequence numbers for a non-UID SEARCH, but Purelymail returns UIDs without theUIDtag. The go-imap v2 library then setsSearchData.UID = false, sofetchAndProcessMessagestakes theMin > 0 && Max > 0branch and adds the UID values to aSeqSet:Since
SeqSetrange 590–597 doesn't exist in a 444-message mailbox,client.Fetch()returns nothing.Why the ESEARCH capability check isn't sufficient
The original code checked
client.Caps().Has(imap.CapESearch)before using ESEARCH, and had a fallback to standard SEARCH if ESEARCH returned an error. This is a reasonable design — use the faster protocol extension when available, degrade gracefully on failure.The problem is that the capability flag only tells you the server claims to support ESEARCH, not that it correctly implements it. Purelymail advertises ESEARCH support, accepts the ESEARCH request without error, and returns a structurally valid response — but the data inside is wrong (UIDs where sequence numbers should be). The existing fallback only triggers on ESEARCH errors, and there is no error here. The server returns confidently wrong data that passes all structural validation.
There is no reliable way to detect this at the application level. You could compare Min/Max against the mailbox EXISTS count as a heuristic, but that's fragile and couples SEARCH logic to mailbox state. The safest fix is to avoid the ESEARCH code path entirely.
Fix
searchMessagesand always use standardSEARCH(no return options), which reliably returns sequence numbers across all IMAP servers.fetchAndProcessMessagesto only useAllSeqNums().This is a minimal, safe change. A more targeted alternative would be to check
SearchData.UIDand useUIDSet/UID FETCHwhen true, but that still wouldn't help when the server incorrectly omits the UID tag (as Purelymail does).Performance note
Standard SEARCH returns the full list of matching sequence numbers rather than just a min/max range. For very large mailboxes with many matches, this means slightly more data over the wire. In practice, since
scan_inbox_sincedefaults to 48 hours, the result set is typically small and the difference is negligible.Test plan