Skip to content

fix: use standard IMAP SEARCH instead of ESEARCH to avoid UID/seqnum mismatch#282

Open
vcazan wants to merge 2 commits intoabhinavxd:mainfrom
vcazan:fix/imap-esearch-uid-mismatch
Open

fix: use standard IMAP SEARCH instead of ESEARCH to avoid UID/seqnum mismatch#282
vcazan wants to merge 2 commits intoabhinavxd:mainfrom
vcazan:fix/imap-esearch-uid-mismatch

Conversation

@vcazan
Copy link
Copy Markdown

@vcazan vcazan commented Apr 13, 2026

Summary

Some IMAP servers (confirmed with Purelymail) return UIDs in their ESEARCH responses without setting the UID tag in the response. This causes fetchAndProcessMessages to interpret UID values (e.g. 590–597) as sequence numbers and pass them to client.Fetch() via a SeqSet. When those values exceed the actual mailbox message count (e.g. 444 messages), the FETCH command returns zero results and all incoming emails are silently dropped.

Root cause

In searchMessages, when the server advertises ESEARCH via its capability flag, the code sends a SEARCH command with RETURN (MIN MAX ALL COUNT) options. The server responds with an ESEARCH result containing MIN, MAX, and ALL values. Per RFC 4731, these should be sequence numbers for a non-UID SEARCH, but Purelymail returns UIDs without the UID tag. The go-imap v2 library then sets SearchData.UID = false, so fetchAndProcessMessages takes the Min > 0 && Max > 0 branch and adds the UID values to a SeqSet:

seqSet.AddRange(searchResults.Min, searchResults.Max) // UIDs treated as seqnums

Since SeqSet range 590–597 doesn't exist in a 444-message mailbox, client.Fetch() returns nothing.

Why the ESEARCH capability check isn't sufficient

The original code checked client.Caps().Has(imap.CapESearch) before using ESEARCH, and had a fallback to standard SEARCH if ESEARCH returned an error. This is a reasonable design — use the faster protocol extension when available, degrade gracefully on failure.

The problem is that the capability flag only tells you the server claims to support ESEARCH, not that it correctly implements it. Purelymail advertises ESEARCH support, accepts the ESEARCH request without error, and returns a structurally valid response — but the data inside is wrong (UIDs where sequence numbers should be). The existing fallback only triggers on ESEARCH errors, and there is no error here. The server returns confidently wrong data that passes all structural validation.

There is no reliable way to detect this at the application level. You could compare Min/Max against the mailbox EXISTS count as a heuristic, but that's fragile and couples SEARCH logic to mailbox state. The safest fix is to avoid the ESEARCH code path entirely.

Fix

  • Remove the ESEARCH code path in searchMessages and always use standard SEARCH (no return options), which reliably returns sequence numbers across all IMAP servers.
  • Simplify fetchAndProcessMessages to only use AllSeqNums().

This is a minimal, safe change. A more targeted alternative would be to check SearchData.UID and use UIDSet/UID FETCH when true, but that still wouldn't help when the server incorrectly omits the UID tag (as Purelymail does).

Performance note

Standard SEARCH returns the full list of matching sequence numbers rather than just a min/max range. For very large mailboxes with many matches, this means slightly more data over the wire. In practice, since scan_inbox_since defaults to 48 hours, the result set is typically small and the difference is negligible.

Test plan

  • Tested against Purelymail IMAP server — 8 emails from the last 48h are now correctly fetched and processed (previously 0)
  • Verify no regression with Gmail IMAP
  • Verify no regression with Microsoft 365 IMAP

vcazan added 2 commits April 13, 2026 15:41
…mismatch

Some IMAP servers (e.g. Purelymail) return UIDs in ESEARCH responses
without setting the UID tag, causing fetchAndProcessMessages to treat
them as sequence numbers. When the UID values exceed the mailbox message
count (e.g. UID 590 in a 444-message mailbox), the FETCH command finds
nothing and all incoming emails are silently dropped.

This removes the ESEARCH code path entirely and always uses standard
SEARCH, which reliably returns sequence numbers. The fetchAndProcess
function is simplified to only use AllSeqNums().

Fixes silent email import failure with IMAP providers whose ESEARCH
implementation does not correctly tag results as UIDs.

Made-with: Cursor
Connect a Shopify store via OAuth to display customer profiles, order
history, and lifetime spend in the conversation sidebar. Includes an
admin UI for configuring credentials, an OAuth callback handler for
token exchange, encrypted secret storage, and a v1.0.4 DB migration.

Made-with: Cursor
@abhinavxd
Copy link
Copy Markdown
Owner

Scope of this PR looks off - the description is about an IMAP ESEARCH fix, but the diff also adds a shopify integration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants