Skip to content

AzureAISearchDocumentStore's 'in' filter joins values with the same comma Azure uses as its search.in() delimiter #3844

Description

@AmirF194

Describe the bug

AzureAISearchDocumentStore's in filter operator builds a search.in(field, valueList, delimiter) OData expression by joining values with a comma (_in() in filters.py), the same character Azure uses as the delimiter argument. Any string metadata value that itself contains a literal comma gets split apart by Azure's own server side parsing of that value list, so the filter never matches the value you passed in. A value like "New York, NY" becomes three fragments (New York, NY, Austin in a two-value list), none of which equal the original string.

_escape_odata_literal only escapes single quotes for the eq/ne operators; _in calls it too, but escaping a quote does nothing for a comma, since the comma is consumed by search.in's own delimiter parsing, not by OData string-literal parsing.

To Reproduce

from haystack_integrations.document_stores.azure_ai_search.filters import _normalize_filters

_normalize_filters({"operator": "in", "field": "meta.city", "value": ["New York, NY", "Austin"]})
# "search.in(city,'New York, NY,Austin',',')"

Per Azure's documented search.in(variable, valueList, delimiter) semantics, the service splits the value-list string on the delimiter before comparing, so the query above resolves to matching against ["New York", " NY", "Austin"], never "New York, NY". I reproduced the query-construction output above in a clean container against current main (39ba76eb3); I do not have a live Azure Search index to confirm the server-side split independently, but it follows Azure's own documented behavior for search.in.

A fix would need a delimiter that cannot appear in the values (or per-value escaping of the chosen delimiter before joining), plus a regression test alongside the existing _in/_escape_odata_literal tests.

Describe your environment

  • Integration: azure-ai-search-haystack
  • File: integrations/azure_ai_search/src/haystack_integrations/document_stores/azure_ai_search/filters.py, _in() (around line 91)
  • Reproduced against main at 39ba76eb3

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions