Wire retrieval fixtures 042/043 (0100/0103)

Un-defer the two retrieval conformance fixtures the v0.107.0 pin
surfaced:

- 042 (malformed chunk usage, 0100): the contains_event matcher compared
  a record-valued field (EmbeddingUsage) against a dict with ==, which
  never matched. Promote wire.py's record-to-dict coercion to a shared
  as_record_mapping helper and use it for a subset match; a named subkey
  must be present, not just non-null.
- 043 (over-cap chunk-and-stitch, 0103): thread the conformance-adapter
  chunk_size cap override into the OpenAI embed provider construction
  (absent -> the fixed vendor 2048).

The present-key tightening is applied to the carries matcher too, so the
two subset comparisons stay consistent. Adds a unit test for the field
comparator. Harness-only; no src change. 052 (same-name collision) stays
deferred for the batched spec review.

Author: chris-colinsky

conformance.toml

@@ -974,7 +974,7 @@ note = "The §8.4 Cohere embed mapping recognizes input_type query / document /
 [proposals."0100"]
 status = "implemented"
 since = "0.16.0"
-note = "A provider figure present on the wire but malformed (a token count as a string / negative / boolean) in an ancillary field (usage, response_id) is treated as not reported: the call succeeds, the figure is nulled, and the verbatim value stays on raw. An implementation MUST NOT raise on it and MUST NOT fabricate / coerce / clamp / repair it. Each figure is judged independently and the outcome follows §4 / §6's record rules; the rule binds the graph-engine §6 EmbeddingEvent / RerankEvent figures as well as the response. Behavior shipped + unit-tested ahead of the pin; the malformed-usage / malformed-id / chunk-stitch fixtures (incl. retrieval 042) ride the v0.17.0 fixture-wiring PR."
+note = "A provider figure present on the wire but malformed (a token count as a string / negative / boolean) in an ancillary field (usage, response_id) is treated as not reported: the call succeeds, the figure is nulled, and the verbatim value stays on raw. An implementation MUST NOT raise on it and MUST NOT fabricate / coerce / clamp / repair it. Each figure is judged independently and the outcome follows §4 / §6's record rules; the rule binds the graph-engine §6 EmbeddingEvent / RerankEvent figures as well as the response. Behavior shipped + unit-tested ahead of the pin; retrieval fixture 042 (chunk-and-stitch malformed usage) runs -- its chunk-stitch usage assertion needed the contains_event record-subset match wired into the harness."
 
 # Spec v0.96.0 (proposal 0101).  A malformed llm usage counter is not
 # reported; the observability guard chain reconciled (llm-provider §7).
@@ -997,7 +997,7 @@ note = "Generalizes the carries directive capability-neutrally (a key MUST name
 [proposals."0103"]
 status = "implemented"
 since = "0.17.0"
-note = "§8.3's cap sentence pins that the §8 Batch chunking rule chunks by input COUNT only (the 2048-input cap); OpenAI's summed-token ceiling is NOT a chunking trigger, so an over-token request is sent and the provider's rejection surfaces as provider_invalid_request (no client-side token estimation). The mapping already implements count-based chunking (0092); the OpenAI embed provider gained a test-only chunk_size cap override (conformance-adapter §5.14) so a fixed-cap mapping's chunking fixture is self-enforcing. Behavior shipped + unit-tested ahead of the pin; fixture 043 (over-cap chunk-and-stitch + count-vs-token fail-loud) + the §5.14 harness directive ride the v0.17.0 fixture-wiring PR."
+note = "§8.3's cap sentence pins that the §8 Batch chunking rule chunks by input COUNT only (the 2048-input cap); OpenAI's summed-token ceiling is NOT a chunking trigger, so an over-token request is sent and the provider's rejection surfaces as provider_invalid_request (no client-side token estimation). The mapping already implements count-based chunking (0092); the OpenAI embed provider gained a test-only chunk_size cap override (conformance-adapter §5.14) so a fixed-cap mapping's chunking fixture is self-enforcing. Behavior shipped + unit-tested ahead of the pin; fixture 043 (over-cap chunk-and-stitch + count-vs-token fail-loud) runs, with the §5.14 chunk_size cap override threaded to the OpenAI embed provider construction in the harness."
 
 # Spec v0.99.0 (proposal 0104).  An empty-string response_id is null
 # (absent); §8.2 Jina maps a bare 400 to provider_invalid_request

tests/conformance/harness/__init__.py

@@ -25,6 +25,7 @@
 from .loader import discover_fixtures, load_fixture
 from .skip import SkipReason
 from .wire import (
+    as_record_mapping,
     assert_error_carries,
     assert_response_format_absent,
     assert_system_references_schema,
@@ -39,6 +40,7 @@
     "GraphFixture",
     "LlmProviderFixture",
     "SkipReason",
+    "as_record_mapping",
     "assert_error_carries",
     "assert_response_format_absent",
     "assert_system_references_schema",

tests/conformance/harness/wire.py

@@ -183,17 +183,19 @@ def assert_error_carries(exc: BaseException, carries: Mapping[str, Any]) -> None
             actual = _get_carries_attr(exc, key)
             if isinstance(expected, Mapping):
                 # A mapping-valued field (e.g. usage) is a subset match: each
-                # named key must equal the actual's value, reading a pydantic
-                # record's fields (Usage) or a plain mapping. Keys the fixture
-                # omits are ignored (the record MAY carry optional extras).
-                actual_map = _as_carries_mapping(actual)
+                # named key must be PRESENT on the actual with an equal value,
+                # reading a pydantic record's fields (Usage) or a plain mapping.
+                # Keys the fixture omits are ignored (the record MAY carry
+                # optional extras); requiring presence keeps a named key expected
+                # as null from matching a record that simply lacks the field.
+                actual_map = as_record_mapping(actual)
                 if actual_map is None:
                     raise AssertionError(
                         f"carries check failed: {key!r} is not a mapping/record "
                         f"(got {type(actual).__name__}); cannot subset-match {expected!r}"
                     )
                 for subkey, subval in cast("Mapping[str, Any]", expected).items():
-                    if actual_map.get(subkey) != subval:
+                    if subkey not in actual_map or actual_map[subkey] != subval:
                         raise AssertionError(
                             f"carries check failed: {key!r}[{subkey!r}] "
                             f"actual={actual_map.get(subkey)!r}, expected={subval!r}"
@@ -204,10 +206,11 @@ def assert_error_carries(exc: BaseException, carries: Mapping[str, Any]) -> None
                 )
 
 
-def _as_carries_mapping(value: Any) -> Mapping[str, Any] | None:
-    """Coerce a carries attribute to a mapping for subset comparison: a plain
-    Mapping passes through; a pydantic record (e.g. Usage) is dumped to a dict;
-    anything else returns None (not comparable as a mapping)."""
+def as_record_mapping(value: Any) -> Mapping[str, Any] | None:
+    """Coerce a value to a mapping for subset comparison: a plain Mapping
+    passes through; a pydantic record (e.g. Usage / EmbeddingUsage) is dumped to
+    a dict; anything else returns None (not comparable as a mapping). Shared by
+    the carries matcher (here) and the typed-event contains_event matcher."""
     if isinstance(value, Mapping):
         return cast("Mapping[str, Any]", value)
     dump = getattr(value, "model_dump", None)

tests/conformance/test_retrieval_provider.py

@@ -33,6 +33,7 @@
     CohereEmbeddingProvider,
     CohereRerankProvider,
     EmbeddingRuntimeConfig,
+    EmbeddingUsage,
     JinaEmbeddingProvider,
     JinaRerankProvider,
     OpenAIEmbeddingProvider,
@@ -42,6 +43,7 @@
 )
 
 from ._deferral import skip_if_deferred
+from .harness import as_record_mapping
 
 # The wire endpoint each mapping posts to, keyed by the fixture's ``mapping``
 # directive (None => the harness-internal default). The wire-capture assertion
@@ -91,27 +93,13 @@
 # TeiEmbeddingProvider.embed() chunk-and-stitches by its construction chunk_size
 # via the shared chunk_and_stitch_embed helper (also adopted by Cohere /v2/embed),
 # so no retrieval fixtures remained deferred at that pin. The v0.107.0 pin bump
-# then re-defers 042 / 043 / 052, whose fixture-model wiring rides the v0.17.0
-# fixture-wiring PR (see the categorized entries below).
+# re-deferred 042 / 043 / 052; 042 (contains_event record match) and 043
+# (chunk_size directive) are now wired, leaving only 052 (the same-name declared
+# collision, held for the batched spec review).
 _DEFERRED_FIXTURES: dict[str, str] = {
     # v0.17.0 spec-pin bump (v0.88.0 -> v0.107.0). Behavior for each of
     # these shipped + is unit-tested ahead of the pin; the conformance
     # fixture wiring rides the v0.17.0 fixture-wiring PR.
-    # Proposal 0100 (spec v0.95.0) malformed ancillary figure not reported.
-    # Behavior shipped in v0.16.0 (chunk-stitch response_id nulling + all-or-
-    # nothing usage, 0092/0093); the gap is the `contains_event` matcher, which
-    # compares a record-valued field (usage) with `==` against a dict expectation
-    # and so never matches an EmbeddingUsage object. Matcher fix rides PR 6.
-    "042-embed-chunk-malformed-usage-not-reported": (
-        "Proposal 0100 malformed ancillary figure; contains_event matcher (record-vs-dict) "
-        "rides the v0.17.0 fixture-wiring PR"
-    ),
-    # Proposal 0103 (spec v0.98.0) §8.3 over-cap chunk-and-stitch + the
-    # conformance-adapter §5.14 chunk_size construction directive.
-    "043-embed-openai-chunk-and-stitch": (
-        "Proposal 0103 chunk_size construction directive (conformance-adapter §5.14); "
-        "harness wiring rides the v0.17.0 fixture-wiring PR"
-    ),
     # Proposal 0108 (spec v0.103.0) same-NAME declared-field collision. The
     # dimensions reject is coded (openai.py) but a declared-field-named extras
     # key routes to the declared field, never into model_extra, so the collision
@@ -233,6 +221,8 @@ def _build_provider(
                 transport=transport,
                 query_prefix=cast("str | None", block.get("query_prefix")),
                 document_prefix=cast("str | None", block.get("document_prefix")),
+                # §5.14 test-only cap override; absent -> the fixed vendor 2048.
+                chunk_size=cast("int | None", block.get("chunk_size")),
             )
         else:
             # The pre-0079 protocol fixtures (001-012, mapping absent) carry no
@@ -559,11 +549,30 @@ def _cleanup() -> None:
     return events, _cleanup
 
 
+def _contains_field_matches(actual: Any, expected: Any) -> bool:
+    # A Mapping-valued expectation (e.g. usage: {input_tokens: 500}) is a subset
+    # match against a record-valued field (EmbeddingUsage): coerce the field to a
+    # dict and require each named key to be PRESENT with an equal value, ignoring
+    # keys the fixture omits. Requiring presence keeps a named key expected as
+    # null from matching a record that simply lacks the field. Scalars and null
+    # use plain equality (so usage: null matches a null record).
+    if isinstance(expected, Mapping):
+        actual_map = as_record_mapping(actual)
+        if actual_map is None:
+            return False
+        return all(
+            k in actual_map and actual_map[k] == v for k, v in cast("Mapping[str, Any]", expected).items()
+        )
+    return bool(actual == expected)
+
+
 def _assert_contains_event(events: list[Any], expected: Mapping[str, Any]) -> None:
     """Assert each observer's contains_event against the collected events.
 
     ``contains_event`` names an event_type and a fields mapping; a collected
-    event of that type must match every listed field (dict / scalar equality).
+    event of that type must match every listed field. A scalar / null field uses
+    equality; a mapping-valued field is a subset match against the event's record
+    field (e.g. ``usage: {input_tokens: 500}`` vs an ``EmbeddingUsage``).
     """
     observers = cast("Mapping[str, Any]", expected.get("observers") or {})
     for obs_name, obs_expect in observers.items():
@@ -575,7 +584,9 @@ def _assert_contains_event(events: list[Any], expected: Mapping[str, Any]) -> No
         want_fields = cast("Mapping[str, Any]", contains.get("fields") or {})
         candidates = [e for e in events if type(e).__name__ == event_type]
         for event in candidates:
-            if all(getattr(event, key, None) == val for key, val in want_fields.items()):
+            if all(
+                _contains_field_matches(getattr(event, key, None), val) for key, val in want_fields.items()
+            ):
                 break
         else:
             raise AssertionError(
@@ -683,3 +694,20 @@ async def _run_rerank_case(
     finally:
         cleanup()
         await provider.aclose()
+
+
+def test_contains_field_matches() -> None:
+    # The contains_event field comparator: scalars / null by equality, a mapping
+    # expectation by record-subset (present-key required).
+    assert _contains_field_matches("embed_node", "embed_node")
+    assert not _contains_field_matches("a", "b")
+    assert _contains_field_matches(None, None)
+    # A mapping expectation subset-matches a record, ignoring unnamed fields.
+    assert _contains_field_matches(EmbeddingUsage(input_tokens=500), {"input_tokens": 500})
+    assert _contains_field_matches({"a": 1, "b": 2}, {"a": 1})
+    # A mapping expectation never matches a null / non-record actual.
+    assert not _contains_field_matches(None, {"input_tokens": 500})
+    # A named key the record lacks does not match even when expected null: the
+    # present-key requirement keeps a missing field from reading as an explicit
+    # null (the .get()-based comparison this replaced would have matched here).
+    assert not _contains_field_matches(EmbeddingUsage(input_tokens=500), {"missing": None})