Generic tool-call parser (parse_tool_calls) misses tool calls when start/end markers aren't the sole content of a stream chunk

[kkakkung2000]

Summary

parse_tool_calls() in src/exo/worker/runner/llm_inference/model_output_parsers.py is the generic fallback parser used for any model with a tool_parser config that isn't GptOssModel or DeepseekV32Model (see dispatch in apply_all_parsers(), lines 33-62). It detects tool-call boundaries with:

if not in_tool_call and response.text.startswith(tool_parser.start_parsing):
    in_tool_call = True
...
if response.text.endswith(tool_parser.end_parsing):

Both checks run against response.text for the current streaming chunk in isolation — there is no accumulator. If the start or end marker doesn't line up exactly with a chunk boundary, the tool call is never recognized as structured output and the raw marker text (e.g. <tool_call>{"name": "...", "arguments": {...}}</tool_call>) leaks into the model's regular content instead of populating tool_calls.

This failure mode is reachable by any model using the <tool_call> marker convention through this generic path — e.g. Qwen3-Next-80B — whenever streaming chunk boundaries don't happen to line up with the marker.

Reproduction

Streaming inference delivers output in arbitrary chunk boundaries — a chunk is not guaranteed to align with the tool-call marker. Two conceptual chunk sequences that defeat the current logic:

1. Preamble shares a chunk with the start marker

["Sure, let me check that.<tool_call>", "{\"name\": \"get_weather\", ...}", "</tool_call>"]

response.text.startswith(tool_parser.start_parsing) is False on the first chunk because the chunk isn't only the marker — it's preamble + marker. in_tool_call never gets set, so nothing downstream of this point is parsed as a tool call.

2. Marker itself is split across a chunk boundary

["<tool_", "call>{\"name\": \"get_weather\", ...}</tool_call>"]

Neither chunk's text starts with the full start_parsing string, and depending on how the end marker is chunked, endswith(tool_parser.end_parsing) can similarly fail to match. The tool call is silently missed.

In both cases the result is the same: the parser falls through to plain-text handling and the raw <tool_call>...</tool_call> payload is surfaced to the caller as assistant content instead of a structured tool_calls entry.

Root cause

src/exo/worker/runner/llm_inference/model_output_parsers.py, function parse_tool_calls() (pre-fix version spans roughly lines 328-379 at commit eb6ae9f). The function checks startswith/endswith against each chunk's response.text independently, with no cross-chunk accumulation buffer and no handling for a marker that only partially appears in a given chunk.

Why this is easy to miss

The existing test suite (test_parse_tool_calls.py, 5 tests as of eb6ae9f) only exercises clean chunk sequences where the start marker is the entire first chunk and the end marker is the entire last chunk, e.g.:

["<tool_call>", "test_fn", "</tool_call>"]

None of the existing tests cover a chunk that mixes preamble text with the start marker, or a marker split mid-chunk — which is exactly the case that fails, and exactly the kind of chunking a real model server produces under normal streaming conditions.

Suggested fix

Replace the per-chunk startswith/endswith checks with accumulated-text matching plus a small buffer for partial markers that might complete on the next chunk: maintain an accumulated string across chunks, search for the start/end markers as substrings of the accumulated text rather than anchors on a single chunk, and hold back a pending_buffer when the tail of the current text could be a prefix of the marker (only flushing it once it's confirmed not to be part of the marker).

This is not a new pattern for this file — parse_deepseek_v32(), elsewhere in the same module, already solves this exact problem for DSML markers (which can also span multiple tokens) using an accumulated string and a _could_be_dsml_prefix() helper. Generalizing the same approach for the generic tool-call path would bring parse_tool_calls() in line with the strategy already proven out for DeepSeek-V3.2, rather than introducing a new mechanism.

Affected models

Any model routed through the generic parser path — i.e., any model configured with a tool_parser that isn't GptOssModel (which uses parse_gpt_oss) or DeepseekV32Model (which uses parse_deepseek_v32), including Qwen3-Next-80B-A3B-Instruct-8bit. Any model using the shared <tool_call>...</tool_call>-style marker convention through this path is exposed to the same failure mode.

Offer

I have a fix implemented and verified against the real parser logic (accumulated-text + partial-marker buffering, following the parse_deepseek_v32 pattern), covering the two failure modes above plus the existing passing cases (normal closed tool call, no-tool-call passthrough, failed-parse-to-text, argument coercion). Happy to open a PR with the fix and three new regression tests if that's useful — wanted to raise this for visibility/discussion first given how the dispatch logic and marker conventions are shared across parsers.