[Bug]: function-call "computer" tool calls are never executed — ToolError: Function computer not found

[sahatnicholas-gdp]

Primary area

MCP or agent integration

Summary

When ComputerAgent's configured model routes to the function-calling "computer" tool schema (_is_native_computer_use_model() returns False — any model not literally matching computer-use-preview), every resulting function_call named "computer" fails with ToolError: Function computer not found. The tool schema is built correctly and the model calls it correctly, but there is no execution path for it — _get_tool() only looks up user-registered custom tools, not the native computer-action semantics that computer_call-type items get.

Reproduction

1. Construct a ComputerAgent with a model that does not match computer-use-preview (e.g. openai/gpt-5.4, or any newer OpenAI model whose name isn't computer-use-preview) and a real computer tool via a computer handler, so openai.py's loop selects the function-calling schema (_map_computer_tool_to_openai(use_native_tool=False)).
2. Run any task that needs a UI action, e.g. "Open a terminal and check the Python version."
3. Observe the model correctly call the computer function with well-formed arguments (verified independently with a raw POST /v1/responses call against the same tool schema — the model returns {"type": "function_call", "name": "computer", "arguments": {"action": "screenshot", ...}} correctly).
4. Watch the agent loop dispatch that function_call item.

Expected behavior

The function_call named "computer" executes the requested action (click/type/screenshot/keypress/etc.) against the computer handler, the same way a native computer_call item would.

Actual behavior

agent.py's dispatch loop raises ToolError:

# libs/python/agent/cua_agent/agent.py (same on PyPI 0.8.4 and current `main`, lines 850-855)
if item_type == "function_call":
    await self._on_function_call_start(item)
    function = self._get_tool(item.get("name"))
    if not function:
        raise ToolError(f"Function {item.get('name')} not found")

_get_tool() (same file, ~line 598) only searches self.tools for a caller-registered BaseTool/callable matching the name — "computer" is never registered there, since the built-in computer-action handling is normally reached only through the separate computer_call item type (handled at ~line 753), not function_call. So every function-call-style computer invocation raises ToolError, which is caught and fed back to the model as a tool error. After one or two such failures the model typically gives up and reports the tool as unavailable — an accurate description of what actually happened, from its perspective.

This means the function-calling harness added for GPT-5.4 support (#1150) has never actually been able to execute a computer action — only the native computer_use_preview path (models literally named computer-use-preview) works.

Environment

• Cua component and version/commit: cua-agent 0.8.4 (PyPI, latest); confirmed the same code is present on main at commit visible via repos/trycua/cua/contents/libs/python/agent/cua_agent/agent.py as of 2026-09.
• Operating system and architecture: macOS (client driving the agent loop); target sandbox was an E2B Linux desktop sandbox (Ubuntu), reached via gl-computer-use (a downstream SDK: https://github.com/GDP-ADMIN/gl-sdk/tree/main/libs/gl-computer-use).
• Application/browser/window system/image: N/A — desktop automation via the computer tool against a remote Linux desktop sandbox.

Evidence

Confirmed via three independent checks:

1. Raw API call (bypassing cua-agent entirely) — POST /v1/responses with model set to a non-computer-use-preview OpenAI model and the same function-tool schema _map_computer_tool_to_openai(use_native_tool=False) builds — the model returns a correct function_call:

   {"type": "function_call", "name": "computer", "arguments": "{\"action\":\"screenshot\"}", ...}

2. Full agent loop trace — the same well-formed function_call items are visible in the loop's output (e.g. {'arguments': '{"action":"keypress",...,"keys":["ctrl","alt","t"],...,"status":"success"}', 'name': 'computer', 'type': 'function_call', 'status': 'completed'} — a "computer" function_call the model believes succeeded), yet the task's final text output reports the tool as unavailable, because it never actually ran.
3. Source read — _get_tool() implementation confirmed to only check self.tools (a caller-supplied list of custom tools), with no special-casing for "computer". No fork or private modification was made anywhere in this chain; only cua-agent as installed from PyPI was exercised.

None of the above involves credentials, private data, or sensitive screenshots — evidence is limited to tool schemas, function-call payloads, and source excerpts.

Workaround and additional context

• No workaround exists for the cua-agent (ComputerAgent) path itself for affected models — switching to a different agent framework (e.g. Agent-S/gui-agents) sidesteps it, since that framework calls providers generically rather than through this per-model dispatch table, but that's a different library, not a fix here.
• Confirmed the native computer_use_preview tool type is genuinely unsupported by at least one such model on a live account (400 "Tool 'computer_use_preview' is not supported with <model>."), so for models in this situation, function-calling is the only valid tool format — making this dispatch gap a hard blocker for actual computer-use with those models, not just a fallback-path nicety.
• Related: Support OpenAI GPT-5.5 built-in computer tool (computer_call.actions[]) #2136 (asking about a newer native computer tool flow for a different model) references the function-call harness path as an existing mode without mentioning it doesn't execute — this issue may be relevant context there too.
• Suspected fix location: in the item_type == "function_call" branch of the dispatch loop, special-case item.get("name") == "computer" to route through the same execution logic already used for native computer_call items, instead of falling through to _get_tool().
• Discovered and reported via GDP-ADMIN/gl-sdk#6521, a downstream SDK that wraps cua-agent.

[c8dhjp4tyv-bit]

Opened #3633 for this.

Implemented the fix location you suspected: a function_call named computer now routes through the same action semantics as a native computer_call (shared _execute_computer_action, so the two item types can't drift). Notes on the details that turned out to matter:

• Both argument shapes are accepted — the flat schema the function tool advertises, and the native action dict that replace_failed_computer_calls_with_function_calls re-emits.
• Flat start_x/start_y/end_x/end_y drags map onto the handler's path; null and non-applicable fields are dropped, since the flat parameter object is shared by every action (your trace's keypress carrying status: "success" would otherwise fail argument validation).
• The reply is a function_call_output, not a computer_call_output — the model holds the function tool, not computer_use_preview. That output is text-only, so the post-action screenshot is delivered as its own user message, and ImageRetentionCallback now trims those captures too.
• A caller-registered tool named computer still takes precedence.

13 new tests; verified 11 of them fail on unmodified main (the 2 that pass are the deliberate no-change guards). Full agent suite green.