fix(shell): decode command output as UTF-8 instead of the platform codepage

[LHMQ878]

Description

platform/coworker/tools/shell.py spawns the persistent shell with text=True and no encoding=, so the child's output is decoded with the platform's preferred encoding — a legacy codepage on a stock Windows install (cp936 on a Chinese system, cp1252 on a Western one), not UTF-8. The file already has explicit Windows/PowerShell support, so this path is reached in normal use.

Command output is the least trustworthy byte stream this module handles — cat on a binary file, a compiler quoting a snippet in another encoding, curl echoing a response body, a test runner printing a non-ASCII assertion message. Under a codepage those bytes fail in two ways, and the first one is severe:

1. Bytes the codepage rejects kill the whole shell session. _read_loop is a daemon thread iterating proc.stdout. A UnicodeDecodeError there escapes the loop, the finally pushes the EOF sentinel, and run() reads that sentinel as "the shell died". The session is gone — and so is every line already buffered, including the plain ASCII on either side of the offending bytes.

Reproduced with the exact structure of _read_loop (text=True, reader in a daemon thread, EOF sentinel in finally), a child writing b"ok-before\n\x81 raw\nok-after\n":

locale codec: cp936

[current: text=True, no encoding]
   lines : []
   raised: UnicodeDecodeError: 'gbk' codec can't decode byte 0x81 in position 10

[fixed: encoding=utf-8, errors=replace]
   lines : ['ok-before\n', '� raw\n', 'ok-after\n']
   raised: None

Note lines: []. ok-before had already been written by the child and was lost too.

2. Bytes the codepage accepts but maps differently become silent mojibake. "café".encode("utf-8") decodes under cp936 to "café" with no exception at all. The model receives a corrupted string it cannot distinguish from the real one — no error to notice, no signal that anything went wrong.

More Information

Two sites, both subprocess.Popen(..., text=True) with no encoding=:

Location	Role
_BackgroundTask.__init__ (~line 96)	background tasks polled via shell_task_output
LocalExecutor._spawn (~line 194)	the persistent foreground shell — the agent's command-execution core

Both now get encoding="utf-8", errors="replace" via a shared module constant that documents the reasoning:

_OUTPUT_DECODING: dict[str, Any] = {"encoding": "utf-8", "errors": "replace"}

errors="replace" rather than strict is the point of the fix: undecodable bytes degrade to U+FFFD and the stream stays in sync, instead of a decode error taking down the reader thread and the session with it.

The subprocess.run call for pgrep (~line 372) is deliberately left alone: it is POSIX-only, emits nothing but PID digits, and is already guarded by ValueError on int(pid).

This is also a consistency fix. The repo already names UTF-8 with a permissive error handler where it reads untrusted text: tools/files.py:76 (encoding="utf-8", errors="replace"), tools/search.py:165 (errors="ignore"), and four call sites in tools/email_tools.py. The shell was the one place reading external bytes with the locale default.

Validation

Three tests added to platform/tests/test_shell.py, in a new "output decoding" section, following the file's existing per-OS parameterized command style:

Test	Guards against
test_undecodable_output_does_not_kill_the_session	dead reader thread → lost session and lost buffered output
test_utf8_output_is_not_mojibake	silent corruption (asserts "café" in output and "café" not in output)
test_background_task_undecodable_output_survives	the second Popen call, reached only by background tasks

The mojibake test needs the bytes to arrive at the pipe unmodified, so it writes them straight to the stdout handle ([Console]::OpenStandardOutput().Write(...) on Windows, printf 'caf\303\251\n' on POSIX). PowerShell's Write-Output would re-encode through its own output encoding first, which is a different concern from the decoding under test.

Control experiment — shell.py reverted to the unpatched Popen calls, new tests kept. The three failures map one-to-one onto the three failure modes:

3 failed, 10 passed

test_shell.py:126  assert None == 0                                # reader died → session considered dead
test_shell.py:147  AssertionError: assert 'café' in 'café\n\n'    # silent mojibake
test_shell.py:163  AssertionError: assert 'ok-before' in ''        # background output entirely lost

The 10 that stay green are the existing tests — the fix does not change any behavior they cover.

With the fix: tests/test_shell.py → 13 passed. Together with the neighbouring tool suites (test_shell.py test_code_tools.py test_tools_permissions.py) → 39 passed.

No regressions. Full platform/tests/ suite:

tree	result
this branch	796 passed, 12 failed, 1 skipped
unmodified main	793 passed, 12 failed, 1 skipped

The same 12 failures on both trees — test_fake_slack.py (6), test_slack_relay.py (4), test_github_installs.py (1), test_ui_refresh_e2e.py (1), mostly TimeoutError from tests that need a live socket. Pre-existing and unrelated. The delta is exactly the 3 new tests.

ruff check → All checks passed. ruff format --check → 2 files already formatted.

On a UTF-8 host the previous behavior and the new one coincide, so nothing that worked before changes.

Linked Issues

None — I did not find an existing issue for this (searched encoding, UnicodeDecodeError, mojibake, codepage, shell decode: 0 relevant results). Happy to open one if you'd prefer the issue-first flow.