feat(testing): add cfn_handler.testing module with replay() and helpers

[igorlg]

Issue

No tracked issue — cfn-handler users currently have no first-class
way to unit-test their custom-resource handlers without reaching into
cfn_handler._internal/ (which the README explicitly says is unstable).
The legacy test_mode=True flag introduced during early v1 development
worked around this but accumulated known issues: mutable state on the
resource, an awkward sentinel string for the polling-defer case, and an
inability to test polling re-invocation because the marker keys are
never written to the event in test mode.

Planned in OpenSpec change add-testing-helpers.
Roadmap context: docs/ROADMAP.md (this is item #1
of the v1.x evolution; item #2 is better logging, item #3 is optional
idempotency).

Summary

Adds a public cfn_handler.testing module so users can unit-test
custom-resource handlers in-process: no HTTP, no boto3, no moto
setup. The headline API is CustomResource.replay(event, context=None)
which executes the full dispatch pipeline and returns a structured
Replay value. Polling-aware: a deferred replay mutates the event
with marker keys and returns Replay(status="DEFERRED"); a follow-up
replay() of the mutated event resumes through the poll handler —
without provisioning a real EventBridge rule.

Soft-deprecates the existing test_mode / last_response surface
(continues to work in v1.x; emits DeprecationWarning; removed in v2.0).
The internal test suite is migrated onto the new API in the same PR
to dogfood it.

Changes

src/cfn_handler/testing/                       NEW public module
├── __init__.py                                Re-exports __all__
├── py.typed
├── fixtures.py                                pytest11 entry point
└── _internal/
    ├── replay_result.py                       Replay frozen dataclass
    ├── runner.py                              run_replay() driver
    ├── event_factory.py                       make_event()
    ├── context_factory.py                     make_context()
    └── assertions.py                          assert_success / _failed / _deferred

src/cfn_handler/resource.py
├── + Transport / PollerProvision / PollerTeardown seam kwargs
│     (late-bound defaults; existing patches keep working)
├── + _replay_seams() context manager
├── + replay() public method
└── + DeprecationWarning when test_mode=True

src/cfn_handler/_internal/{response,poller}.py
└── + Transport, PollerProvision, PollerTeardown type aliases

pyproject.toml                                 + [pytest11] entry point
justfile                                       test-cov uses `coverage run`
.github/workflows/ci.yml                       same fix in CI
README.md                                      + Testing section

tests/                                         migrated + new
├── unit/test_resource.py                      ~104 refs migrated to replay()
├── unit/test_backstops.py                     test_mode-specific test moved
├── unit/test_polling_dispatch.py              sentinel test moved
├── unit/test_state_machine.py                 hypothesis tests migrated
├── unit/test_test_mode_deprecation.py         NEW (deprecation tests)
├── unit/test_transport_seam.py                NEW
├── unit/test_poller_seam.py                   NEW
├── unit/testing/                              NEW (4 files, 42 tests)
├── integration/test_replay_parity.py          NEW (production-vs-replay)
└── integration/test_fixture_discovery.py      NEW (pytest11 entry point)

Public API surface (cfn_handler.testing)

Name	Kind	Purpose
Replay	frozen dataclass	Outcome of a replay (status / data / reason / no_echo / payload / ...)
make_event(...)	factory	Canonical CFN custom-resource event with safe defaults
make_context(...)	factory	Minimal LambdaContext-protocol object
assert_success	helper	Pass when status=SUCCESS, optionally match data / pid / no_echo
assert_failed	helper	Pass when status=FAILED, optionally match reason_contains
assert_deferred	helper	Pass when status=DEFERRED (replay-only sentinel)
cfn_*_event	pytest fixture	Auto-discovered via pytest11
cfn_lambda_context	pytest fixture	Auto-discovered via pytest11

The module supports zero-pytest, zero-boto3 environments — the entry
point only loads inside a pytest run, and boto3 is never imported
on the non-polling path.

One-line example

from cfn_handler import CustomResource
from cfn_handler.testing import assert_success, make_event

resource = CustomResource()

@resource.create
def on_create(event, ctx):
    return {"Endpoint": "https://x.example"}

assert_success(resource.replay(make_event()), data={"Endpoint": "https://x.example"})

Migration for users on test_mode

Mechanical translation:

# Before (still works in v1.x, emits DeprecationWarning)
resource = CustomResource(test_mode=True)
resource(event, ctx)
assert resource.last_response["Status"] == "SUCCESS"
assert resource.last_response["Data"] == {"x": 1}

# After
resource = CustomResource()
replay = resource.replay(event, ctx)
assert replay.status == "SUCCESS"
assert replay.data == {"x": 1}
# or, more idiomatically:
assert_success(replay, data={"x": 1})

The polling case improves materially: where test_mode only wrote a
{"__cfn_handler_polling__": True} sentinel and short-circuited the
provisioning, replay() now mutates the event with the real marker
keys, so a follow-up replay() correctly resumes through the poll
handler — letting users test both halves of a polled lifecycle without
moto.

Tests

• New tests added for new behaviour (~50 across replay, factories, assertions, fixtures, parity)
• Existing tests still pass after migration onto the new API (~100 sites translated)
• just test-cov — 155 passing, 98% line+branch coverage (gate: 95%)
• just typecheck — mypy strict + pyright strict, 16 source files, no issues
• just lint — ruff + ruff format + cfn-lint, all checks passed
• just gha-pre-release — all locally-replayable gating jobs green (CodeQL skipped under act due to its post-analysis REST API call against a synthesized run id; validated by real GH Actions on this PR — see commit ac19152)
• Manual smoke test in a fresh venv outside the repo: replay(make_event(), make_context()) returns SUCCESS, fixtures auto-discover

Coverage measurement note

Switched test-cov to coverage run -m pytest (from pytest --cov) in
both justfile and ci.yml. The new pytest11 entry point causes
pytest to import cfn_handler.testing.fixtures (and transitively
cfn_handler itself) at plugin-collection time — before pytest-cov
attaches its instrumentation hooks. Module-level code in __init__.py
then runs uninstrumented and the report shows artificial 0% coverage on
those lines, dropping aggregate to ~68%. coverage run -m pytest
attaches the tracer before any imports. Documented in both the recipe
and the workflow.

Breaking changes?

No. v1.3 ships the new replay() API as purely additive. The
test_mode flag and last_response attribute continue to function
exactly as before — they just emit a DeprecationWarning directing
users to the new API. Removal is scheduled for v2.0 (separate change,
no timeline).

The internal test suite is migrated as part of this PR, but that's
a project-internal concern, not a breaking change for downstream
consumers.

Checklist

• Conventional Commits prefix in the PR title (feat(testing):)
• CHANGELOG entry will be generated automatically by release-please
• Documentation updated (README "Testing" section, docstrings on every public name in cfn_handler.testing, docs/ROADMAP.md updated to reflect "shipped in v1.3.0")
• OpenSpec change opened (add-testing-helpers, validates --strict)

Commit-by-commit story (for review)

The squash-merge will collapse these, but each is reviewable on its own
and leaves the suite in a green state at every checkpoint:

1. chore(openspec): propose v1.3 testing helpers + roadmap — plan only
2. feat(testing): add cfn_handler.testing module with replay() and helpers — the implementation
3. feat(resource): deprecate test_mode and last_response in favour of replay() — DeprecationWarning + tests + temporary filterwarnings ignore for the migration window
4. test: migrate existing test suite from test_mode to replay() — bulk translation, removes the temporary filter
5. docs(readme): add Testing section showing replay() workflow
6. ci(justfile): skip CodeQL in gha-pre-release (act/CodeQL incompatibility) — tangential discovery, kept on the same branch since it directly affects validation of this PR

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ac19152e16

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve legacy poller callable arity in provisioning call

The provisioning path now always calls provision(..., None) with four positional arguments. Any existing monkey-patched setup_polling (or custom callable) that still uses the prior 3-argument shape will raise TypeError; this exception is not handled in this block (only CfnHandlerError is), so dispatch falls into the generic backstop and returns an internal-error failure instead of the expected polling behavior. Using keyword args (or omitting the explicit region when not needed) avoids this regression.

Useful? React with 👍 / 👎.