Turn a service failure into a permanent CI guard¶

This companion to the service evidence loop takes one recovery failure to a portable CI guard.

explore → coverage → exact replay → bounded finding → portable regression
        → verify --ci → workload-strength control

Before you start: name the promise¶

Tell Ordeal how to reach the service, which container it may disrupt, and what a correct response looks like:

[compose]
file = "compose.test.yaml"
base_url = "http://127.0.0.1:8080"
services = ["worker"]
faults = ["kill"]
replay_attempts = 3

[[compose.requests]]
name = "read_order"
path = "/orders/{order_id}"
expect_status = 200
expect_json = {"json.state" = "committed"}

expect_status and expect_json are business promises. Start with safe GET requests; mutating requests are not faultable by default.

1. Explore the real service¶

ordeal explore --runner compose -c ordeal.toml

Ordeal keeps the topology alive, introduces faults, waits for recovery, and reports operation × fault × property:

Status	Meaning
`PASS`	This exact combination ran and the promise held
`FAIL`	This exact combination ran and the promise broke
`NOT EXERCISED`	The combination did not run; it is still a gap

2. Demand repeated, exact replay¶

attempted 3 / reproduced 3 means three exact action signatures matched; external timing is not assumed deterministic.

For an existing trace, choose the count explicitly:

ordeal replay .ordeal/traces/compose-42-abc.json --attempts 10

3. Save only a bounded finding¶

ordeal explore --runner compose -c ordeal.toml --save-artifacts

Promotion requires a replay match on failure kind, message, action index, and action name. Ordeal then writes:

tests/ordeal-compose-regressions/fnd_compose_<id>.json
tests/ordeal-regressions.json

The portable trace and shared manifest use repository-relative paths and are CI-provider neutral.

Harness and configuration errors remain diagnostics, not service defects.

4. Prove red, then prove green¶

ordeal verify fnd_compose_<id> --allow-unsafe-artifacts

Before the fix, exit 1 proves the bound witness still catches the bug. After the fix, exit 0 requires every configured attempt to complete cleanly. A different failure is not accepted as a fix.

Commit the portable trace and manifest together, then guard all records:

ordeal verify --ci

verify --ci is read-only and provider-neutral. It rejects hashes or paths that escape the workspace. It needs Docker Compose and any credentials referenced by the trace.

5. Test that the workload can notice wrong answers¶

After the fixed control is clean, add a bounded budget:

[compose]
workload_mutations = 20

Ordeal first replays an unchanged response. If stable, it changes recorded expectations and checks the workload. killed means it caught the wrong answer; survived means it accepted it; inconclusive means the control was unstable.

This changes recorded expectations, not application code. It measures only observed trace prefixes.

Run the checked-in acceptance example¶

With Docker Compose available, run from the repository root:

uv run python scripts/verify_compose_evidence_loop.py \
  --output .artifacts/compose-evidence-loop.json
uv run python scripts/verify_compose_service_matrix.py \
  --recovery-report .artifacts/compose-evidence-loop.json \
  --output .artifacts/compose-service-matrix.json

The first fixture proves a kill/restart defect and normal-CLI post-fix control. The matrix adds two-service persistence across API restart and eight-way fan-out under delay and corruption faults. Every declared cell passes and each workload catches four wrong expectations.

Evidence boundary and deeper reference¶

This proves a bounded observation and post-fix control, not root cause, deterministic scheduling, or universal correctness.