fix: harden engine, enrich failure diagnostics, close adoption gaps

- P0: CLI verify now honors  test budget with seeded multi-sample
- P0: Observe sampling enforced via Math.random() gate in hook-validator
- P1: Remove misleading undici-mock-agent isolation option
- P1: Qualify reuses shared discoverRouteDetails() with warnings
- P1: Chaos/scenario config exposed via preset schema
- P1: README/docs limitations updated to current state
- P2: Nested response annotations prefer 2xx deterministically
- P2: --changed documented as heuristic in verify.md

- Add observe sink tests (sampling 0/1, sink failure non-interference)
- Add verify runs regression tests (scale, determinism, variants)
- Add configured-scenario qualify test (independent of OAuth fixture)
- Add coverageBreakdown to qualify artifacts (per-gate route coverage)
- Add production-style observe example with real sink in docs/observe.md
- Add nightly/staging vs PR gating guidance to docs/qualify.md

- Enrich VerifyFailure with formula-aware diagnostics:
  status:201 => 'HTTP 200', body field checks => actual values
- Remove stale observe CLI activation message
- Document outbound mocks as process-global in getting-started.md
- Refresh APOPHIS_ADOPTION_AUDIT.md with current state

903 tests pass, build clean, typecheck clean.
This commit is contained in:
John Dvorak
2026-05-21 20:39:36 -07:00
parent 55b0262799
commit d0523fcc2d
128 changed files with 4004 additions and 3631 deletions
+22 -11
View File
@@ -1,22 +1,33 @@
# APOPHIS Setup — safe-ci preset
# APOPHIS Setup — llm-safe preset
This project was scaffolded with `apophis init --preset safe-ci`.
This project was scaffolded with `apophis init --preset llm-safe`.
## Quick Start
1. Confirm the Fastify app registers `@fastify/swagger`.
1. Ensure you have a Fastify app with @fastify/swagger registered.
2. Add behavioral contracts to your route schemas using `x-ensures`.
3. Run: apophis verify --profile quick
3. Run: apophis verify --profile llm-check
## What This Preset Does
- Runs only behavioral contracts (not schema-only routes).
- No chaos, no observe, no stateful testing.
- Safe for CI pipelines.
- Timeout: 5s per route.
- Ultra-minimal preset for LLM-generated codebases.
- 3s timeout per route (fast feedback).
- No observe, no qualify, no chaos — verify only.
- Conservative defaults to avoid surprising failures.
## Example Behavioral Contract
Add this inside your route schema to check that a created resource is retrievable:
```javascript
"x-ensures": [
"response_code(GET /users/{response_body(this).id}) == 200"
]
```
If `apophis verify` says "No behavioral contracts found", it means your routes have schemas but no `x-ensures` or `x-requires` clauses. Add at least one clause per route you want to verify.
## Next Steps
- Add more routes to the `routes` array in your profile.
- Try `apophis init --preset platform-observe` to configure observe-mode policy and runtime drift reporting.
- Try `apophis init --preset protocol-lab` for multi-step flows.
- Add routes to the `routes` array once you have behavioral contracts.
- Run `apophis doctor` to check for missing dependencies.