fix: harden engine, enrich failure diagnostics, close adoption gaps

- P0: CLI verify now honors  test budget with seeded multi-sample
- P0: Observe sampling enforced via Math.random() gate in hook-validator
- P1: Remove misleading undici-mock-agent isolation option
- P1: Qualify reuses shared discoverRouteDetails() with warnings
- P1: Chaos/scenario config exposed via preset schema
- P1: README/docs limitations updated to current state
- P2: Nested response annotations prefer 2xx deterministically
- P2: --changed documented as heuristic in verify.md

- Add observe sink tests (sampling 0/1, sink failure non-interference)
- Add verify runs regression tests (scale, determinism, variants)
- Add configured-scenario qualify test (independent of OAuth fixture)
- Add coverageBreakdown to qualify artifacts (per-gate route coverage)
- Add production-style observe example with real sink in docs/observe.md
- Add nightly/staging vs PR gating guidance to docs/qualify.md

- Enrich VerifyFailure with formula-aware diagnostics:
  status:201 => 'HTTP 200', body field checks => actual values
- Remove stale observe CLI activation message
- Document outbound mocks as process-global in getting-started.md
- Refresh APOPHIS_ADOPTION_AUDIT.md with current state

903 tests pass, build clean, typecheck clean.
This commit is contained in:
John Dvorak
2026-05-21 20:39:36 -07:00
parent 55b0262799
commit d0523fcc2d
128 changed files with 4004 additions and 3631 deletions
+44 -9
View File
@@ -4,14 +4,14 @@ Behavioral confidence for Fastify services.
APOPHIS checks whether route behavior holds across operations, states, and protocol flows.
Inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): instead of only checking payload shape, APOPHIS encodes intended behavior as executable contracts and verifies them with property-based and stateful testing.
Inspired by the concept of invariant-driven automated testing: instead of only checking payload shape, APOPHIS encodes intended behavior as executable contracts and verifies them with property-based and stateful testing.
Supported Node.js versions: 20.x and 22.x.
Supported Node.js versions: >=20.18.1 (20.x) and 22.x.
```bash
npm install apophis-fastify fastify @fastify/swagger
apophis init --preset safe-ci
apophis verify --profile quick --routes "POST /users"
npm install @apophis/fastify fastify @fastify/swagger
npx apophis init --preset safe-ci
npx apophis verify --profile quick --routes "POST /users"
```
`x-ensures` is an OpenAPI schema extension for behavioral contracts — statements about what a route must guarantee.
@@ -79,16 +79,16 @@ JSON Schema cannot express this relationship. APOPHIS turns it into an executabl
```bash
# 1. Install
npm install apophis-fastify fastify @fastify/swagger
npm install @apophis/fastify fastify @fastify/swagger
# 2. Scaffold
apophis init --preset safe-ci
npx apophis init --preset safe-ci
# 3. Verify
apophis verify --profile quick --routes "POST /users"
npx apophis verify --profile quick --routes "POST /users"
# 4. Doctor
apophis doctor
npx apophis doctor
```
See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough.
@@ -127,6 +127,41 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
- [LLM-Safe Adoption](docs/llm-safe-adoption.md) — Scaffolds and CI guards
- [Protocol Extensions](docs/attic/protocol-extensions-spec.md) — JWT, X.509, SPIFFE, WIMSE
## Recommended Integration
**New projects:** Use `createFastify()` to ensure route discovery is installed before any routes are registered.
```ts
import { createFastify } from '@apophis/fastify'
const app = await createFastify({
logger: true,
apophis: { runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off' },
})
// Register swagger, auth, plugins, and routes after app creation.
```
**Existing projects:** Register APOPHIS or install route discovery before routes. Run `apophis doctor` to verify routes are discovered with full schema metadata.
**Schema-less fallback:** If APOPHIS is registered after routes, `printRoutes()` can recover paths but not route schemas or behavioral contracts. `apophis doctor` and `apophis verify` will warn when discovery is schema-less.
## Current Limitations
These reflect current implementation behavior. All are actively tracked for improvement.
- **Route discovery requires ordering.** If the APOPHIS plugin or route discovery hook is not installed before routes are registered, behavioral contract annotations (x-ensures, x-requires, x-outbound, x-variants, x-timeout) cannot be recovered. Use `createFastify()` for new projects or register APOPHIS early.
- **Observe has two faces.** The runtime plugin supports non-blocking sink emission via `observe.enabled` + `observe.sinks` when registered programmatically. `apophis observe` CLI validates config readiness; it does not activate a long-running runtime observer. See `docs/observe.md` for the distinction between programmatic runtime observation and CLI config validation.
- **CLI verify samples once per contract by default.** Set `runs` in your preset to increase the number of property-based test samples per route. The programmatic `fastify.apophis.contract()` API supports the same `runs` configuration.
- **Outbound mocks are process-global.** The mock runtime patches `globalThis.fetch`. Only one mock runtime can be installed at a time. Run mock-dependent tests serially or isolate by process. Undici `MockAgent` integration is not yet implemented.
- **Qualify coverage depends on profile configuration.** Qualify runs scenario, stateful, and chaos checks based on profile gates. Chaos route selection uses the configured strategy (one/all/sample/routes).
## Compatibility
- **Fastify v5 only.** Fastify v4 and earlier are not supported.
- **ESM only.** This package is `"type": "module"` and does not provide a CommonJS build. Use `import` syntax.
- **Node.js `>=20.18.1 <21 || >=22 <23`**.
- **`@fastify/swagger` must be registered before routes** (APOPHIS auto-registers it if missing).
## License
MIT