fix: harden engine, enrich failure diagnostics, close adoption gaps
- P0: CLI verify now honors test budget with seeded multi-sample - P0: Observe sampling enforced via Math.random() gate in hook-validator - P1: Remove misleading undici-mock-agent isolation option - P1: Qualify reuses shared discoverRouteDetails() with warnings - P1: Chaos/scenario config exposed via preset schema - P1: README/docs limitations updated to current state - P2: Nested response annotations prefer 2xx deterministically - P2: --changed documented as heuristic in verify.md - Add observe sink tests (sampling 0/1, sink failure non-interference) - Add verify runs regression tests (scale, determinism, variants) - Add configured-scenario qualify test (independent of OAuth fixture) - Add coverageBreakdown to qualify artifacts (per-gate route coverage) - Add production-style observe example with real sink in docs/observe.md - Add nightly/staging vs PR gating guidance to docs/qualify.md - Enrich VerifyFailure with formula-aware diagnostics: status:201 => 'HTTP 200', body field checks => actual values - Remove stale observe CLI activation message - Document outbound mocks as process-global in getting-started.md - Refresh APOPHIS_ADOPTION_AUDIT.md with current state 903 tests pass, build clean, typecheck clean.
This commit is contained in:
@@ -4,14 +4,14 @@ Behavioral confidence for Fastify services.
|
||||
|
||||
APOPHIS checks whether route behavior holds across operations, states, and protocol flows.
|
||||
|
||||
Inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): instead of only checking payload shape, APOPHIS encodes intended behavior as executable contracts and verifies them with property-based and stateful testing.
|
||||
Inspired by the concept of invariant-driven automated testing: instead of only checking payload shape, APOPHIS encodes intended behavior as executable contracts and verifies them with property-based and stateful testing.
|
||||
|
||||
Supported Node.js versions: 20.x and 22.x.
|
||||
Supported Node.js versions: >=20.18.1 (20.x) and 22.x.
|
||||
|
||||
```bash
|
||||
npm install apophis-fastify fastify @fastify/swagger
|
||||
apophis init --preset safe-ci
|
||||
apophis verify --profile quick --routes "POST /users"
|
||||
npm install @apophis/fastify fastify @fastify/swagger
|
||||
npx apophis init --preset safe-ci
|
||||
npx apophis verify --profile quick --routes "POST /users"
|
||||
```
|
||||
|
||||
`x-ensures` is an OpenAPI schema extension for behavioral contracts — statements about what a route must guarantee.
|
||||
@@ -79,16 +79,16 @@ JSON Schema cannot express this relationship. APOPHIS turns it into an executabl
|
||||
|
||||
```bash
|
||||
# 1. Install
|
||||
npm install apophis-fastify fastify @fastify/swagger
|
||||
npm install @apophis/fastify fastify @fastify/swagger
|
||||
|
||||
# 2. Scaffold
|
||||
apophis init --preset safe-ci
|
||||
npx apophis init --preset safe-ci
|
||||
|
||||
# 3. Verify
|
||||
apophis verify --profile quick --routes "POST /users"
|
||||
npx apophis verify --profile quick --routes "POST /users"
|
||||
|
||||
# 4. Doctor
|
||||
apophis doctor
|
||||
npx apophis doctor
|
||||
```
|
||||
|
||||
See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough.
|
||||
@@ -127,6 +127,41 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
|
||||
- [LLM-Safe Adoption](docs/llm-safe-adoption.md) — Scaffolds and CI guards
|
||||
- [Protocol Extensions](docs/attic/protocol-extensions-spec.md) — JWT, X.509, SPIFFE, WIMSE
|
||||
|
||||
## Recommended Integration
|
||||
|
||||
**New projects:** Use `createFastify()` to ensure route discovery is installed before any routes are registered.
|
||||
|
||||
```ts
|
||||
import { createFastify } from '@apophis/fastify'
|
||||
|
||||
const app = await createFastify({
|
||||
logger: true,
|
||||
apophis: { runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off' },
|
||||
})
|
||||
// Register swagger, auth, plugins, and routes after app creation.
|
||||
```
|
||||
|
||||
**Existing projects:** Register APOPHIS or install route discovery before routes. Run `apophis doctor` to verify routes are discovered with full schema metadata.
|
||||
|
||||
**Schema-less fallback:** If APOPHIS is registered after routes, `printRoutes()` can recover paths but not route schemas or behavioral contracts. `apophis doctor` and `apophis verify` will warn when discovery is schema-less.
|
||||
|
||||
## Current Limitations
|
||||
|
||||
These reflect current implementation behavior. All are actively tracked for improvement.
|
||||
|
||||
- **Route discovery requires ordering.** If the APOPHIS plugin or route discovery hook is not installed before routes are registered, behavioral contract annotations (x-ensures, x-requires, x-outbound, x-variants, x-timeout) cannot be recovered. Use `createFastify()` for new projects or register APOPHIS early.
|
||||
- **Observe has two faces.** The runtime plugin supports non-blocking sink emission via `observe.enabled` + `observe.sinks` when registered programmatically. `apophis observe` CLI validates config readiness; it does not activate a long-running runtime observer. See `docs/observe.md` for the distinction between programmatic runtime observation and CLI config validation.
|
||||
- **CLI verify samples once per contract by default.** Set `runs` in your preset to increase the number of property-based test samples per route. The programmatic `fastify.apophis.contract()` API supports the same `runs` configuration.
|
||||
- **Outbound mocks are process-global.** The mock runtime patches `globalThis.fetch`. Only one mock runtime can be installed at a time. Run mock-dependent tests serially or isolate by process. Undici `MockAgent` integration is not yet implemented.
|
||||
- **Qualify coverage depends on profile configuration.** Qualify runs scenario, stateful, and chaos checks based on profile gates. Chaos route selection uses the configured strategy (one/all/sample/routes).
|
||||
|
||||
## Compatibility
|
||||
|
||||
- **Fastify v5 only.** Fastify v4 and earlier are not supported.
|
||||
- **ESM only.** This package is `"type": "module"` and does not provide a CommonJS build. Use `import` syntax.
|
||||
- **Node.js `>=20.18.1 <21 || >=22 <23`**.
|
||||
- **`@fastify/swagger` must be registered before routes** (APOPHIS auto-registers it if missing).
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
Reference in New Issue
Block a user