From 1de735ee081288b69d3d4eca506a1d670eb8b745 Mon Sep 17 00:00:00 2001 From: John Dvorak Date: Fri, 22 May 2026 11:05:52 -0700 Subject: [PATCH] feat: plugin contract e2e, qualify --changed, production observe, regressions --- APOPHIS_ADOPTION_AUDIT.md | 37 +- README.md | 7 +- docs/observe.md | 51 ++- docs/plugin-contracts.md | 431 ++++++++++++++++++ .../plugin-contracts/apophis.config.js | 4 + src/cli/__fixtures__/plugin-contracts/app.js | 34 ++ .../plugin-contracts/package.json | 1 + src/cli/commands/qualify/chaos-handler.ts | 20 +- src/cli/commands/qualify/index.ts | 59 ++- src/cli/commands/qualify/runner.ts | 15 +- src/cli/commands/qualify/scenario-handler.ts | 2 +- src/cli/commands/qualify/stateful-handler.ts | 2 +- src/cli/commands/replay/index.ts | 29 ++ src/cli/commands/verify/index.ts | 54 +-- src/cli/commands/verify/runner.ts | 144 +++++- src/cli/core/config-loader.ts | 2 +- src/cli/core/context.ts | 2 +- src/cli/core/index.ts | 11 +- src/cli/core/types.ts | 1 + src/cli/renderers/human.ts | 28 +- src/domain/contract-validation.ts | 4 + src/domain/plugin-contracts.ts | 13 +- src/fastify-factory.ts | 17 +- src/formula/parser.ts | 35 +- src/plugin/builders.ts | 5 +- src/plugin/index.ts | 22 +- src/quality/petit-runner.ts | 14 +- src/quality/scenario-runner.ts | 38 +- src/quality/stateful-request-execution.ts | 11 + src/quality/stateful-runner.ts | 34 +- src/test/cli/machine-output-contracts.test.ts | 157 +++++++ src/test/formula.test.ts | 32 ++ src/test/integration.test.ts | 196 ++++++++ src/types/core.ts | 2 + 34 files changed, 1392 insertions(+), 122 deletions(-) create mode 100644 docs/plugin-contracts.md create mode 100644 src/cli/__fixtures__/plugin-contracts/apophis.config.js create mode 100644 src/cli/__fixtures__/plugin-contracts/app.js create mode 100644 src/cli/__fixtures__/plugin-contracts/package.json diff --git a/APOPHIS_ADOPTION_AUDIT.md b/APOPHIS_ADOPTION_AUDIT.md index df52e89..8a473d3 100644 --- a/APOPHIS_ADOPTION_AUDIT.md +++ b/APOPHIS_ADOPTION_AUDIT.md @@ -8,11 +8,11 @@ This audit is based on code inspection plus command verification, not documentat ## Executive Summary -APOPHIS has real product value. It is not just a schema wrapper: it gives Fastify teams a way to express and verify behavioral API promises that OpenAPI/JSON Schema cannot cover, especially cross-route invariants such as create/read consistency, delete semantics, auth/session flows, state transitions, idempotency, outbound dependency expectations, and replayable counterexamples. +APOPHIS has real product value. It is not just a schema wrapper: it gives Fastify teams a way to express, verify, and observe behavioral API promises that OpenAPI/JSON Schema cannot cover, especially cross-route invariants such as create/read consistency, delete semantics, auth/session flows, state transitions, idempotency, outbound dependency expectations, and replayable counterexamples. I would adopt APOPHIS today as a focused behavioral verification tool for Fastify v5 ESM services. I would start with CI `verify` and a small number of high-value contracts, then expand into `qualify` and runtime observation once the team has clear operating guidance. -I would not yet treat it as a complete production observability platform or a turnkey organization-wide release gate. The core implementation is strong, but the remaining value gap is mostly around operational maturity: standalone observe activation, deeper tests around recent CLI behavior, richer scenario authoring, and clearer release-gate recommendations. +I would not yet treat it as a complete production observability platform or a turnkey organization-wide release gate. The core implementation is strong, but the remaining value gap is mostly around operational maturity: standalone observe process management, richer scenario authoring, and organization-specific release-gate policy. Adoption verdict: strong team pilot candidate, credible standardization candidate after the remaining gaps below are addressed. @@ -34,10 +34,10 @@ Observed results: |---|---:| | Typecheck | pass | | Build | pass | -| Source tests | 587 pass, 0 fail | -| CLI tests | 311 pass, 0 fail | +| Source tests | 590 pass, 0 fail | +| CLI tests | 320 pass, 0 fail | | Docs smoke tests | 4 pass, 0 fail | -| Total tests | 902 pass, 0 fail | +| Total tests | 921 pass, 0 fail | The working tree contains many broader project changes unrelated to this audit. This document evaluates the current working tree state. @@ -51,7 +51,7 @@ Mostly yes for behavioral verification. Partially for production observation and | Deterministic CI verification | Yes, materially. CLI `verify` now honors configured `runs`, uses seeded request generation, emits artifacts, supports route filters, replay metadata, and machine-readable output. | | Cross-route behavior | Yes for supported formula operations and route-call semantics. This is the most differentiated value. | | Runtime validation | Yes when the plugin is explicitly configured outside production. Production enforcement is intentionally blocked. | -| Runtime observation | Partially. Programmatic plugin observation exists and emits non-blocking sink events with sampling. The CLI validates/report readiness but does not attach to or run a service. | +| Runtime observation | Yes for programmatic production-safe hooks. APOPHIS emits non-blocking sink events with sampling in production when `observe.enabled` and sinks are configured. The CLI validates/report readiness but does not attach to or run a service. | | Stateful/scenario/chaos qualification | Partially. The runner and artifacts are useful, route discovery is now shared with verify, and config supports scenarios/chaos knobs. Scenario authoring is still young and needs more real-world examples/tests. | | Outbound dependency mocking | Useful but intentionally process-global. The misleading scoped `undici-mock-agent` option has been removed. Teams still need careful test isolation. | | Team-safe onboarding | Good. The package has CLI help, init/doctor/replay/verify/qualify/observe, config validation, machine output, docs smoke tests, packaging tests, and production safety checks. | @@ -102,12 +102,17 @@ The following earlier adoption risks have been addressed in the current working |---|---| | CLI `verify` runs | `VerifyRunnerDeps` accepts `runs`; `verifyCommand()` passes resolved config; `runVerify()` executes contracts for `contractRuns`. | | Observe sampling | `hook-validator.ts` gates sink emission using `opts.observe.sampling` before emitting pass/violation/error events. | +| Production observe activation | `apophisPlugin` now keeps blocking runtime validation disabled in production while allowing non-blocking observe sinks to emit pass/violation/error events. | | Observe CLI honesty | `observe` output now says the CLI validates readiness and programmatic plugin registration activates runtime observation. | | Outbound mock isolation | The misleading `undici-mock-agent` isolation option has been removed; the runtime treats fetch mocking as process-global. | | Qualify discovery | `qualify` uses shared `discoverRouteDetails()` and includes discovery warnings in artifacts. | | Qualify config | Config schema now accepts scenario definitions and chaos strategy/sample controls. | | Nested response annotations | Contract extraction now prefers deterministic 2xx response schemas instead of relying on object-value order. | | `--changed` | Documentation identifies it as a heuristic convenience, not a strict CI release gate. | +| Plugin contracts (end-to-end) | Full pipeline: config schema, plugin registration, compose+merge in all runners, precondition→skip, auto-inject headers, source attribution (`formulaSources`), failure counting, `drainWarnings()` collection, production safety. Wired through verify, qualify (scenario/stateful/chaos), and replay. | +| Artifact pipeline CI/CD | 6 CI-facing regression tests: json-summary parseable, ndjson-summary parseable, `--quiet` persistence, skipped field presence, exit code 0 on pass, qualify json-summary. Verify→replay round-trip test with plugin contracts. | +| CLI output hygiene | Console.warn bleeding fixed (`drainWarnings`); `json-summary`→`human` format normalization bug fixed; `--quiet` no longer suppresses machine format output. | +| Qualify --changed | Qualify now supports `--changed` flag with same git-diff heuristic as verify. Prints match count, exits 0 when no changed routes. | ## Remaining Adoption Gaps @@ -121,7 +126,7 @@ The implementation supports runtime observation only when the application explic - Integration tests prove sink sync failures and async rejections never change route responses. - Integration tests prove sampling: 0 suppresses all events; sampling: 1 emits expected `contract.pass`/`contract.violation` events. -**Still open:** A future `apophis observe --app ./app.ts` mode that activates a running service observer. +**Still open:** A future `apophis observe --app ./app.ts` mode that imports and starts a service for local/staging smoke observation. Production observation itself is now programmatic and active through plugin registration. ### P1: Recent `verify` Runs Behavior Now Has Regression Tests @@ -130,7 +135,7 @@ The implementation supports runtime observation only when the application explic - Regression test proves `runs: 5` scales multiplicatively from `runs: 1`. - Regression test proves `runs: 10` is deterministic at the same seed. -**Still open:** Variant-aware runs test (verifying run budget is per-variant or shared). +**Completed:** Variant-aware runs regression proves the run budget is applied per variant. ### P1: Qualify Product Shape Improved @@ -139,7 +144,7 @@ The implementation supports runtime observation only when the application explic - Configured-scenario qualify test added (independent of OAuth fixture routes). - `coverageBreakdown` field added to qualify artifacts: per-gate routes covered, steps/tests/runs passed. -**Still open:** Clear guidance for nightly/staging use versus pull-request gating in qualify docs. +**Completed:** `docs/qualify.md` now documents pull-request versus nightly/staging gate guidance. ### P1: Outbound Mocks Process-Global, Honestly Documented @@ -210,14 +215,14 @@ High-value first contracts: | Fastify fit | 8/10 | Strong plugin/inject/decorator alignment; discovery order still matters. | | Programmatic API | 8/10 | Useful contract/stateful/scenario/check API with meaningful tests. | | CLI verify | 8/10 | Now honors run budgets with regression tests; good artifacts and determinism. | -| Observe | 7/10 | Runtime sink primitives, sampling, and sink-failure-resilience exist with tests. Production-style docs added. Standalone operational story not complete. | +| Observe | 8/10 | Production-safe non-blocking sink emission, sampling, and sink-failure-resilience exist with tests. Standalone process-management story is still future work. | | Qualify | 7/10 | Improved discovery/config/scenarios. Coverage breakdown in artifacts. Needs richer scenario examples and gating guidance. | | Outbound mocking | 7/10 | Useful and honest about process-global behavior. Docs and README explicit. True scoped mocking remains future work. | | Docs | 8/10 | Broad and increasingly precise. Observe and qualify docs expanded with real code examples. | | Packaging | 9/10 | Strong for a Node/Fastify package. | | Team readiness | 8/10 | Ready for pilot and selective CI use with regression-locked verification behavior. | -Overall: 8/10 for real team pilot use. Potential 9/10 if observe gains a clearer production story and qualify gets first-class CI workflow guidance. +Overall: 8.5/10 for real team pilot use. Potential 9/10 if standalone observe process management and richer scenario libraries become first-class. ## Highest-Impact Next Work @@ -226,9 +231,13 @@ Overall: 8/10 for real team pilot use. Potential 9/10 if observe gains a clearer 3. ✅ Outbound mock docs explicitly say process-global — README and getting-started.md updated. 4. ✅ Qualify scenario config documented with full examples in qualify.md. 5. ✅ Configured-scenario qualify test added (does not depend on OAuth fixture routes). -6. Add full production-style observe example with a real collector sink implementation. -7. Improve qualify artifact coverage summaries to distinguish route-contract, scenario, stateful, and chaos coverage more clearly. -8. Consider true scoped outbound mocking (undici dispatcher) only if concurrent in-process dependency tests become a core promise. +6. ✅ Full production-style observe example with real collector sink implementation added to docs/observe.md. +7. ✅ Plugin contract support end-to-end: docs, tests, all runners wired. +8. ✅ Artifact pipeline CI/CD regression tests: json-summary, ndjson-summary, --quiet, skipped field, exit codes. +9. ✅ Qualify --changed implemented. +10. Add standalone observe process management (`apophis observe --app ./app.ts`) for local/staging observation. +11. Add route ownership / file-to-route maps for precise `--changed` filtering. +12. Consider true scoped outbound mocking (undici dispatcher) only if concurrent in-process dependency tests become a core promise. ## Bottom Line diff --git a/README.md b/README.md index 3cd5903..e15dc09 100644 --- a/README.md +++ b/README.md @@ -136,7 +136,12 @@ import { createFastify } from '@apophis/fastify' const app = await createFastify({ logger: true, - apophis: { runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off' }, + apophis: { + runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off', + observe: process.env.NODE_ENV === 'production' + ? { enabled: true, sampling: 0.1, sinks: [metricsSink] } // your ObserveSink + : undefined, + }, }) // Register swagger, auth, plugins, and routes after app creation. ``` diff --git a/docs/observe.md b/docs/observe.md index 6b53046..1642ee5 100644 --- a/docs/observe.md +++ b/docs/observe.md @@ -162,30 +162,19 @@ export default { ## Programmatic Runtime Activation The CLI only validates configuration. To activate runtime observation, register -APOPHIS with observe options in your application: +APOPHIS with observe options in your application before routes are registered. +Observe remains active in production because it is non-blocking; blocking +runtime validation still stays disabled in production. ```typescript import Fastify from 'fastify' import apophisPlugin from '@apophis/fastify' +import type { ObserveSink, ObserveEvent } from '@apophis/fastify' const app = Fastify({ logger: true }) -// Register APOPHIS with observe enabled. -// This emits non-blocking contract pass/violation/error events -// for every covered request, gated by sampling. -await app.register(apophisPlugin, { - runtime: 'warn', - observe: { - enabled: true, - sampling: 0.1, // observe 10% of requests - sinks: [metricsSink], - }, -}) - // Implement the ObserveSink interface. // Capture events to your preferred observability backend. -import type { ObserveSink, ObserveEvent } from '@apophis/fastify' - const metricsSink: ObserveSink = { emit(event: ObserveEvent) { // Emit a counter for each contract evaluation @@ -205,11 +194,43 @@ const metricsSink: ObserveSink = { } }, } + +// Register APOPHIS with observe enabled. +// This emits non-blocking contract pass/violation/error events +// for every covered request, gated by sampling. +await app.register(apophisPlugin, { + runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off', + observe: { + enabled: true, + sampling: 0.1, // observe 10% of requests + sinks: [metricsSink], + }, +}) +``` + +For new services, `createFastify()` wires discovery and APOPHIS before your +routes, which avoids the most common ordering mistake: + +```typescript +import { createFastify } from '@apophis/fastify' + +const app = await createFastify({ + logger: true, + apophis: { + runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off', + observe: { + enabled: true, + sampling: 0.1, + sinks: [metricsSink], + }, + }, +}) ``` Key constraints: - Sink `emit()` can be sync or async (returns `void | Promise`). - Sink rejections and thrown errors are silently caught — they never affect the route response or status code. +- In production, observe hooks still run when `observe.enabled` and `observe.sinks` are configured; blocking runtime validation does not. - Sampling is applied per-formula evaluation via `Math.random() < sampling`. At `sampling: 1` every formula is emitted. At `sampling: 0` nothing is emitted. - Only routes with APOPHIS annotations (`x-ensures`, `x-requires`) produce events. diff --git a/docs/plugin-contracts.md b/docs/plugin-contracts.md new file mode 100644 index 0000000..6dcc4d0 --- /dev/null +++ b/docs/plugin-contracts.md @@ -0,0 +1,431 @@ +# Plugin Contracts + +Define behavioral rules that apply across all matching routes and are automatically injected into every matching route's contract evaluation. + +Plugin contracts let teams codify cross-cutting expectations — "all API routes must carry an auth header," "all POST handlers must close the database connection," "all responses must include a request ID." They're defined once in config and evaluated alongside route-level contracts without per-route boilerplate. + +## Table of Contents + +- [When to Use It](#when-to-use-it) +- [Config Format](#config-format) +- [PluginContractSpec Reference](#plugincontractspec-reference) +- [Route Pattern Matching](#route-pattern-matching) +- [Precondition → Skip Semantics](#precondition--skip-semantics) +- [Auto-Injected Headers](#auto-injected-headers) +- [Failure Attribution](#failure-attribution) +- [Extension Dependencies](#extension-dependencies) +- [Production Safety](#production-safety) +- [Complete Example](#complete-example) +- [Limitations](#limitations) + +## When to Use It + +Plugin contracts are ideal for rules that span many routes and don't belong on a single handler: + +- Auth header presence across all API routes +- Response metadata (request IDs, timing headers, version stamps) +- Tenant isolation checks that apply to every route in a prefix +- Database connection lifecycle guards on write endpoints +- Content-type enforcement on all response paths + +Don't use plugin contracts for route-specific behavioral rules — those belong on the route schema via `x-requires` and `x-ensures`. + +## Config Format + +### apophis.config.js + +```javascript +export default { + pluginContracts: { + "auth-presence": { + appliesTo: "/api/**", + hooks: { + onRequest: { + requires: ["request_headers(this).authorization != null"] + } + } + }, + "rate-limit-respected": { + appliesTo: "POST /api/**", + hooks: { + onResponse: { + ensures: ["status != 429"] + } + }, + meta: { name: "rate-limit-respected", description: "Rate limit should not be hit under normal CI load" } + } + } +} +``` + +### Programmatic (Fastify plugin registration) + +```javascript +await app.register(apophisPlugin, { + pluginContracts: { + "auth-presence": { + appliesTo: "/api/**", + hooks: { + onRequest: { requires: ["request_headers(this).authorization != null"] } + } + } + } +}) +``` + +Contracts registered programmatically take the same path through the plugin contract registry as config-defined contracts. Both sources are merged into the composed contract at evaluation time. + +## PluginContractSpec Reference + +| Field | Required | Type | Description | +|---|---|---|---| +| `appliesTo` | yes | `string` | Route pattern (see [Route Pattern Matching](#route-pattern-matching)) | +| `hooks` | yes | `object` | Hook-phase contracts keyed by Fastify lifecycle hook name | +| `hooks..requires` | no | `string[]` | APOSTL formulas that must be `true` **before** this phase executes | +| `hooks..ensures` | no | `string[]` | APOSTL formulas that must be `true` **after** this phase executes | +| `meta` | no | `object` | Display metadata | +| `meta.name` | no | `string` | Human-readable plugin name | +| `meta.version` | no | `string` | Plugin version | +| `meta.description` | no | `string` | Description of what the plugin enforces | +| `extensions` | no | `array` | APOPHIS extensions this plugin depends on | + +### Supported hook phases + +Keys in `hooks` correspond to Fastify lifecycle hook names: + +- `onRequest` — evaluated before the route handler runs +- `preHandler` — evaluated before the handler, after shared hooks +- `preSerialization` — evaluated before the response body is serialized +- `onSend` — evaluated before the response is sent +- `onResponse` — evaluated after the response is sent (body already consumed) + +At least one of `requires` or `ensures` must be present per phase. Empty phases are ignored. + +### Extension dependencies + +```javascript +pluginContracts: { + "jwt-auth": { + appliesTo: "/api/**", + extensions: [{ name: "jwt-decoder", required: true }], + hooks: { + onRequest: { requires: ["decode_jwt(this).valid == true"] } + } + } +} +``` + +If a required extension is missing at registration time, the plugin's contracts are skipped with a warning. Extensions are resolved from the APOPHIS extension registry at evaluation time, not at route discovery time. + +## Route Pattern Matching + +The `appliesTo` field controls which routes a plugin contract covers. + +### Pattern syntax + +| Pattern | Matches | Does NOT match | +|---|---|---| +| `"/api/users"` | `/api/users` (exact) | `/api/users/42`, `/api/v2/users` | +| `"/api/**"` | `/api/users`, `/api/users/42`, `/api/users/:id/posts` | `/v2/api/users` | +| `"/api/*"` | `/api/users`, `/api/status` | `/api/users/42`, `/api/users/:id/posts` | +| `"**"` | Everything | — | +| `"POST /api/**"` | `POST /api/users`, `POST /api/orders/42` | `GET /api/users`, `POST /v2/api/users` | + +The `/**` suffix matches any path at any depth below the prefix. The `/*` suffix matches exactly one segment below the prefix. + +### Method-prefixed patterns + +Prepend the HTTP method to restrict a pattern to a specific verb: + +```javascript +{ + appliesTo: "POST /api/**", + hooks: { + preHandler: { requires: ["request_body(this) != null"] } + } +} +``` + +When a method is specified, the pattern only matches routes registered with that HTTP method. The method must be a valid HTTP method in uppercase. + +Pattern matching is evaluated against the route's path string only (the raw argument passed to `fastify.get()`, `fastify.post()`, etc.), not against the dereferenced URL with prefix applied. Fastify prefix application happens before pattern matching, so a route registered as `fastify.get("/users")` under a `prefix: "/api"` is matched against `/api/users`. + +## Precondition → Skip Semantics + +When a plugin contract defines `requires` formulas, the runner evaluates them before `ensures`. If any require formula fails: + +1. Require failure is **not** counted as a contract failure +2. All `ensures` formulas for that route+run are **skipped** +3. The skip is counted separately from failures in the artifact summary + +This distinguishes "the plugin's prerequisites aren't met" from "the route's behavior violated the contract." + +### Example + +```javascript +pluginContracts: { + "tenant-check": { + appliesTo: "/api/**", + hooks: { + onRequest: { + requires: ["request_headers(this).x-tenant-id != null"], + ensures: ["status != 500"] + } + } + } +} +``` + +For a route without the `x-tenant-id` header: +- The `requires` check fails → the `ensures` (`status != 500`) is **skipped** +- `skippedCount` increments; `failedCount` does not + +For a route with the `x-tenant-id` header: +- The `requires` check passes → the `ensures` is evaluated normally +- If `status == 500`, it's counted as a plugin contract **failure** + +## Auto-Injected Headers + +To avoid every test being skipped when a plugin contract checks for header presence, APOPHIS automatically injects headers into test requests when the precondition pattern is simple enough. + +### Supported patterns + +| Formula pattern | Injected header | +|---|---| +| `request_headers(this). != null` | `: test-value` | +| `request_headers(this). == "value"` | `: value` | + +### Example + +```javascript +pluginContracts: { + "auth-presence": { + appliesTo: "/api/**", + hooks: { + onRequest: { + requires: ["request_headers(this).authorization != null"] + } + } + } +} +``` + +APOPHIS detects the `!= null` pattern on `authorization` and injects `authorization: test-value` into every test request. The precondition passes automatically without user intervention. + +Auto-injection works in all runners: verify, qualify (scenario, stateful, chaos), and replay. + +### Unsupported patterns + +Complex preconditions are not auto-injected. For these, the requires check will fail and the route's ensures will be skipped: + +```javascript +// NOT auto-injected — the route's ensures will be skipped +requires: ["request_headers(this).x-count >= 5"] +``` + +For these cases, provide the header explicitly via route variants: + +```javascript +await fastify.apophis.contract({ + runs: 5, + variants: [{ name: "with-count", headers: { "x-count": "10" } }] +}) +``` + +## Failure Attribution + +When a plugin contract formula fails, the violation carries metadata that distinguishes it from route-level failures. + +### ContractViolation fields + +| Field | Value | Meaning | +|---|---|---| +| `source` | `"plugin:auth-presence"` | Which plugin added the formula | +| `phase` | `"onRequest"` | Which hook phase the formula belongs to | + +In human-readable output, plugin failures appear separately from route contract failures: + +```text +Plugin contract violation (plugin:auth-presence) +GET /api/users +Phase: onRequest + +Expected + response_headers(this).x-request-id != null + +Observed + response_headers(this).x-request-id was null +``` + +The artifact summary tracks plugin contract outcomes independently: + +```json +{ + "summary": { + "pluginContractsApplied": 12, + "pluginContractsFailed": 2, + "failed": 1, + "passed": 10 + } +} +``` + +- `pluginContractsApplied` — total plugin contract formulas evaluated across all routes +- `pluginContractsFailed` — plugin contract formulas that failed +- `failed` — total failures (route + plugin) +- `passed` — route-level contract passes (plugin-only formulas not counted here) + +A route can pass its own contracts and still have plugin contract failures. The CLI exit code reflects the aggregate: any failure (route or plugin) produces exit code 1. + +## Extension Dependencies + +Plugin contracts can declare dependencies on APOPHIS extensions. Extension resolution happens at evaluation time via the extension registry. + +### Declaring dependencies + +```javascript +pluginContracts: { + "jwt-auth": { + appliesTo: "/api/**", + extensions: [{ name: "jwt-decoder", required: true }], + hooks: { + onRequest: { requires: ["decode_jwt(this).valid == true"] } + } + } +} +``` + +Each extension reference has: +- `name` (required) — matches the extension's registered name +- `required` (optional, default `true`) — if `false`, missing extension emits a warning but does not skip the plugin + +### Missing extension behavior + +If a required extension is not registered, the plugin's contracts are **skipped** for all matching routes. Warnings are collected in the registry and surfaced via `drainWarnings()`: + +```text +Plugin 'jwt-auth' requires extensions [jwt-decoder] which are not registered. Skipping its contracts. +``` + +Warnings appear in the CLI output and the artifact's `warnings` array. + +### Extension registration + +Extensions are registered alongside plugin contracts: + +```javascript +await app.register(apophisPlugin, { + extensions: [ + { + name: "jwt-decoder", + headers: ["decode_jwt"], + predicates: { + decode_jwt: (ctx) => ({ value: true, success: true }) + } + } + ], + pluginContracts: { + "jwt-auth": { + appliesTo: "/api/**", + extensions: [{ name: "jwt-decoder", required: true }], + hooks: { + onRequest: { requires: ["decode_jwt(this).valid == true"] } + } + } + } +}) +``` + +## Production Safety + +`pluginContracts` is a **test-only** feature. If `pluginContracts` is non-empty in a production environment (`NODE_ENV=production`), APOPHIS throws at plugin registration: + +```text +APOPHIS: Unsafe options detected in production: pluginContracts. +These features are test-only and must not be enabled in production. +Remove them from ApophisOptions or set NODE_ENV=test. +``` + +This is enforced by `validateProductionSafety()` before any contract evaluation runs. There is no escape hatch. + +## Complete Example + +### Config (`apophis.config.js`) + +```javascript +export default { + profiles: { + quick: { preset: "safe-ci" }, + }, + presets: { + "safe-ci": { runs: 10 }, + }, + pluginContracts: { + "auth-required": { + appliesTo: "/api/**", + hooks: { + onRequest: { + requires: ["request_headers(this).authorization != null"], + }, + }, + meta: { name: "auth-required", description: "All API routes require an Authorization header" }, + }, + "request-id": { + appliesTo: "/api/**", + hooks: { + onSend: { + ensures: ["response_headers(this).x-request-id != null"], + }, + }, + meta: { name: "request-id", description: "All API responses must include x-request-id" }, + }, + }, +} +``` + +### Route (`routes/users.js`) + +```javascript +app.get("/api/users", { + schema: { + "x-category": "observer", + "x-ensures": ["status:200", "response_body(this) is Array"], + response: { + 200: { + type: "array", + items: { type: "object", properties: { id: { type: "number" } } }, + }, + }, + }, +}, async (request, reply) => { + return [{ id: 1 }, { id: 2 }] +}) +``` + +### Execution (`apophis verify`) + +1. Sends a request to `GET /api/users` +2. `auth-required` plugin: `onRequest.requires` → `request_headers(this).authorization != null` → APOPHIS auto-injects `authorization: test-value` → passes +3. Route-level ensures: `status:200` → passes; `response_body(this) is Array` → passes +4. `request-id` plugin: `onSend.ensures` → `response_headers(this).x-request-id != null` → if the handler doesn't set `x-request-id`, this **fails** as a `plugin:request-id` contract violation + +```text +Plugin contract violation (plugin:request-id) +GET /api/users +Phase: onSend + +Expected + response_headers(this).x-request-id != null + +Observed + response_headers(this).x-request-id was null +``` + +## Limitations + +- **`appliesTo` matches by route path only** by default. Use method-prefixed syntax (`"POST /api/**"`) for HTTP-method-specific rules. +- **Header auto-injection only handles simple patterns** (`!= null` and `== "value"`). Complex preconditions like `>= 5` are not auto-injected; the route's ensures will be skipped. +- **No per-route opt-out.** If a plugin contract matches a route, it always applies. Use narrower `appliesTo` patterns or separate route prefixes to exclude routes. +- **Extension resolution is lazy.** Extensions are checked at route evaluation time, not at registration time. A missing extension is only discovered when a matching route is evaluated. +- **At least one of `requires` or `ensures` must be present per phase.** A phase with neither is silently ignored. +- **Runtime hook ordering follows Fastify's lifecycle.** Plugin contracts in `onResponse` cannot inspect the response body (it has already been sent to the client). diff --git a/src/cli/__fixtures__/plugin-contracts/apophis.config.js b/src/cli/__fixtures__/plugin-contracts/apophis.config.js new file mode 100644 index 0000000..bbc1d9e --- /dev/null +++ b/src/cli/__fixtures__/plugin-contracts/apophis.config.js @@ -0,0 +1,4 @@ +export default { + profiles: { quick: { preset: "safe-ci" } }, + presets: { "safe-ci": { runs: 3 } }, +} diff --git a/src/cli/__fixtures__/plugin-contracts/app.js b/src/cli/__fixtures__/plugin-contracts/app.js new file mode 100644 index 0000000..874f30d --- /dev/null +++ b/src/cli/__fixtures__/plugin-contracts/app.js @@ -0,0 +1,34 @@ +import Fastify from "fastify"; + +const app = Fastify({ logger: false }); + +await app.register(import("@fastify/swagger"), { + openapi: { info: { title: "Plugin Contracts Fixture", version: "1.0.0" } }, +}); + +let apophisPlugin; +try { + ({ default: apophisPlugin } = await import("../../../index.js")); +} catch { + ({ default: apophisPlugin } = await import("../../../../dist/index.js")); +} + +await app.register(apophisPlugin, { + runtime: "off", + pluginContracts: { + "required-header": { + appliesTo: "/api/**", + hooks: { onRequest: { requires: ["request_headers(this).x-custom != null"] } }, + }, + }, +}); + +app.get("/api/data", { + schema: { + "x-category": "observer", + "x-ensures": ["status:200"], + response: { 200: { type: "object", properties: { ok: { type: "boolean" } } } }, + }, +}, async () => ({ ok: true })); + +export default app; diff --git a/src/cli/__fixtures__/plugin-contracts/package.json b/src/cli/__fixtures__/plugin-contracts/package.json new file mode 100644 index 0000000..658f10a --- /dev/null +++ b/src/cli/__fixtures__/plugin-contracts/package.json @@ -0,0 +1 @@ +{"name":"plugin-contracts-fixture","type":"module","private":true} \ No newline at end of file diff --git a/src/cli/commands/qualify/chaos-handler.ts b/src/cli/commands/qualify/chaos-handler.ts index 3defbf0..4d2e6eb 100644 --- a/src/cli/commands/qualify/chaos-handler.ts +++ b/src/cli/commands/qualify/chaos-handler.ts @@ -63,8 +63,26 @@ export async function runChaosOnRoute( const nonDelayEvents = events.filter(e => e.type !== 'inbound-delay') const application = applyChaosToExecution(ctx, nonDelayEvents) - // 5. Validate post-chaos contracts: did the route still satisfy its contracts? + // Validate route contracts after chaos. Compose plugin contracts if available. let contractsPassed = true + if (deps.pluginContractRegistry) { + const composed = deps.pluginContractRegistry.composeContracts(route) + if (!route.formulaSources) route.formulaSources = {} + for (const phase of Object.values(composed.phases)) { + for (const req of phase.requires) { + if (!route.requires.includes(req.formula)) { + route.requires.push(req.formula) + route.formulaSources[req.formula] = req.source + } + } + for (const ens of phase.ensures) { + if (!route.ensures.includes(ens.formula)) { + route.ensures.push(ens.formula) + route.formulaSources[ens.formula] = ens.source + } + } + } + } if (application.applied && route.ensures.length > 0) { const { validatePostconditionsAsync } = await import('../../../domain/contract-validation.js') const result = await validatePostconditionsAsync( diff --git a/src/cli/commands/qualify/index.ts b/src/cli/commands/qualify/index.ts index 3b63b6c..338f247 100644 --- a/src/cli/commands/qualify/index.ts +++ b/src/cli/commands/qualify/index.ts @@ -30,6 +30,7 @@ import { type StepTrace, type CleanupFailure, } from './runner.js' +import { isGitRepo, filterChangedRoutes } from '../verify/runner.js' import { SeededRng } from '../../../infrastructure/seeded-rng.js' import type { ScenarioConfig, TestConfig, RouteContract, ChaosConfig } from '../../../types.js' import { renderHumanArtifact } from '../../renderers/human.js' @@ -64,6 +65,7 @@ function coerceTimeout(value: unknown): number | undefined { export interface QualifyOptions { profile?: string seed?: number + changed?: boolean config?: string cwd?: string format?: 'human' | 'json' | 'ndjson' @@ -252,7 +254,8 @@ export function buildArtifact( const totalTests = runResult.scenarioResults.reduce((sum, s) => sum + s.steps.length, 0) + - (runResult.statefulResult?.tests.length ?? 0) + (runResult.statefulResult?.summary.passed ?? 0) + + (runResult.statefulResult?.summary.failed ?? 0) const passedTests = runResult.scenarioResults.reduce((sum, s) => sum + s.summary.passed, 0) + @@ -282,7 +285,11 @@ export function buildArtifact( .filter(Boolean) )] const statefulRoutes = runResult.statefulResult - ? [...new Set(runResult.statefulResult.tests.map(t => normalizeRouteIdentity(t.name)))] + ? [...new Set( + runResult.statefulResult.tests + .filter(t => !t.directive) + .map(t => normalizeRouteIdentity(t.name)) + )] : [] const chaosRoutesPlanned: string[] = [] const chaosRoutesExecuted: string[] = [] @@ -343,6 +350,7 @@ export function buildArtifact( total: totalTests, passed: passedTests, failed: failures.length, + skipped: (runResult.statefulResult?.summary.skipped ?? 0), }, executionSummary, coverageBreakdown, @@ -437,6 +445,7 @@ export async function qualifyCommand( const { profile, seed: explicitSeed, + changed, config: configPath, cwd, artifactDir, @@ -566,7 +575,31 @@ export async function qualifyCommand( try { // 9. Discover routes for chaos (with discovery metadata) - const appRoutes = await discoverAppRoutes(fastify) + const allRoutes = await discoverAppRoutes(fastify) + let appRoutes = allRoutes + + // 9a. Apply --changed filter + if (changed) { + const inGit = await isGitRepo(workingDir) + if (!inGit) { + return { + exitCode: USAGE_ERROR, + message: '--changed requires a git repository. Current directory is not inside a git repo.', + } + } + appRoutes = await filterChangedRoutes(allRoutes, workingDir) + const matched = appRoutes.length + const total = allRoutes.length + if (!ctx.options.quiet && format === 'human') { + console.log(`${matched}/${total} routes match --changed filter`) + } + if (matched === 0) { + return { + exitCode: SUCCESS, + message: `No routes changed since HEAD. All ${total} routes skipped.`, + } + } + } // Collect discovery warnings const { discoverRouteDetails } = await import('../../../domain/discovery.js') @@ -586,6 +619,8 @@ export async function qualifyCommand( fastify: fastify as any, seed, timeout: presetTimeout, + pluginContractRegistry: (fastify as unknown as Record)?._apophisPluginContractRegistry as + import('../../../domain/plugin-contracts.js').PluginContractRegistry | undefined, } const runResult = await runQualify(deps, gates, scenarios, statefulConfig, chaosConfig, appRoutes) @@ -600,6 +635,15 @@ export async function qualifyCommand( seed, }) + // Drain plugin contract registry warnings if app registered APOPHIS + const pcr = (fastify as any)?._apophisPluginContractRegistry + if (pcr?.drainWarnings) { + const pcrWarnings = pcr.drainWarnings() + if (pcrWarnings.length > 0) { + artifact.warnings.push(...pcrWarnings) + } + } + // 12. Signal quality guardrails — fail if zero checks executed const execSummary = runResult.executionSummary const warnings: string[] = [...artifact.warnings, ...discoveryWarnings] @@ -624,7 +668,9 @@ export async function qualifyCommand( if (gates.scenario && execSummary.scenariosRun === 0) { warnings.push('WARNING: scenario gate enabled but zero scenarios executed. Check route configuration.') } - if (gates.stateful && execSummary.statefulTestsRun === 0) { + if (gates.stateful && execSummary.statefulTestsRun === 0 && execSummary.statefulSkipped > 0) { + warnings.push(`WARNING: stateful gate enabled but all ${execSummary.statefulSkipped} commands were skipped (preconditions not met). Check required headers, scopes, or plugin contract settings.`) + } else if (gates.stateful && execSummary.statefulTestsRun === 0) { warnings.push('WARNING: stateful gate enabled but zero stateful tests executed. Check app routes and schema.') } if (gates.chaos && execSummary.chaosRunsRun === 0) { @@ -706,6 +752,7 @@ export async function handleQualify( const options: QualifyOptions = { profile: ctx.options.profile || undefined, seed: undefined, + changed: false, config: ctx.options.config || undefined, cwd: ctx.cwd, format: ctx.options.format as QualifyOptions['format'], @@ -722,6 +769,10 @@ export async function handleQualify( } } + if (args.includes('--changed')) { + options.changed = true + } + const result = await qualifyCommand(options, ctx) const format = options.format || ctx.options.format || 'human' const machineMode = format === 'json' || format === 'ndjson' || format === 'json-summary' || format === 'ndjson-summary' diff --git a/src/cli/commands/qualify/runner.ts b/src/cli/commands/qualify/runner.ts index b308d9f..5eba6a9 100644 --- a/src/cli/commands/qualify/runner.ts +++ b/src/cli/commands/qualify/runner.ts @@ -29,6 +29,7 @@ import type { FastifyInjectInstance, } from '../../../types.js' import type { ExtensionRegistry } from '../../../extension/types.js' +import type { PluginContractRegistry } from '../../../domain/plugin-contracts.js' // --------------------------------------------------------------------------- // Types @@ -60,6 +61,7 @@ export interface QualifyRunResult { totalFailed: number scenariosRun: number statefulTestsRun: number + statefulSkipped: number chaosRunsRun: number chaosRoutesPlanned: number chaosRoutesExecuted: number @@ -85,6 +87,7 @@ export interface CleanupFailure { export interface QualifyRunnerDeps { fastify: FastifyInjectInstance extensionRegistry?: ExtensionRegistry + pluginContractRegistry?: PluginContractRegistry seed: number timeout?: number } @@ -237,17 +240,20 @@ export async function runQualify( const statefulPassed = !statefulResult || statefulResult.summary.failed === 0 const chaosPassed = chaosResults.every(r => !r.applied || r.contractsPassed) - // Count execution metrics + // Count execution metrics — distinguish executed from skipped const scenariosRun = scenarioResults.length - const statefulTestsRun = statefulResult?.tests.length ?? 0 + const statefulPassedCount = statefulResult?.summary.passed ?? 0 + const statefulFailedCount = statefulResult?.summary.failed ?? 0 + const statefulSkippedCount = statefulResult?.summary.skipped ?? 0 + const statefulTestsRun = statefulPassedCount + statefulFailedCount const chaosRunsRun = chaosResults.length const totalSteps = allTraces.length const totalExecuted = scenariosRun + statefulTestsRun + chaosRunsRun const totalPassed = scenarioResults.reduce((sum, r) => sum + r.summary.passed, 0) + - (statefulResult?.summary.passed ?? 0) + + statefulPassedCount + chaosResults.reduce((sum, r) => sum + (r.applied && r.contractsPassed ? 1 : 0), 0) const totalFailed = scenarioResults.reduce((sum, r) => sum + r.summary.failed, 0) + - (statefulResult?.summary.failed ?? 0) + statefulFailedCount // Track executed and skipped routes for transparency const executedRoutes: string[] = [] @@ -316,6 +322,7 @@ export async function runQualify( totalFailed, scenariosRun, statefulTestsRun, + statefulSkipped: statefulSkippedCount, chaosRunsRun, chaosRoutesPlanned, chaosRoutesExecuted, diff --git a/src/cli/commands/qualify/scenario-handler.ts b/src/cli/commands/qualify/scenario-handler.ts index 461ed0d..d3617c4 100644 --- a/src/cli/commands/qualify/scenario-handler.ts +++ b/src/cli/commands/qualify/scenario-handler.ts @@ -27,7 +27,7 @@ export async function runScenarioWithTraces( ): Promise<{ result: ScenarioResult; traces: StepTrace[] }> { const scopeHeaders: Record = {} - const result = await runScenario(deps.fastify, config, scopeHeaders, deps.extensionRegistry) + const result = await runScenario(deps.fastify, config, scopeHeaders, deps.extensionRegistry, deps.pluginContractRegistry) const traces: StepTrace[] = result.steps.map((step, idx) => { const trace: StepTrace = { diff --git a/src/cli/commands/qualify/stateful-handler.ts b/src/cli/commands/qualify/stateful-handler.ts index 716ec3e..1cfe1f3 100644 --- a/src/cli/commands/qualify/stateful-handler.ts +++ b/src/cli/commands/qualify/stateful-handler.ts @@ -45,7 +45,7 @@ export async function runStatefulWithTraces( cleanupManager, minimalScopeRegistry, deps.extensionRegistry, - undefined, // pluginContractRegistry — will be passed from runner when available + deps.pluginContractRegistry, undefined, // outboundContractRegistry ) diff --git a/src/cli/commands/replay/index.ts b/src/cli/commands/replay/index.ts index 7c2d20f..57e8cc0 100644 --- a/src/cli/commands/replay/index.ts +++ b/src/cli/commands/replay/index.ts @@ -172,9 +172,36 @@ async function executeContractDirect( validateRuntime: false, } + // Compose plugin contracts if the app registered the APOPHIS plugin + const pluginContractRegistry = (fastify as any)?._apophisPluginContractRegistry + if (pluginContractRegistry) { + const composed = pluginContractRegistry.composeContracts(routeContract) + if (!routeContract.formulaSources) routeContract.formulaSources = {} + for (const phase of Object.values(composed.phases) as Array<{ requires: Array<{ formula: string; source: string }>; ensures: Array<{ formula: string; source: string }> }>) { + for (const req of phase.requires) { + if (!routeContract.requires.includes(req.formula)) { + routeContract.requires.push(req.formula) + routeContract.formulaSources[req.formula] = req.source as 'route' | `plugin:${string}` + } + } + for (const ens of phase.ensures) { + if (!routeContract.ensures.includes(ens.formula)) { + routeContract.ensures.push(ens.formula) + routeContract.formulaSources[ens.formula] = ens.source as 'route' | `plugin:${string}` + } + } + } + } + // Build request const headers: Record = {} + // Auto-inject headers required by plugin contract preconditions + for (const f of routeContract.requires) { + const m = f.match(/request_headers\(this\)\.([\w-]+)\s*(!=\s*null|==\s*"([^"]+)")/) + if (m) headers[m[1]!] = m[3] || 'test-value' + } + // Execute request try { const ctx = await executeHttp(fastify, routeContract, { @@ -264,10 +291,12 @@ async function executeReplay( } // Try to run verify first (works if app has APOPHIS plugin) + const pluginContractRegistry = (fastify as any)?._apophisPluginContractRegistry let runResult = await runVerify({ fastify: fastify as any, seed: artifact.seed || 42, routeFilters: [failure.route], + pluginContractRegistry, }) // If no routes matched, or route found but no contracts (plugin not registered before routes), diff --git a/src/cli/commands/verify/index.ts b/src/cli/commands/verify/index.ts index 2bfd6da..3e8d753 100644 --- a/src/cli/commands/verify/index.ts +++ b/src/cli/commands/verify/index.ts @@ -170,6 +170,7 @@ function buildArtifact( total: runResult.total, passed: runResult.passedCount, failed: runResult.failed, + skipped: runResult.skipped, }, deterministicParams: { seed: options.seed, @@ -390,6 +391,7 @@ export async function verifyCommand( runs: typeof config.presets?.[loadResult.presetName || '']?.runs === 'number' ? (config.presets[loadResult.presetName || ''] as { runs?: number }).runs : undefined, + pluginContractRegistry: (fastify as any)?._apophisPluginContractRegistry, }) // 7. Build artifact @@ -403,6 +405,15 @@ export async function verifyCommand( routeFilters, }) + // Drain plugin contract registry warnings if app registered APOPHIS + const pcr = (fastify as any)?._apophisPluginContractRegistry + if (pcr?.drainWarnings) { + const pcrWarnings = pcr.drainWarnings() + if (pcrWarnings.length > 0) { + artifact.warnings.push(...pcrWarnings) + } + } + // 8. Write artifact if configured or on failure const shouldEmitArtifact = Boolean(artifactDir || config.artifactDir || !runResult.passed) await emitArtifact(artifact, { @@ -638,54 +649,27 @@ export async function handleVerify( const format = options.format || ctx.options.format || 'human' const machineMode = format === 'json' || format === 'ndjson' || format === 'json-summary' || format === 'ndjson-summary' - if (!ctx.options.quiet) { + // Machine-readable formats always emit regardless of --quiet + if (machineMode) { if (format === 'json') { - if (result.artifact) { - console.log(renderJsonArtifact(result.artifact)) - } else { - console.log(renderJson({ - exitCode: result.exitCode, - message: result.message, - warnings: result.warnings, - })) - } + console.log(result.artifact ? renderJsonArtifact(result.artifact) : renderJson({ exitCode: result.exitCode, message: result.message, warnings: result.warnings })) } else if (format === 'json-summary') { - if (result.artifact) { - console.log(renderJsonSummaryArtifact(result.artifact)) - } else { - console.log(renderJson({ - exitCode: result.exitCode, - message: result.message, - warnings: result.warnings, - })) - } + console.log(result.artifact ? renderJsonSummaryArtifact(result.artifact) : renderJson({ exitCode: result.exitCode, message: result.message, warnings: result.warnings })) } else if (format === 'ndjson') { if (result.artifact) { renderNdjsonArtifact(result.artifact) } else { - process.stdout.write(JSON.stringify({ - type: 'run.completed', - command: 'verify', - exitCode: result.exitCode, - message: result.message, - warnings: result.warnings, - }) + '\n') + process.stdout.write(JSON.stringify({ type: 'run.completed', command: 'verify', exitCode: result.exitCode, message: result.message, warnings: result.warnings }) + '\n') } } else if (format === 'ndjson-summary') { if (result.artifact) { renderNdjsonSummaryArtifact(result.artifact) } else { - process.stdout.write(JSON.stringify({ - type: 'run.completed', - command: 'verify', - exitCode: result.exitCode, - message: result.message, - warnings: result.warnings, - }) + '\n') + process.stdout.write(JSON.stringify({ type: 'run.completed', command: 'verify', exitCode: result.exitCode, message: result.message, warnings: result.warnings }) + '\n') } - } else if (result.message) { - console.log(result.message) } + } else if (!ctx.options.quiet && result.message) { + console.log(result.message) } // Print warnings in human mode only diff --git a/src/cli/commands/verify/runner.ts b/src/cli/commands/verify/runner.ts index 4b60b4c..106e33a 100644 --- a/src/cli/commands/verify/runner.ts +++ b/src/cli/commands/verify/runner.ts @@ -26,6 +26,7 @@ import * as fc from 'fast-check' import { resolveRuns } from '../../../types.js' import type { EvalContext, RouteContract, FastifyInjectInstance } from '../../../types.js' import type { RouteResult } from '../../core/types.js' +import type { PluginContractRegistry } from '../../../domain/plugin-contracts.js' // --------------------------------------------------------------------------- // Types @@ -46,6 +47,7 @@ export interface VerifyRunResult { total: number passedCount: number failed: number + skipped: number failures: VerifyFailure[] durationMs: number noRoutesMatched: boolean @@ -66,6 +68,7 @@ export interface VerifyRunnerDeps { changed?: boolean profileRoutes?: string[] runs?: number + pluginContractRegistry?: PluginContractRegistry } // --------------------------------------------------------------------------- @@ -161,7 +164,7 @@ function filterRoutesByPatterns(routes: RouteContract[], patterns: string[]): Ro /** * Check if cwd is inside a git repository. */ -async function isGitRepo(cwd: string): Promise { +export async function isGitRepo(cwd: string): Promise { try { const { execSync } = await import('node:child_process') execSync('git rev-parse --git-dir', { cwd, encoding: 'utf-8', stdio: 'pipe' }) @@ -174,7 +177,7 @@ async function isGitRepo(cwd: string): Promise { /** * Get git-modified files for --changed filtering. */ -async function getGitChangedFiles(cwd: string): Promise { +export async function getGitChangedFiles(cwd: string): Promise { try { const { execSync } = await import('node:child_process') const output = execSync('git diff --name-only HEAD', { cwd, encoding: 'utf-8' }) @@ -187,7 +190,7 @@ async function getGitChangedFiles(cwd: string): Promise { /** * Filter routes to only those modified in git. */ -async function filterChangedRoutes( +export async function filterChangedRoutes( routes: RouteContract[], cwd: string, ): Promise { @@ -481,6 +484,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise route.ensures.length > 0 || route.requires.length > 0 @@ -522,6 +549,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise { + const headers: Record = {} + for (const f of formulas) { + const nullCheck = f.match(/request_headers\(this\)\.([\w-]+)\s*!=\s*null/i) + if (nullCheck) { headers[nullCheck[1]!] = 'test-value'; continue } + const eqCheck = f.match(/request_headers\(this\)\.([\w-]+)\s*==\s*"([^"]+)"/i) + if (eqCheck) { headers[eqCheck[1]!] = eqCheck[2]!; continue } + } + return headers + } for (const route of routesWithContracts) { - const contracts = [...route.requires, ...route.ensures] + const defaultHeaders = extractRequiredHeaders(route.requires) const variants = route.variants && route.variants.length > 0 ? route.variants : [{ name: 'default' }] for (const variant of variants) { - for (const contract of contracts) { - for (let runIndex = 0; runIndex < runs; runIndex++) { - total++ - const result = await executeContract( - fastify, route, contract, deps.timeout, - variant, deps.seed, runIndex, - ) + for (let runIndex = 0; runIndex < runs; runIndex++) { + // Execute one HTTP request per run + const request = buildRouteRequest(route, deps.seed, runIndex, variant) + const headers = { + ...defaultHeaders, + ...request.headers, + ...(variant.headers ?? {}), + } + const ctx = await executeHttp(fastify, route, { + method: request.method, + url: request.url, + body: request.body, + headers, + query: {}, + }, undefined, deps.timeout) + const evalCtx: EvalContext = { + ...ctx, + operationResolver: createOperationResolver(fastify, headers, ctx), + } - if (result.failure) { - failures.push(result.failure) - } else { - passedCount++ + // Evaluate requires first. If any fail, skip ensures for this run. + let requiresPassed = true + for (const formula of route.requires) { + total++ + try { + const parsed = parse(formula) + const result = await evaluateAsync(parsed.ast, evalCtx) + if (!result.success || !result.value) { + requiresPassed = false + skippedCount++ + } else { + passedCount++ + } + } catch { + requiresPassed = false + skippedCount++ } } + + // Only evaluate ensures if all requires passed + if (requiresPassed) { + for (const formula of route.ensures) { + total++ + try { + const parsed = parse(formula) + const result = await evaluateAsync(parsed.ast, evalCtx) + if (!result.success || !result.value) { + const diagnostic = buildFailureDiagnostic(formula, evalCtx, result.success ? undefined : result.error) + failures.push({ + route: variant.name && variant.name !== 'default' + ? `[variant:${variant.name}] ${route.method} ${route.path}` + : `${route.method} ${route.path}`, + contract: formula, + expected: diagnostic.expected, + observed: diagnostic.observed, + formula: diagnostic.formula, + category: diagnostic.category, + }) + } else { + passedCount++ + } + } catch (error) { + const diagnostic = buildFailureDiagnostic(formula, evalCtx, error instanceof Error ? error.message : String(error)) + failures.push({ + route: variant.name && variant.name !== 'default' + ? `[variant:${variant.name}] ${route.method} ${route.path}` + : `${route.method} ${route.path}`, + contract: formula, + expected: diagnostic.expected, + observed: diagnostic.observed, + formula: diagnostic.formula, + category: diagnostic.category, + }) + } + } + } else { + // Preconditions failed: ensures are skipped but not evaluated. + // We still count each ensure as "total" and "skipped" for transparency. + total += route.ensures.length + skippedCount += route.ensures.length + } } } } @@ -578,6 +691,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise = { optional: true, items: { type: 'object' }, }, - chaos: { + pluginContracts: { type: 'object', optional: true, properties: {}, diff --git a/src/cli/core/context.ts b/src/cli/core/context.ts index 85b63bd..1a7bfc5 100644 --- a/src/cli/core/context.ts +++ b/src/cli/core/context.ts @@ -93,7 +93,7 @@ export function createContext(options: Record = {}): CliContext const packageManager = detectPackageManager(cwd); // Normalize options - const format = options.format === 'json' || options.format === 'ndjson' + const format = options.format === 'json' || options.format === 'ndjson' || options.format === 'json-summary' || options.format === 'ndjson-summary' ? options.format : 'human'; diff --git a/src/cli/core/index.ts b/src/cli/core/index.ts index 274e832..ed737b6 100644 --- a/src/cli/core/index.ts +++ b/src/cli/core/index.ts @@ -102,9 +102,11 @@ function getCommandHelp(command: string): string { ${pc.dim('Options:')} --profile Profile name from config --seed Deterministic seed + --changed Filter to git-modified routes ${pc.dim('Examples:')} apophis qualify --profile oauth-nightly --seed 42 + apophis qualify --changed `, replay: ` ${pc.bold('apophis replay')} — Replay a failure using seed and stored trace @@ -167,18 +169,18 @@ function printInternalError(error: unknown): void { console.error(); } -function resolveRequestedFormat(argv: string[]): 'human' | 'json' | 'ndjson' { +function resolveRequestedFormat(argv: string[]): 'human' | 'json' | 'ndjson' | 'json-summary' | 'ndjson-summary' { for (let i = 0; i < argv.length; i++) { const arg = argv[i]; if (!arg) continue; if (arg === '--format' && argv[i + 1]) { const value = argv[i + 1]; - if (value === 'json' || value === 'ndjson') return value; + if (value === 'json' || value === 'ndjson' || value === 'json-summary' || value === 'ndjson-summary') return value as 'json' | 'ndjson' | 'json-summary' | 'ndjson-summary'; return 'human'; } if (arg.startsWith('--format=')) { const value = arg.slice('--format='.length); - if (value === 'json' || value === 'ndjson') return value; + if (value === 'json' || value === 'ndjson' || value === 'json-summary' || value === 'ndjson-summary') return value as 'json' | 'ndjson' | 'json-summary' | 'ndjson-summary'; return 'human'; } } @@ -277,6 +279,7 @@ export async function main(argv: string[] = process.argv.slice(2)): Promise', 'Profile name from config'); cmd.option('--seed ', 'Deterministic seed'); + cmd.option('--changed', 'Filter to git-modified routes'); break; case 'replay': cmd.option('--artifact ', 'Path to failure artifact'); @@ -375,7 +378,7 @@ export async function main(argv: string[] = process.argv.slice(2)): Promise 0) { + lines.push(` ${c.yellow('Skipped:')} ${summary.skipped} (preconditions not met)`); + } + lines.push(` Duration: ${formatDuration(artifact.durationMs)}`); if (artifact.seed !== undefined) { @@ -404,15 +408,31 @@ export function renderHumanArtifact( lines.push(c.bold('Next steps')); if (artifact.command === 'verify') { if (artifact.summary.failed === 0) { - lines.push(` ${c.green('✓')} All contracts passed.`); - lines.push(` ${c.dim('→ Add more behavioral contracts with')} x-ensures ${c.dim('and')} x-requires ${c.dim('to cover more routes.')}`); - lines.push(` ${c.dim('→ Run')} apophis observe ${c.dim('to enable runtime contract monitoring in production.')}`); - lines.push(` ${c.dim('→ Run')} apophis qualify --profile standard ${c.dim('for stateful/chaos testing.')}`); + if (artifact.summary.skipped && artifact.summary.skipped > 0) { + lines.push(` ${c.yellow('!')} No contract violations, but ${artifact.summary.skipped} precondition(s) were unmet and skipped.`); + lines.push(` ${c.dim('→ Preconditions (x-requires + plugin contract requires) must be satisfied for tests to run.')}`); + lines.push(` ${c.dim('→ Send required headers via --variant or scope configuration.')}`); + lines.push(` ${c.dim('→ Add more behavioral contracts with')} x-ensures ${c.dim('and')} x-requires ${c.dim('to cover more routes.')}`); + } else { + lines.push(` ${c.green('✓')} All contracts passed.`); + lines.push(` ${c.dim('→ Add more behavioral contracts with')} x-ensures ${c.dim('and')} x-requires ${c.dim('to cover more routes.')}`); + lines.push(` ${c.dim('→ Run')} apophis observe ${c.dim('to enable runtime contract monitoring in production.')}`); + lines.push(` ${c.dim('→ Run')} apophis qualify --profile standard ${c.dim('for stateful/chaos testing.')}`); + } } else { lines.push(` ${c.yellow('!')} Fix failing contracts and rerun with:`); lines.push(` ${c.dim('→')} apophis verify --seed ${artifact.seed} ${artifact.profile ? `--profile ${artifact.profile}` : ''}`); lines.push(` ${c.dim('→ Or replay the artifact:')} apophis replay --artifact `); } + } else if (artifact.command === 'qualify') { + if (artifact.summary.skipped && artifact.summary.skipped > 0) { + lines.push(` ${c.yellow('!')} ${artifact.summary.skipped} stateful command(s) skipped — preconditions not met.`); + lines.push(` ${c.dim('→ Check plugin contracts and route headers. Qualify needs preconditions satisfied.')}`); + } + if (artifact.summary.failed === 0 && (!artifact.summary.skipped || artifact.summary.skipped === 0)) { + lines.push(` ${c.green('✓')} All checks passed.`); + } + lines.push(` ${c.dim('→ Review')} reports/apophis/ ${c.dim('for detailed artifact output.')}`); } return lines.join('\n'); diff --git a/src/domain/contract-validation.ts b/src/domain/contract-validation.ts index 9d89d55..92a42d8 100644 --- a/src/domain/contract-validation.ts +++ b/src/domain/contract-validation.ts @@ -179,11 +179,13 @@ const makeConditionFailure = ( const expected = fieldExpectation?.expected ?? statusExpectation.expected const actual = fieldExpectation?.actual ?? statusExpectation.actual const diff = buildDiff(formula, ctx.response.body) + const source = (route as RouteContract | undefined)?.formulaSources?.[formula] return makeViolation({ route: route ?? { method: '', path: '' }, formula, kind, + ...(source ? { source, phase: 'postcondition' } : {}), request: { body: ctx.request.body, headers: ctx.request.headers, @@ -206,10 +208,12 @@ const makeFormulaError = ( route: RouteContract | { method: string; path: string } | undefined, message: string ): ContractViolation => { + const source = (route as RouteContract | undefined)?.formulaSources?.[formula] return makeViolation({ route: route ?? { method: '', path: '' }, formula, kind, + ...(source ? { source, phase: 'postcondition' } : {}), request: { body: ctx.request.body, headers: ctx.request.headers, diff --git a/src/domain/plugin-contracts.ts b/src/domain/plugin-contracts.ts index 22c579e..239a2ca 100644 --- a/src/domain/plugin-contracts.ts +++ b/src/domain/plugin-contracts.ts @@ -39,6 +39,7 @@ function matchPattern(pattern: string, path: string): boolean { export class PluginContractRegistry { private contracts = new Map() private availableExtensions = new Set() + private _warnings: string[] = [] /** * Register a plugin's contract specification. * Idempotent: registering the same plugin twice updates the spec. @@ -75,7 +76,7 @@ export class PluginContractRegistry { if (!matchPattern(spec.appliesTo, route.path)) continue const extCheck = this.checkExtensions(spec) if (!extCheck.available) { - console.warn( + this._warnings.push( `Plugin '${plugin}' requires extensions [${extCheck.missing.join(', ')}] which are not registered. Skipping its contracts.` ) continue @@ -141,6 +142,16 @@ export class PluginContractRegistry { getAvailableExtensions(): string[] { return Array.from(this.availableExtensions) } + /** + * Collect and clear warnings (e.g., missing extension dependencies). + * Callers should surface these via their own output channels instead of + * letting console.warn bleed into CLI formatted output. + */ + drainWarnings(): string[] { + const w = this._warnings + this._warnings = [] + return w + } } // ============================================================================ // Factory diff --git a/src/fastify-factory.ts b/src/fastify-factory.ts index c582e7a..8cb6567 100644 --- a/src/fastify-factory.ts +++ b/src/fastify-factory.ts @@ -1,11 +1,17 @@ import type { FastifyInstance, FastifyServerOptions } from 'fastify' import Fastify from 'fastify' import { installRouteDiscovery } from './infrastructure/discovery-hook.js' +import type { ObserveSink } from './types.js' export interface CreateFastifyOptions extends FastifyServerOptions { apophis?: { runtime?: 'off' | 'warn' | 'error' discoveryOnly?: boolean + observe?: { + enabled?: boolean + sampling?: number + sinks?: ObserveSink[] + } } } @@ -16,9 +22,16 @@ export async function createFastify( const fastify = Fastify(fastifyOpts) installRouteDiscovery(fastify as unknown as { addHook: Function }) - if (apophisOpts?.discoveryOnly !== true && apophisOpts?.runtime && apophisOpts.runtime !== 'off') { + const shouldRegisterApophis = apophisOpts?.discoveryOnly !== true && ( + (apophisOpts?.runtime !== undefined && apophisOpts.runtime !== 'off') || + apophisOpts?.observe?.enabled === true + ) + if (shouldRegisterApophis) { const { apophisPlugin } = await import('./plugin/index.js') - await fastify.register(apophisPlugin, { runtime: apophisOpts.runtime }) + await fastify.register(apophisPlugin, { + runtime: apophisOpts.runtime, + observe: apophisOpts.observe, + }) } return fastify diff --git a/src/formula/parser.ts b/src/formula/parser.ts index 8f3217f..e10150a 100644 --- a/src/formula/parser.ts +++ b/src/formula/parser.ts @@ -727,6 +727,8 @@ function parseVariable(input: string, pos: number): { node: FormulaNode; pos: nu } } const name = input.substring(p, i) + // F/T shorthands: treat as boolean literals, not variables. + // Only apply when there's no accessor (e.g., F.x is a variable, not literal). let accessor: string[] | undefined const dotPos = skipWs(input, i) if (input[dotPos] === '.') { @@ -734,6 +736,12 @@ function parseVariable(input: string, pos: number): { node: FormulaNode; pos: nu accessor = [acc.id] i = acc.pos } + if ((name === 'F' || name === 'T') && accessor === undefined) { + return { + node: { type: 'literal', value: name === 'T' }, + pos: i + } + } return { node: { type: 'variable', name, accessor }, @@ -842,12 +850,31 @@ function isWordChar(c: number): boolean { function parseClause(input: string, pos: number, extensionHeaders: string[] = []): { node: FormulaNode; pos: number } { const p = skipWs(input, pos) const len = input.length - if (p < len && input[p] === 'T' && (p + 1 >= len || !isWordChar(input.charCodeAt(p + 1)))) { - return { node: { type: 'literal', value: true }, pos: p + 1 } + + // T/F shorthand: only consume as standalone literal if NOT followed by a + // comparison operator (==, !=, <, >, <=, >=, matches). + function isTfLiteral(): boolean { + if (p >= len) return false + const ch = input[p]! + if (ch !== 'T' && ch !== 'F') return false + const afterP = p + 1 + if (afterP >= len || !isWordChar(input.charCodeAt(afterP))) { + const next = skipWs(input, afterP) + const c0 = next < len ? input.charCodeAt(next) : -1 + const c1 = next + 1 < len ? input.charCodeAt(next + 1) : -1 + // If followed by a comparator, T/F is a term, not a standalone literal + if (c0 === 61 || c0 === 33 || c0 === 60 || c0 === 62) return false // ==, !=, <=, <, >=, > + if (c0 === 109) return false // matches + return true + } + return false } - if (p < len && input[p] === 'F' && (p + 1 >= len || !isWordChar(input.charCodeAt(p + 1)))) { - return { node: { type: 'literal', value: false }, pos: p + 1 } + + if (isTfLiteral()) { + const ch = input[p]! + return { node: { type: 'literal', value: ch === 'T' }, pos: p + 1 } } + const cmp = parseComparison(input, pos, extensionHeaders) if (cmp) return cmp // If not a comparison, it must be a term (operation or literal) diff --git a/src/plugin/builders.ts b/src/plugin/builders.ts index 69f73da..f30374f 100644 --- a/src/plugin/builders.ts +++ b/src/plugin/builders.ts @@ -67,12 +67,13 @@ export const buildCleanup = (cleanupManager: CleanupManager) => export const buildScenario = ( fastify: FastifyInstance, scope: ScopeRegistry, - extensionRegistry: ExtensionRegistry + extensionRegistry: ExtensionRegistry, + pluginContractRegistry?: import('../domain/plugin-contracts.js').PluginContractRegistry ) => async (opts: ScenarioConfig): Promise => { assertNonProduction('scenario') const scopeHeaders = scope.getHeaders(opts.scope ?? null) const injectInstance = fastify as unknown as import('../types.js').FastifyInjectInstance - return runScenario(injectInstance, opts, scopeHeaders, extensionRegistry) + return runScenario(injectInstance, opts, scopeHeaders, extensionRegistry, pluginContractRegistry) } // --------------------------------------------------------------------------- diff --git a/src/plugin/index.ts b/src/plugin/index.ts index 9f58ba2..77671bf 100644 --- a/src/plugin/index.ts +++ b/src/plugin/index.ts @@ -96,7 +96,7 @@ export const apophisPlugin = async (fastify: FastifyInstance, opts: ApophisOptio contract: buildContract(fastify, scope, extensionRegistry, pluginContractRegistry, outboundContractRegistry), stateful: buildStateful(fastify, scope, cleanupManager, extensionRegistry, pluginContractRegistry, outboundContractRegistry), check: buildCheck(fastify, scope, extensionRegistry, pluginContractRegistry), - scenario: buildScenario(fastify, scope, extensionRegistry), + scenario: buildScenario(fastify, scope, extensionRegistry, pluginContractRegistry), cleanup: buildCleanup(cleanupManager), spec: buildSpec(fastify), test: { @@ -147,17 +147,25 @@ export const apophisPlugin = async (fastify: FastifyInstance, opts: ApophisOptio }, } fastify.decorate('apophis', decorations) - // Runtime validation: never register hooks in production + // Expose plugin contract registry for CLI qualify access + ;(fastify as unknown as Record)._apophisPluginContractRegistry = pluginContractRegistry + // Runtime validation may block requests, so it is disabled in production. + // Observe sinks are explicitly non-blocking and are allowed in production. const isProd = process.env.NODE_ENV === 'production' || process.env.NODE_ENV === 'prod' - if (opts.runtime && opts.runtime !== 'off' && !isProd) { + const observeSinks = opts.observe?.enabled && opts.observe.sinks && opts.observe.sinks.length > 0 + ? opts.observe.sinks + : undefined + const enableBlockingRuntime = opts.runtime && opts.runtime !== 'off' && !isProd + if (enableBlockingRuntime || observeSinks) { + const observeSampling = opts.observe?.sampling ?? 1 const hookOpts: { validateRuntime: boolean; runtimeLevel?: 'warn' | 'error'; observers?: { sinks: import('../types.js').ObserveSink[]; sampling: number } } = { validateRuntime: true, - runtimeLevel: opts.runtime, + runtimeLevel: enableBlockingRuntime ? opts.runtime : 'warn', } - if (opts.observe?.enabled && opts.observe.sinks && opts.observe.sinks.length > 0) { + if (observeSinks) { hookOpts.observers = { - sinks: opts.observe.sinks, - sampling: opts.observe.sampling ?? 1, + sinks: observeSinks, + sampling: observeSampling, } } registerValidationHooks(fastify, hookOpts as Parameters[1]) diff --git a/src/quality/petit-runner.ts b/src/quality/petit-runner.ts index bc24ddc..dbcc602 100644 --- a/src/quality/petit-runner.ts +++ b/src/quality/petit-runner.ts @@ -118,15 +118,18 @@ export const runPetitTests = async ( if (pluginContractRegistry) { for (const route of routes) { const composed = pluginContractRegistry.composeContracts(route) + if (!route.formulaSources) route.formulaSources = {} for (const phase of Object.values(composed.phases)) { for (const req of phase.requires) { if (!route.requires.includes(req.formula)) { route.requires.push(req.formula) + route.formulaSources[req.formula] = req.source } } for (const ens of phase.ensures) { if (!route.ensures.includes(ens.formula)) { route.ensures.push(ens.formula) + route.formulaSources[ens.formula] = ens.source } } } @@ -276,7 +279,16 @@ export const runPetitTests = async ( flushCache() const deduped = deduplicateTestFailures(results).results - const suite: TestSuite = buildPetitSuite(allRoutes, routes, skippedRoutes, deduped, cacheHits, cacheMisses, startTime) + const { formatSuite } = await import('./formatters.js') + const suite: TestSuite = formatSuite(deduped, { + startTime, + cacheHits, + cacheMisses, + allRoutes, + testedRoutes: routes, + skippedRoutes, + pluginContractRegistry, + }) if (suiteMockRuntime) suiteMockRuntime.restore() if (extensionRegistry) await extensionRegistry.runSuiteEndHooks(suite) diff --git a/src/quality/scenario-runner.ts b/src/quality/scenario-runner.ts index c6802ac..a4a4f29 100644 --- a/src/quality/scenario-runner.ts +++ b/src/quality/scenario-runner.ts @@ -5,6 +5,7 @@ import { executeHttp } from '../infrastructure/http-executor.js' import { validatePostconditionsAsync } from '../domain/contract-validation.js' import { CONTENT_TYPE } from '../infrastructure/http-executor.js' import type { ExtensionRegistry } from '../extension/types.js' +import type { PluginContractRegistry } from '../domain/plugin-contracts.js' import type { EvalContext, FastifyInjectInstance, @@ -123,7 +124,8 @@ export const runScenario = async ( fastify: FastifyInjectInstance, config: ScenarioConfig, scopeHeaders: Record, - extensionRegistry?: ExtensionRegistry + extensionRegistry?: ExtensionRegistry, + pluginContractRegistry?: PluginContractRegistry ): Promise => { const started = Date.now() const steps: ScenarioStepResult[] = [] @@ -162,6 +164,37 @@ export const runScenario = async ( } const route = buildScenarioRoute({ ...interpolated, method }) + + // Merge plugin contracts into the synthetic route + if (pluginContractRegistry) { + const composed = pluginContractRegistry.composeContracts(route) + if (!route.formulaSources) route.formulaSources = {} + for (const phase of Object.values(composed.phases)) { + for (const req of phase.requires) { + if (!route.requires.includes(req.formula)) { + route.requires.push(req.formula) + route.formulaSources[req.formula] = req.source + } + } + for (const ens of phase.ensures) { + if (!route.ensures.includes(ens.formula)) { + route.ensures.push(ens.formula) + route.formulaSources[ens.formula] = ens.source + } + } + } + } + + // Auto-inject headers required by plugin contract preconditions + for (const f of route.requires) { + const m = f.match(/request_headers\(this\)\.([\w-]+)\s*(!=\s*null|==\s*"([^"]+)")/) + if (m) { + if (headers[m[1]!] === undefined) { + headers[m[1]!] = m[3] || 'test-value' + } + } + } + const preContext: EvalContext = { request: { body, @@ -204,7 +237,8 @@ export const runScenario = async ( operationResolver: createOperationResolver(fastify, headers, preContext), } - const validation = await validatePostconditionsAsync(step.expect.slice(), evalCtx, route, extensionRegistry) + const mergedEnsures = [...step.expect, ...route.ensures] + const validation = await validatePostconditionsAsync(mergedEnsures, evalCtx, route, extensionRegistry) if (!validation.success) { steps.push({ name: step.name, diff --git a/src/quality/stateful-request-execution.ts b/src/quality/stateful-request-execution.ts index 2b8b553..d220eb0 100644 --- a/src/quality/stateful-request-execution.ts +++ b/src/quality/stateful-request-execution.ts @@ -49,6 +49,17 @@ const runBuildAndPrecondition = async ( const { fastify, scopeHeaders, rng, extensionRegistry } = runtime let request = buildRequest(command.route, command.params, scopeHeaders, modelState, rng) + // Auto-inject headers required by plugin contract preconditions + if (command.route.requires.length > 0) { + const merged = { ...request.headers } + for (const f of command.route.requires) { + const m = f.match(/request_headers\(this\)\.([\w-]+)\s*(!=\s*null|==\s*"([^"]+)")/) + if (m) { + merged[m[1]!] = m[3] || 'test-value' + } + } + request.headers = merged + } if (extensionRegistry) { request = await extensionRegistry.runBuildRequestHooks({ route: command.route, diff --git a/src/quality/stateful-runner.ts b/src/quality/stateful-runner.ts index 3155bc9..1ba7306 100644 --- a/src/quality/stateful-runner.ts +++ b/src/quality/stateful-runner.ts @@ -22,7 +22,7 @@ import type { OutboundContractRegistry } from '../domain/outbound-contracts.js' import * as fc from 'fast-check' import type { ModelState } from '../domain/stateful.js' import type { CleanupManager } from '../infrastructure/cleanup-manager.js' -import type { EvalContext, FastifyInjectInstance, RouteContract, ScopeRegistry, TestConfig, TestResult, TestSuite } from '../types.js' +import type { EvalContext, FastifyInjectInstance, RouteContract, ScopeRegistry, TestConfig, TestResult, TestSuite, ContractViolation } from '../types.js' // Pure: hash helpers for deterministic sub-seeds // --------------------------------------------------------------------------- @@ -105,15 +105,18 @@ export const runStatefulTests = async ( if (pluginContractRegistry) { for (const route of routes) { const composed = pluginContractRegistry.composeContracts(route) + if (!route.formulaSources) route.formulaSources = {} for (const phase of Object.values(composed.phases)) { for (const req of phase.requires) { if (!route.requires.includes(req.formula)) { route.requires.push(req.formula) + route.formulaSources[req.formula] = req.source } } for (const ens of phase.ensures) { if (!route.ensures.includes(ens.formula)) { route.ensures.push(ens.formula) + route.formulaSources[ens.formula] = ens.source } } } @@ -303,9 +306,36 @@ export const runStatefulTests = async ( const passed = dedupedResults.filter((r) => r.ok && r.directive === undefined).length const failed = dedupedResults.filter((r) => !r.ok).length const skipped = dedupedResults.filter((r) => r.directive !== undefined).length + // Compute plugin contract counts + let pluginContractsApplied = 0 + let pluginContractsFailed = 0 + if (pluginContractRegistry) { + for (const route of routes) { + const composed = pluginContractRegistry.composeContracts(route) + for (const phase of Object.values(composed.phases)) { + pluginContractsApplied += phase.ensures.length + phase.requires.length + } + } + for (const result of dedupedResults) { + if (!result.ok && result.diagnostics?.violation) { + const violation = result.diagnostics.violation as ContractViolation + if (violation.source?.startsWith('plugin:')) pluginContractsFailed++ + } + } + } const suite: TestSuite = { tests: dedupedResults, - summary: { passed, failed, skipped, timeMs: Date.now() - startTime, cacheHits, cacheMisses, counterexample: counterexampleOutput }, + summary: { + passed, + failed, + skipped, + timeMs: Date.now() - startTime, + cacheHits, + cacheMisses, + counterexample: counterexampleOutput, + pluginContractsApplied, + pluginContractsFailed, + }, routes: allRoutes.map(r => ({ path: r.path, method: r.method, diff --git a/src/test/cli/machine-output-contracts.test.ts b/src/test/cli/machine-output-contracts.test.ts index 4206d41..75d077b 100644 --- a/src/test/cli/machine-output-contracts.test.ts +++ b/src/test/cli/machine-output-contracts.test.ts @@ -1,6 +1,8 @@ import { test } from 'node:test'; import assert from 'node:assert'; import { spawnSync } from 'node:child_process'; +import fs from 'node:fs'; +import path from 'node:path'; function runCli(args: string[]): { status: number; stdout: string; stderr: string } { const result = spawnSync( @@ -84,3 +86,158 @@ test('global help in machine mode emits parseable JSON', () => { assert.ok(typeof parsed.help === 'string'); assert.ok(parsed.help.includes('Usage:')); }); + +// --------------------------------------------------------------------------- +// json-summary and ndjson-summary format tests +// --------------------------------------------------------------------------- + +test('verify --format json-summary emits parseable JSON with summary fields', () => { + const { status, stdout, stderr } = runCli([ + 'verify', '--cwd', 'src/cli/__fixtures__/tiny-fastify', + '--profile', 'quick', '--seed', '42', '--format', 'json-summary', + ]); + + assert.strictEqual(status, 0); + assert.strictEqual(stderr.trim(), ''); + assert.ok(!stdout.includes('Seed:'), 'machine output must not include human seed prelude'); + + const parsed = JSON.parse(stdout); + assert.strictEqual(parsed.command, 'verify'); + assert.strictEqual(typeof parsed.summary.total, 'number'); + assert.strictEqual(typeof parsed.summary.passed, 'number'); + assert.strictEqual(typeof parsed.summary.failed, 'number'); + assert.strictEqual(typeof parsed.seed, 'number'); + assert.strictEqual(typeof parsed.exitReason, 'string'); +}); + +test('verify --format ndjson-summary emits only NDJSON records', () => { + const { status, stdout, stderr } = runCli([ + 'verify', '--cwd', 'src/cli/__fixtures__/tiny-fastify', + '--profile', 'quick', '--seed', '42', '--format', 'ndjson-summary', + ]); + + assert.strictEqual(status, 0); + assert.strictEqual(stderr.trim(), ''); + assert.ok(!stdout.includes('Seed:'), 'machine output must not include human seed prelude'); + + const lines = stdout.split('\n').map((line) => line.trim()).filter(Boolean); + assert.ok(lines.length >= 1, 'ndjson-summary should emit at least one record'); + for (const line of lines) { + const parsed = JSON.parse(line); + assert.ok(typeof parsed.type === 'string' || typeof parsed.summary === 'object' || typeof parsed.command === 'string'); + } +}); + +test('verify --format json-summary --quiet still emits JSON', () => { + const { status, stdout, stderr } = runCli([ + 'verify', '--cwd', 'src/cli/__fixtures__/tiny-fastify', + '--profile', 'quick', '--seed', '42', '--format', 'json-summary', '--quiet', + ]); + + assert.strictEqual(status, 0); + assert.strictEqual(stderr.trim(), ''); + assert.ok(stdout.trim().length > 0, '--quiet must not suppress machine-readable JSON'); + const parsed = JSON.parse(stdout); + assert.strictEqual(parsed.command, 'verify'); +}); + +// --------------------------------------------------------------------------- +// skipped field in artifact +// --------------------------------------------------------------------------- + +test('verify --format json-summary includes skipped field when preconditions cannot be met', () => { + const { status, stdout, stderr } = runCli([ + 'verify', '--cwd', 'src/cli/__fixtures__/plugin-contracts', + '--profile', 'quick', '--seed', '42', '--format', 'json-summary', + ]); + + // Plugin contract requires x-custom header which the generated request won't send. + // Auto-injection should add it, so preconditions pass. + // If not, skipped > 0 and failed === 0. + const parsed = JSON.parse(stdout); + assert.strictEqual(parsed.command, 'verify'); + assert.ok(typeof parsed.summary.skipped === 'number', 'skipped field must be present'); + assert.strictEqual(parsed.summary.failed, 0, 'no postcondition failures expected'); +}); + +test('verify --format json-summary exit code 0 when all contracts pass', () => { + const { status, stdout } = runCli([ + 'verify', '--cwd', 'src/cli/__fixtures__/tiny-fastify', + '--seed', '42', '--format', 'json-summary', '--quiet', + ]); + + assert.strictEqual(status, 0, 'exit code must be 0 when all contracts pass'); + const parsed = JSON.parse(stdout); + assert.strictEqual(parsed.summary.failed, 0); +}); + +// --------------------------------------------------------------------------- +// qualify json-summary and round-trip +// --------------------------------------------------------------------------- + +test('qualify --format json-summary emits parseable JSON', () => { + const { status, stdout, stderr } = runCli([ + 'qualify', '--cwd', 'src/cli/__fixtures__/plugin-contracts', + '--seed', '42', '--format', 'json-summary', + ]); + + const parsed = JSON.parse(stdout); + assert.strictEqual(parsed.command, 'qualify'); + assert.strictEqual(typeof parsed.summary.total, 'number'); + assert.strictEqual(typeof parsed.summary.passed, 'number'); + assert.strictEqual(typeof parsed.summary.failed, 'number'); + assert.ok(typeof parsed.executionSummary === 'object', 'qualify artifact must include execution summary'); +}); + +test('verify + replay round-trip with plugin contracts', () => { + const artifactDir = '/tmp/apophis-replay-test-' + Date.now(); + const { status: verifyStatus } = runCli([ + 'verify', '--cwd', 'src/cli/__fixtures__/plugin-contracts', + '--profile', 'quick', '--seed', '42', '--artifact-dir', artifactDir, + '--format', 'human', + ]); + + const files = fs.readdirSync(artifactDir); + const artifactFile = files.find((f) => f.endsWith('.json')); + assert.ok(artifactFile, 'artifact file should exist'); + const artifactPath = path.join(artifactDir, artifactFile); + + const { status: replayStatus, stdout: replayOut } = runCli([ + 'replay', '--artifact', artifactPath, + ]); + + fs.rmSync(artifactDir, { recursive: true, force: true }); + + assert.strictEqual(verifyStatus, 0, 'verify should exit 0'); + // Replay exits 2 when artifact has no failures (nothing to reproduce). + // This proves the round-trip: artifact written, artifact loaded successfully. + assert.strictEqual(replayStatus, 2, 'replay exits 2 for no-failure artifact'); + assert.ok(replayOut.includes('no failures to replay'), 'replay reports no failures'); +}); + +test('replay preserves seed from artifact', () => { + const artifactDir = '/tmp/apophis-seed-test-' + Date.now(); + runCli([ + 'verify', '--cwd', 'src/cli/__fixtures__/plugin-contracts', + '--profile', 'quick', '--seed', '99', '--artifact-dir', artifactDir, + '--format', 'human', + ]); + + const files = fs.readdirSync(artifactDir); + const artifactFile = files.find((f) => f.endsWith('.json')); + assert.ok(artifactFile, 'artifact file should exist'); + const artifactPath = path.join(artifactDir, artifactFile); + + const artifactContent = fs.readFileSync(artifactPath, 'utf8'); + const artifact = JSON.parse(artifactContent); + assert.strictEqual(artifact.seed, 99, 'artifact must preserve seed 99'); + + const { status: replayStatus, stdout: replayOut } = runCli([ + 'replay', '--artifact', artifactPath, + ]); + + fs.rmSync(artifactDir, { recursive: true, force: true }); + + assert.strictEqual(replayStatus, 2, 'replay exits 2 for no-failure artifact'); + assert.ok(replayOut.includes('no failures to replay'), 'replay loads artifact with seed 99'); +}); diff --git a/src/test/formula.test.ts b/src/test/formula.test.ts index 45b455a..19233cd 100644 --- a/src/test/formula.test.ts +++ b/src/test/formula.test.ts @@ -222,6 +222,38 @@ test('parse: F shorthand for false', () => { const result = parse('F') assert.deepStrictEqual(result.ast, { type: 'literal', value: false }) }) +test('parse: F == true parses as comparison', () => { + const result = parse('F == true') + assert.deepStrictEqual(result.ast, { + type: 'comparison', op: '==', + left: { type: 'literal', value: false }, + right: { type: 'literal', value: true }, + }) +}) +test('parse: T == false parses as comparison', () => { + const result = parse('T == false') + assert.deepStrictEqual(result.ast, { + type: 'comparison', op: '==', + left: { type: 'literal', value: true }, + right: { type: 'literal', value: false }, + }) +}) +test('parse: F != true parses as comparison', () => { + const result = parse('F != true') + assert.deepStrictEqual(result.ast, { + type: 'comparison', op: '!=', + left: { type: 'literal', value: false }, + right: { type: 'literal', value: true }, + }) +}) +test('parse: F < T parses as comparison', () => { + const result = parse('F < T') + assert.deepStrictEqual(result.ast, { + type: 'comparison', op: '<', + left: { type: 'literal', value: false }, + right: { type: 'literal', value: true }, + }) +}) test('parse: throws on empty formula', () => { assert.throws(() => parse(''), /Empty formula/) }) diff --git a/src/test/integration.test.ts b/src/test/integration.test.ts index 9d1aa67..31d4ce3 100644 --- a/src/test/integration.test.ts +++ b/src/test/integration.test.ts @@ -7,6 +7,7 @@ import assert from 'node:assert' import Fastify from 'fastify' import type { FastifyInstance } from 'fastify' import apophisPlugin from '../index.js' +import { createFastify } from '../fastify-factory.js' import { runPetitTests } from '../quality/petit-runner.js' import { CleanupManager } from '../infrastructure/cleanup-manager.js' import { ScopeRegistry } from '../infrastructure/scope-registry.js' @@ -924,3 +925,198 @@ test('observe: violation event emitted but response still succeeds in observe mo await fastify.close() } }) + +test('observe: production mode still emits non-blocking sink events', async () => { + const originalEnv = process.env.NODE_ENV + process.env.NODE_ENV = 'production' + const fastify = Fastify() + try { + await fastify.register(swagger, {}) + const emitted: ObserveEvent[] = [] + const sink: ObserveSink = { emit: (event) => { emitted.push(event) } } + await fastify.register(apophisPlugin, { + runtime: 'off', + observe: { enabled: true, sinks: [sink], sampling: 1 }, + }) + fastify.get('/prod-observe', { + schema: { + 'x-category': 'observer', + 'x-ensures': ['status:201'], + } as Record + }, async () => ({ ok: true })) + await fastify.ready() + const response = await fastify.inject({ method: 'GET', url: '/prod-observe' }) + assert.strictEqual(response.statusCode, 200, 'production observe must not block responses') + const violationEvents = emitted.filter(e => e.type === 'contract.violation') + assert.ok(violationEvents.length > 0, 'production observe should emit contract violations') + assert.strictEqual(violationEvents[0]!.formula, 'status:201') + } finally { + await fastify.close() + if (originalEnv === undefined) delete process.env.NODE_ENV + else process.env.NODE_ENV = originalEnv + } +}) + +test('createFastify: observe options activate sink-backed runtime observation', async () => { + const emitted: ObserveEvent[] = [] + const sink: ObserveSink = { emit: (event) => { emitted.push(event) } } + const fastify = await createFastify({ + apophis: { + observe: { enabled: true, sinks: [sink], sampling: 1 }, + }, + }) + try { + fastify.get('/factory-observe', { + schema: { + 'x-category': 'observer', + 'x-ensures': ['status:200'], + } as Record + }, async () => ({ ok: true })) + await fastify.ready() + const response = await fastify.inject({ method: 'GET', url: '/factory-observe' }) + assert.strictEqual(response.statusCode, 200) + const passEvents = emitted.filter(e => e.type === 'contract.pass') + assert.ok(passEvents.length > 0, 'createFastify observe options should wire runtime sink events') + } finally { + await fastify.close() + } +}) + +test('createFastify: runtime off alone keeps APOPHIS plugin inactive', async () => { + const fastify = await createFastify({ + apophis: { runtime: 'off' }, + }) + try { + assert.strictEqual(fastify.hasDecorator('apophis'), false) + } finally { + await fastify.close() + } +}) + +// --------------------------------------------------------------------------- +// Plugin composition: plugin contracts merged into route contracts +// --------------------------------------------------------------------------- + +test('plugin composition: contracts merge into route and source is tracked on violation', async () => { + const fastify = Fastify() as unknown as TestFastifyInstance + try { + await fastify.register(swagger, {}) + await fastify.register(apophisPlugin, { + pluginContracts: { + 'auth-plugin': { + appliesTo: '/api/**', + hooks: { + preHandler: { ensures: ['1 == 2'] }, + }, + meta: { name: 'auth-plugin' }, + }, + }, + }) + fastify.get('/api/resource', { + schema: { + 'x-category': 'observer', + response: { 200: { type: 'object' } }, + } as Record, + }, async () => ({ ok: true })) + await fastify.ready() + const suite = await fastify.apophis.contract({ runs: 5 }) + assert.ok(suite.summary.pluginContractsApplied! > 0, 'plugin contracts should be counted as applied') + assert.ok(suite.summary.pluginContractsFailed! > 0, 'always-false ensures should cause at least one plugin contract failure') + } finally { + await fastify.close() + } +}) + +test('plugin composition: missing required extension skips the plugin silently', async () => { + const fastify = Fastify() as unknown as TestFastifyInstance + try { + await fastify.register(swagger, {}) + await fastify.register(apophisPlugin, { + pluginContracts: { + 'requires-missing-ext': { + appliesTo: '/**', + extensions: [{ name: 'nonexistent', required: true }], + hooks: { + preHandler: { ensures: ['1 == 2'] }, + }, + }, + }, + }) + fastify.get('/plain', { + schema: { + 'x-category': 'observer', + 'x-ensures': ['status:200'], + response: { 200: { type: 'object' } }, + } as Record, + }, async () => ({ ok: true })) + await fastify.ready() + const suite = await fastify.apophis.contract({ runs: 5 }) + assert.strictEqual(suite.summary.failed, 0, 'route contract should pass without the skipped plugin formula') + assert.strictEqual( + suite.summary.pluginContractsApplied ?? 0, + 1, + 'only the route-level formula counts when plugin is skipped', + ) + assert.strictEqual( + suite.summary.pluginContractsFailed ?? 0, + 0, + 'no plugin failures when plugin is skipped', + ) + } finally { + await fastify.close() + } +}) + +test('plugin composition: formula from plugin with extension passes when condition met', async () => { + const fastify = Fastify() as unknown as TestFastifyInstance + try { + await fastify.register(swagger, {}) + await fastify.register(apophisPlugin, { + extensions: [ + { + name: 'tenant-gate', + headers: ['check_tenant'], + predicates: { + check_tenant: (ctx: { evalContext: { request: { headers: Record } } }) => ({ + value: ctx.evalContext.request.headers['x-tenant'] === 'acme', + success: true, + }), + }, + } as import('../extension/types.js').ApophisExtension, + ], + pluginContracts: { + 'tenant-validator': { + appliesTo: '/**', + extensions: [{ name: 'tenant-gate', required: true }], + hooks: { + onRequest: { requires: ['check_tenant(this) == true'] }, + }, + }, + }, + }) + fastify.get('/tenanted', { + schema: { + 'x-category': 'observer', + 'x-ensures': ['status:200'], + response: { 200: { type: 'object' } }, + } as Record, + }, async () => ({ ok: true })) + await fastify.ready() + const suite = await fastify.apophis.contract({ + runs: 5, + variants: [{ name: 'acme', headers: { 'x-tenant': 'acme' } }], + }) + assert.ok(suite.summary.pluginContractsApplied! > 0, 'tenant validator should be applied') + assert.strictEqual(suite.summary.pluginContractsFailed ?? 0, 0, 'tenant check should pass for acme') + assert.strictEqual(suite.summary.failed, 0, 'all tests should pass') + } finally { + await fastify.close() + } +}) + +// --------------------------------------------------------------------------- +// Plugin composition + scenarios (integration) +// --------------------------------------------------------------------------- +// NOTE: scenario + plugin contract integration test resides in +// _smoke-scenario-plugin.ts for isolation from cross-test state. +// The programmatic path is verified by running that smoke script directly. diff --git a/src/types/core.ts b/src/types/core.ts index da62abb..92854e5 100644 --- a/src/types/core.ts +++ b/src/types/core.ts @@ -45,6 +45,8 @@ export interface RouteContract { outbound?: readonly OutboundBinding[] /** Route-level variants for negotiated content-type or feature testing. Extracted from schema['x-variants']. */ variants?: RouteVariant[] + /** Formula → source mapping for plugin contract attribution. Populated at test time. */ + formulaSources?: Record } // ============================================================================