feat: plugin contract e2e, qualify --changed, production observe, regressions

This commit is contained in:
John Dvorak
2026-05-22 11:05:52 -07:00
parent d0523fcc2d
commit 1de735ee08
34 changed files with 1392 additions and 122 deletions
+23 -14
View File
@@ -8,11 +8,11 @@ This audit is based on code inspection plus command verification, not documentat
## Executive Summary ## Executive Summary
APOPHIS has real product value. It is not just a schema wrapper: it gives Fastify teams a way to express and verify behavioral API promises that OpenAPI/JSON Schema cannot cover, especially cross-route invariants such as create/read consistency, delete semantics, auth/session flows, state transitions, idempotency, outbound dependency expectations, and replayable counterexamples. APOPHIS has real product value. It is not just a schema wrapper: it gives Fastify teams a way to express, verify, and observe behavioral API promises that OpenAPI/JSON Schema cannot cover, especially cross-route invariants such as create/read consistency, delete semantics, auth/session flows, state transitions, idempotency, outbound dependency expectations, and replayable counterexamples.
I would adopt APOPHIS today as a focused behavioral verification tool for Fastify v5 ESM services. I would start with CI `verify` and a small number of high-value contracts, then expand into `qualify` and runtime observation once the team has clear operating guidance. I would adopt APOPHIS today as a focused behavioral verification tool for Fastify v5 ESM services. I would start with CI `verify` and a small number of high-value contracts, then expand into `qualify` and runtime observation once the team has clear operating guidance.
I would not yet treat it as a complete production observability platform or a turnkey organization-wide release gate. The core implementation is strong, but the remaining value gap is mostly around operational maturity: standalone observe activation, deeper tests around recent CLI behavior, richer scenario authoring, and clearer release-gate recommendations. I would not yet treat it as a complete production observability platform or a turnkey organization-wide release gate. The core implementation is strong, but the remaining value gap is mostly around operational maturity: standalone observe process management, richer scenario authoring, and organization-specific release-gate policy.
Adoption verdict: strong team pilot candidate, credible standardization candidate after the remaining gaps below are addressed. Adoption verdict: strong team pilot candidate, credible standardization candidate after the remaining gaps below are addressed.
@@ -34,10 +34,10 @@ Observed results:
|---|---:| |---|---:|
| Typecheck | pass | | Typecheck | pass |
| Build | pass | | Build | pass |
| Source tests | 587 pass, 0 fail | | Source tests | 590 pass, 0 fail |
| CLI tests | 311 pass, 0 fail | | CLI tests | 320 pass, 0 fail |
| Docs smoke tests | 4 pass, 0 fail | | Docs smoke tests | 4 pass, 0 fail |
| Total tests | 902 pass, 0 fail | | Total tests | 921 pass, 0 fail |
The working tree contains many broader project changes unrelated to this audit. This document evaluates the current working tree state. The working tree contains many broader project changes unrelated to this audit. This document evaluates the current working tree state.
@@ -51,7 +51,7 @@ Mostly yes for behavioral verification. Partially for production observation and
| Deterministic CI verification | Yes, materially. CLI `verify` now honors configured `runs`, uses seeded request generation, emits artifacts, supports route filters, replay metadata, and machine-readable output. | | Deterministic CI verification | Yes, materially. CLI `verify` now honors configured `runs`, uses seeded request generation, emits artifacts, supports route filters, replay metadata, and machine-readable output. |
| Cross-route behavior | Yes for supported formula operations and route-call semantics. This is the most differentiated value. | | Cross-route behavior | Yes for supported formula operations and route-call semantics. This is the most differentiated value. |
| Runtime validation | Yes when the plugin is explicitly configured outside production. Production enforcement is intentionally blocked. | | Runtime validation | Yes when the plugin is explicitly configured outside production. Production enforcement is intentionally blocked. |
| Runtime observation | Partially. Programmatic plugin observation exists and emits non-blocking sink events with sampling. The CLI validates/report readiness but does not attach to or run a service. | | Runtime observation | Yes for programmatic production-safe hooks. APOPHIS emits non-blocking sink events with sampling in production when `observe.enabled` and sinks are configured. The CLI validates/report readiness but does not attach to or run a service. |
| Stateful/scenario/chaos qualification | Partially. The runner and artifacts are useful, route discovery is now shared with verify, and config supports scenarios/chaos knobs. Scenario authoring is still young and needs more real-world examples/tests. | | Stateful/scenario/chaos qualification | Partially. The runner and artifacts are useful, route discovery is now shared with verify, and config supports scenarios/chaos knobs. Scenario authoring is still young and needs more real-world examples/tests. |
| Outbound dependency mocking | Useful but intentionally process-global. The misleading scoped `undici-mock-agent` option has been removed. Teams still need careful test isolation. | | Outbound dependency mocking | Useful but intentionally process-global. The misleading scoped `undici-mock-agent` option has been removed. Teams still need careful test isolation. |
| Team-safe onboarding | Good. The package has CLI help, init/doctor/replay/verify/qualify/observe, config validation, machine output, docs smoke tests, packaging tests, and production safety checks. | | Team-safe onboarding | Good. The package has CLI help, init/doctor/replay/verify/qualify/observe, config validation, machine output, docs smoke tests, packaging tests, and production safety checks. |
@@ -102,12 +102,17 @@ The following earlier adoption risks have been addressed in the current working
|---|---| |---|---|
| CLI `verify` runs | `VerifyRunnerDeps` accepts `runs`; `verifyCommand()` passes resolved config; `runVerify()` executes contracts for `contractRuns`. | | CLI `verify` runs | `VerifyRunnerDeps` accepts `runs`; `verifyCommand()` passes resolved config; `runVerify()` executes contracts for `contractRuns`. |
| Observe sampling | `hook-validator.ts` gates sink emission using `opts.observe.sampling` before emitting pass/violation/error events. | | Observe sampling | `hook-validator.ts` gates sink emission using `opts.observe.sampling` before emitting pass/violation/error events. |
| Production observe activation | `apophisPlugin` now keeps blocking runtime validation disabled in production while allowing non-blocking observe sinks to emit pass/violation/error events. |
| Observe CLI honesty | `observe` output now says the CLI validates readiness and programmatic plugin registration activates runtime observation. | | Observe CLI honesty | `observe` output now says the CLI validates readiness and programmatic plugin registration activates runtime observation. |
| Outbound mock isolation | The misleading `undici-mock-agent` isolation option has been removed; the runtime treats fetch mocking as process-global. | | Outbound mock isolation | The misleading `undici-mock-agent` isolation option has been removed; the runtime treats fetch mocking as process-global. |
| Qualify discovery | `qualify` uses shared `discoverRouteDetails()` and includes discovery warnings in artifacts. | | Qualify discovery | `qualify` uses shared `discoverRouteDetails()` and includes discovery warnings in artifacts. |
| Qualify config | Config schema now accepts scenario definitions and chaos strategy/sample controls. | | Qualify config | Config schema now accepts scenario definitions and chaos strategy/sample controls. |
| Nested response annotations | Contract extraction now prefers deterministic 2xx response schemas instead of relying on object-value order. | | Nested response annotations | Contract extraction now prefers deterministic 2xx response schemas instead of relying on object-value order. |
| `--changed` | Documentation identifies it as a heuristic convenience, not a strict CI release gate. | | `--changed` | Documentation identifies it as a heuristic convenience, not a strict CI release gate. |
| Plugin contracts (end-to-end) | Full pipeline: config schema, plugin registration, compose+merge in all runners, precondition→skip, auto-inject headers, source attribution (`formulaSources`), failure counting, `drainWarnings()` collection, production safety. Wired through verify, qualify (scenario/stateful/chaos), and replay. |
| Artifact pipeline CI/CD | 6 CI-facing regression tests: json-summary parseable, ndjson-summary parseable, `--quiet` persistence, skipped field presence, exit code 0 on pass, qualify json-summary. Verify→replay round-trip test with plugin contracts. |
| CLI output hygiene | Console.warn bleeding fixed (`drainWarnings`); `json-summary``human` format normalization bug fixed; `--quiet` no longer suppresses machine format output. |
| Qualify --changed | Qualify now supports `--changed` flag with same git-diff heuristic as verify. Prints match count, exits 0 when no changed routes. |
## Remaining Adoption Gaps ## Remaining Adoption Gaps
@@ -121,7 +126,7 @@ The implementation supports runtime observation only when the application explic
- Integration tests prove sink sync failures and async rejections never change route responses. - Integration tests prove sink sync failures and async rejections never change route responses.
- Integration tests prove sampling: 0 suppresses all events; sampling: 1 emits expected `contract.pass`/`contract.violation` events. - Integration tests prove sampling: 0 suppresses all events; sampling: 1 emits expected `contract.pass`/`contract.violation` events.
**Still open:** A future `apophis observe --app ./app.ts` mode that activates a running service observer. **Still open:** A future `apophis observe --app ./app.ts` mode that imports and starts a service for local/staging smoke observation. Production observation itself is now programmatic and active through plugin registration.
### P1: Recent `verify` Runs Behavior Now Has Regression Tests ### P1: Recent `verify` Runs Behavior Now Has Regression Tests
@@ -130,7 +135,7 @@ The implementation supports runtime observation only when the application explic
- Regression test proves `runs: 5` scales multiplicatively from `runs: 1`. - Regression test proves `runs: 5` scales multiplicatively from `runs: 1`.
- Regression test proves `runs: 10` is deterministic at the same seed. - Regression test proves `runs: 10` is deterministic at the same seed.
**Still open:** Variant-aware runs test (verifying run budget is per-variant or shared). **Completed:** Variant-aware runs regression proves the run budget is applied per variant.
### P1: Qualify Product Shape Improved ### P1: Qualify Product Shape Improved
@@ -139,7 +144,7 @@ The implementation supports runtime observation only when the application explic
- Configured-scenario qualify test added (independent of OAuth fixture routes). - Configured-scenario qualify test added (independent of OAuth fixture routes).
- `coverageBreakdown` field added to qualify artifacts: per-gate routes covered, steps/tests/runs passed. - `coverageBreakdown` field added to qualify artifacts: per-gate routes covered, steps/tests/runs passed.
**Still open:** Clear guidance for nightly/staging use versus pull-request gating in qualify docs. **Completed:** `docs/qualify.md` now documents pull-request versus nightly/staging gate guidance.
### P1: Outbound Mocks Process-Global, Honestly Documented ### P1: Outbound Mocks Process-Global, Honestly Documented
@@ -210,14 +215,14 @@ High-value first contracts:
| Fastify fit | 8/10 | Strong plugin/inject/decorator alignment; discovery order still matters. | | Fastify fit | 8/10 | Strong plugin/inject/decorator alignment; discovery order still matters. |
| Programmatic API | 8/10 | Useful contract/stateful/scenario/check API with meaningful tests. | | Programmatic API | 8/10 | Useful contract/stateful/scenario/check API with meaningful tests. |
| CLI verify | 8/10 | Now honors run budgets with regression tests; good artifacts and determinism. | | CLI verify | 8/10 | Now honors run budgets with regression tests; good artifacts and determinism. |
| Observe | 7/10 | Runtime sink primitives, sampling, and sink-failure-resilience exist with tests. Production-style docs added. Standalone operational story not complete. | | Observe | 8/10 | Production-safe non-blocking sink emission, sampling, and sink-failure-resilience exist with tests. Standalone process-management story is still future work. |
| Qualify | 7/10 | Improved discovery/config/scenarios. Coverage breakdown in artifacts. Needs richer scenario examples and gating guidance. | | Qualify | 7/10 | Improved discovery/config/scenarios. Coverage breakdown in artifacts. Needs richer scenario examples and gating guidance. |
| Outbound mocking | 7/10 | Useful and honest about process-global behavior. Docs and README explicit. True scoped mocking remains future work. | | Outbound mocking | 7/10 | Useful and honest about process-global behavior. Docs and README explicit. True scoped mocking remains future work. |
| Docs | 8/10 | Broad and increasingly precise. Observe and qualify docs expanded with real code examples. | | Docs | 8/10 | Broad and increasingly precise. Observe and qualify docs expanded with real code examples. |
| Packaging | 9/10 | Strong for a Node/Fastify package. | | Packaging | 9/10 | Strong for a Node/Fastify package. |
| Team readiness | 8/10 | Ready for pilot and selective CI use with regression-locked verification behavior. | | Team readiness | 8/10 | Ready for pilot and selective CI use with regression-locked verification behavior. |
Overall: 8/10 for real team pilot use. Potential 9/10 if observe gains a clearer production story and qualify gets first-class CI workflow guidance. Overall: 8.5/10 for real team pilot use. Potential 9/10 if standalone observe process management and richer scenario libraries become first-class.
## Highest-Impact Next Work ## Highest-Impact Next Work
@@ -226,9 +231,13 @@ Overall: 8/10 for real team pilot use. Potential 9/10 if observe gains a clearer
3. ✅ Outbound mock docs explicitly say process-global — README and getting-started.md updated. 3. ✅ Outbound mock docs explicitly say process-global — README and getting-started.md updated.
4. ✅ Qualify scenario config documented with full examples in qualify.md. 4. ✅ Qualify scenario config documented with full examples in qualify.md.
5. ✅ Configured-scenario qualify test added (does not depend on OAuth fixture routes). 5. ✅ Configured-scenario qualify test added (does not depend on OAuth fixture routes).
6. Add full production-style observe example with a real collector sink implementation. 6. ✅ Full production-style observe example with real collector sink implementation added to docs/observe.md.
7. Improve qualify artifact coverage summaries to distinguish route-contract, scenario, stateful, and chaos coverage more clearly. 7. ✅ Plugin contract support end-to-end: docs, tests, all runners wired.
8. Consider true scoped outbound mocking (undici dispatcher) only if concurrent in-process dependency tests become a core promise. 8. ✅ Artifact pipeline CI/CD regression tests: json-summary, ndjson-summary, --quiet, skipped field, exit codes.
9. ✅ Qualify --changed implemented.
10. Add standalone observe process management (`apophis observe --app ./app.ts`) for local/staging observation.
11. Add route ownership / file-to-route maps for precise `--changed` filtering.
12. Consider true scoped outbound mocking (undici dispatcher) only if concurrent in-process dependency tests become a core promise.
## Bottom Line ## Bottom Line
+6 -1
View File
@@ -136,7 +136,12 @@ import { createFastify } from '@apophis/fastify'
const app = await createFastify({ const app = await createFastify({
logger: true, logger: true,
apophis: { runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off' }, apophis: {
runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off',
observe: process.env.NODE_ENV === 'production'
? { enabled: true, sampling: 0.1, sinks: [metricsSink] } // your ObserveSink
: undefined,
},
}) })
// Register swagger, auth, plugins, and routes after app creation. // Register swagger, auth, plugins, and routes after app creation.
``` ```
+36 -15
View File
@@ -162,30 +162,19 @@ export default {
## Programmatic Runtime Activation ## Programmatic Runtime Activation
The CLI only validates configuration. To activate runtime observation, register The CLI only validates configuration. To activate runtime observation, register
APOPHIS with observe options in your application: APOPHIS with observe options in your application before routes are registered.
Observe remains active in production because it is non-blocking; blocking
runtime validation still stays disabled in production.
```typescript ```typescript
import Fastify from 'fastify' import Fastify from 'fastify'
import apophisPlugin from '@apophis/fastify' import apophisPlugin from '@apophis/fastify'
import type { ObserveSink, ObserveEvent } from '@apophis/fastify'
const app = Fastify({ logger: true }) const app = Fastify({ logger: true })
// Register APOPHIS with observe enabled.
// This emits non-blocking contract pass/violation/error events
// for every covered request, gated by sampling.
await app.register(apophisPlugin, {
runtime: 'warn',
observe: {
enabled: true,
sampling: 0.1, // observe 10% of requests
sinks: [metricsSink],
},
})
// Implement the ObserveSink interface. // Implement the ObserveSink interface.
// Capture events to your preferred observability backend. // Capture events to your preferred observability backend.
import type { ObserveSink, ObserveEvent } from '@apophis/fastify'
const metricsSink: ObserveSink = { const metricsSink: ObserveSink = {
emit(event: ObserveEvent) { emit(event: ObserveEvent) {
// Emit a counter for each contract evaluation // Emit a counter for each contract evaluation
@@ -205,11 +194,43 @@ const metricsSink: ObserveSink = {
} }
}, },
} }
// Register APOPHIS with observe enabled.
// This emits non-blocking contract pass/violation/error events
// for every covered request, gated by sampling.
await app.register(apophisPlugin, {
runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off',
observe: {
enabled: true,
sampling: 0.1, // observe 10% of requests
sinks: [metricsSink],
},
})
```
For new services, `createFastify()` wires discovery and APOPHIS before your
routes, which avoids the most common ordering mistake:
```typescript
import { createFastify } from '@apophis/fastify'
const app = await createFastify({
logger: true,
apophis: {
runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off',
observe: {
enabled: true,
sampling: 0.1,
sinks: [metricsSink],
},
},
})
``` ```
Key constraints: Key constraints:
- Sink `emit()` can be sync or async (returns `void | Promise<void>`). - Sink `emit()` can be sync or async (returns `void | Promise<void>`).
- Sink rejections and thrown errors are silently caught — they never affect the route response or status code. - Sink rejections and thrown errors are silently caught — they never affect the route response or status code.
- In production, observe hooks still run when `observe.enabled` and `observe.sinks` are configured; blocking runtime validation does not.
- Sampling is applied per-formula evaluation via `Math.random() < sampling`. - Sampling is applied per-formula evaluation via `Math.random() < sampling`.
At `sampling: 1` every formula is emitted. At `sampling: 0` nothing is emitted. At `sampling: 1` every formula is emitted. At `sampling: 0` nothing is emitted.
- Only routes with APOPHIS annotations (`x-ensures`, `x-requires`) produce events. - Only routes with APOPHIS annotations (`x-ensures`, `x-requires`) produce events.
+431
View File
@@ -0,0 +1,431 @@
# Plugin Contracts
Define behavioral rules that apply across all matching routes and are automatically injected into every matching route's contract evaluation.
Plugin contracts let teams codify cross-cutting expectations — "all API routes must carry an auth header," "all POST handlers must close the database connection," "all responses must include a request ID." They're defined once in config and evaluated alongside route-level contracts without per-route boilerplate.
## Table of Contents
- [When to Use It](#when-to-use-it)
- [Config Format](#config-format)
- [PluginContractSpec Reference](#plugincontractspec-reference)
- [Route Pattern Matching](#route-pattern-matching)
- [Precondition → Skip Semantics](#precondition--skip-semantics)
- [Auto-Injected Headers](#auto-injected-headers)
- [Failure Attribution](#failure-attribution)
- [Extension Dependencies](#extension-dependencies)
- [Production Safety](#production-safety)
- [Complete Example](#complete-example)
- [Limitations](#limitations)
## When to Use It
Plugin contracts are ideal for rules that span many routes and don't belong on a single handler:
- Auth header presence across all API routes
- Response metadata (request IDs, timing headers, version stamps)
- Tenant isolation checks that apply to every route in a prefix
- Database connection lifecycle guards on write endpoints
- Content-type enforcement on all response paths
Don't use plugin contracts for route-specific behavioral rules — those belong on the route schema via `x-requires` and `x-ensures`.
## Config Format
### apophis.config.js
```javascript
export default {
pluginContracts: {
"auth-presence": {
appliesTo: "/api/**",
hooks: {
onRequest: {
requires: ["request_headers(this).authorization != null"]
}
}
},
"rate-limit-respected": {
appliesTo: "POST /api/**",
hooks: {
onResponse: {
ensures: ["status != 429"]
}
},
meta: { name: "rate-limit-respected", description: "Rate limit should not be hit under normal CI load" }
}
}
}
```
### Programmatic (Fastify plugin registration)
```javascript
await app.register(apophisPlugin, {
pluginContracts: {
"auth-presence": {
appliesTo: "/api/**",
hooks: {
onRequest: { requires: ["request_headers(this).authorization != null"] }
}
}
}
})
```
Contracts registered programmatically take the same path through the plugin contract registry as config-defined contracts. Both sources are merged into the composed contract at evaluation time.
## PluginContractSpec Reference
| Field | Required | Type | Description |
|---|---|---|---|
| `appliesTo` | yes | `string` | Route pattern (see [Route Pattern Matching](#route-pattern-matching)) |
| `hooks` | yes | `object` | Hook-phase contracts keyed by Fastify lifecycle hook name |
| `hooks.<phase>.requires` | no | `string[]` | APOSTL formulas that must be `true` **before** this phase executes |
| `hooks.<phase>.ensures` | no | `string[]` | APOSTL formulas that must be `true` **after** this phase executes |
| `meta` | no | `object` | Display metadata |
| `meta.name` | no | `string` | Human-readable plugin name |
| `meta.version` | no | `string` | Plugin version |
| `meta.description` | no | `string` | Description of what the plugin enforces |
| `extensions` | no | `array` | APOPHIS extensions this plugin depends on |
### Supported hook phases
Keys in `hooks` correspond to Fastify lifecycle hook names:
- `onRequest` — evaluated before the route handler runs
- `preHandler` — evaluated before the handler, after shared hooks
- `preSerialization` — evaluated before the response body is serialized
- `onSend` — evaluated before the response is sent
- `onResponse` — evaluated after the response is sent (body already consumed)
At least one of `requires` or `ensures` must be present per phase. Empty phases are ignored.
### Extension dependencies
```javascript
pluginContracts: {
"jwt-auth": {
appliesTo: "/api/**",
extensions: [{ name: "jwt-decoder", required: true }],
hooks: {
onRequest: { requires: ["decode_jwt(this).valid == true"] }
}
}
}
```
If a required extension is missing at registration time, the plugin's contracts are skipped with a warning. Extensions are resolved from the APOPHIS extension registry at evaluation time, not at route discovery time.
## Route Pattern Matching
The `appliesTo` field controls which routes a plugin contract covers.
### Pattern syntax
| Pattern | Matches | Does NOT match |
|---|---|---|
| `"/api/users"` | `/api/users` (exact) | `/api/users/42`, `/api/v2/users` |
| `"/api/**"` | `/api/users`, `/api/users/42`, `/api/users/:id/posts` | `/v2/api/users` |
| `"/api/*"` | `/api/users`, `/api/status` | `/api/users/42`, `/api/users/:id/posts` |
| `"**"` | Everything | — |
| `"POST /api/**"` | `POST /api/users`, `POST /api/orders/42` | `GET /api/users`, `POST /v2/api/users` |
The `/**` suffix matches any path at any depth below the prefix. The `/*` suffix matches exactly one segment below the prefix.
### Method-prefixed patterns
Prepend the HTTP method to restrict a pattern to a specific verb:
```javascript
{
appliesTo: "POST /api/**",
hooks: {
preHandler: { requires: ["request_body(this) != null"] }
}
}
```
When a method is specified, the pattern only matches routes registered with that HTTP method. The method must be a valid HTTP method in uppercase.
Pattern matching is evaluated against the route's path string only (the raw argument passed to `fastify.get()`, `fastify.post()`, etc.), not against the dereferenced URL with prefix applied. Fastify prefix application happens before pattern matching, so a route registered as `fastify.get("/users")` under a `prefix: "/api"` is matched against `/api/users`.
## Precondition → Skip Semantics
When a plugin contract defines `requires` formulas, the runner evaluates them before `ensures`. If any require formula fails:
1. Require failure is **not** counted as a contract failure
2. All `ensures` formulas for that route+run are **skipped**
3. The skip is counted separately from failures in the artifact summary
This distinguishes "the plugin's prerequisites aren't met" from "the route's behavior violated the contract."
### Example
```javascript
pluginContracts: {
"tenant-check": {
appliesTo: "/api/**",
hooks: {
onRequest: {
requires: ["request_headers(this).x-tenant-id != null"],
ensures: ["status != 500"]
}
}
}
}
```
For a route without the `x-tenant-id` header:
- The `requires` check fails → the `ensures` (`status != 500`) is **skipped**
- `skippedCount` increments; `failedCount` does not
For a route with the `x-tenant-id` header:
- The `requires` check passes → the `ensures` is evaluated normally
- If `status == 500`, it's counted as a plugin contract **failure**
## Auto-Injected Headers
To avoid every test being skipped when a plugin contract checks for header presence, APOPHIS automatically injects headers into test requests when the precondition pattern is simple enough.
### Supported patterns
| Formula pattern | Injected header |
|---|---|
| `request_headers(this).<name> != null` | `<name>: test-value` |
| `request_headers(this).<name> == "value"` | `<name>: value` |
### Example
```javascript
pluginContracts: {
"auth-presence": {
appliesTo: "/api/**",
hooks: {
onRequest: {
requires: ["request_headers(this).authorization != null"]
}
}
}
}
```
APOPHIS detects the `!= null` pattern on `authorization` and injects `authorization: test-value` into every test request. The precondition passes automatically without user intervention.
Auto-injection works in all runners: verify, qualify (scenario, stateful, chaos), and replay.
### Unsupported patterns
Complex preconditions are not auto-injected. For these, the requires check will fail and the route's ensures will be skipped:
```javascript
// NOT auto-injected — the route's ensures will be skipped
requires: ["request_headers(this).x-count >= 5"]
```
For these cases, provide the header explicitly via route variants:
```javascript
await fastify.apophis.contract({
runs: 5,
variants: [{ name: "with-count", headers: { "x-count": "10" } }]
})
```
## Failure Attribution
When a plugin contract formula fails, the violation carries metadata that distinguishes it from route-level failures.
### ContractViolation fields
| Field | Value | Meaning |
|---|---|---|
| `source` | `"plugin:auth-presence"` | Which plugin added the formula |
| `phase` | `"onRequest"` | Which hook phase the formula belongs to |
In human-readable output, plugin failures appear separately from route contract failures:
```text
Plugin contract violation (plugin:auth-presence)
GET /api/users
Phase: onRequest
Expected
response_headers(this).x-request-id != null
Observed
response_headers(this).x-request-id was null
```
The artifact summary tracks plugin contract outcomes independently:
```json
{
"summary": {
"pluginContractsApplied": 12,
"pluginContractsFailed": 2,
"failed": 1,
"passed": 10
}
}
```
- `pluginContractsApplied` — total plugin contract formulas evaluated across all routes
- `pluginContractsFailed` — plugin contract formulas that failed
- `failed` — total failures (route + plugin)
- `passed` — route-level contract passes (plugin-only formulas not counted here)
A route can pass its own contracts and still have plugin contract failures. The CLI exit code reflects the aggregate: any failure (route or plugin) produces exit code 1.
## Extension Dependencies
Plugin contracts can declare dependencies on APOPHIS extensions. Extension resolution happens at evaluation time via the extension registry.
### Declaring dependencies
```javascript
pluginContracts: {
"jwt-auth": {
appliesTo: "/api/**",
extensions: [{ name: "jwt-decoder", required: true }],
hooks: {
onRequest: { requires: ["decode_jwt(this).valid == true"] }
}
}
}
```
Each extension reference has:
- `name` (required) — matches the extension's registered name
- `required` (optional, default `true`) — if `false`, missing extension emits a warning but does not skip the plugin
### Missing extension behavior
If a required extension is not registered, the plugin's contracts are **skipped** for all matching routes. Warnings are collected in the registry and surfaced via `drainWarnings()`:
```text
Plugin 'jwt-auth' requires extensions [jwt-decoder] which are not registered. Skipping its contracts.
```
Warnings appear in the CLI output and the artifact's `warnings` array.
### Extension registration
Extensions are registered alongside plugin contracts:
```javascript
await app.register(apophisPlugin, {
extensions: [
{
name: "jwt-decoder",
headers: ["decode_jwt"],
predicates: {
decode_jwt: (ctx) => ({ value: true, success: true })
}
}
],
pluginContracts: {
"jwt-auth": {
appliesTo: "/api/**",
extensions: [{ name: "jwt-decoder", required: true }],
hooks: {
onRequest: { requires: ["decode_jwt(this).valid == true"] }
}
}
}
})
```
## Production Safety
`pluginContracts` is a **test-only** feature. If `pluginContracts` is non-empty in a production environment (`NODE_ENV=production`), APOPHIS throws at plugin registration:
```text
APOPHIS: Unsafe options detected in production: pluginContracts.
These features are test-only and must not be enabled in production.
Remove them from ApophisOptions or set NODE_ENV=test.
```
This is enforced by `validateProductionSafety()` before any contract evaluation runs. There is no escape hatch.
## Complete Example
### Config (`apophis.config.js`)
```javascript
export default {
profiles: {
quick: { preset: "safe-ci" },
},
presets: {
"safe-ci": { runs: 10 },
},
pluginContracts: {
"auth-required": {
appliesTo: "/api/**",
hooks: {
onRequest: {
requires: ["request_headers(this).authorization != null"],
},
},
meta: { name: "auth-required", description: "All API routes require an Authorization header" },
},
"request-id": {
appliesTo: "/api/**",
hooks: {
onSend: {
ensures: ["response_headers(this).x-request-id != null"],
},
},
meta: { name: "request-id", description: "All API responses must include x-request-id" },
},
},
}
```
### Route (`routes/users.js`)
```javascript
app.get("/api/users", {
schema: {
"x-category": "observer",
"x-ensures": ["status:200", "response_body(this) is Array"],
response: {
200: {
type: "array",
items: { type: "object", properties: { id: { type: "number" } } },
},
},
},
}, async (request, reply) => {
return [{ id: 1 }, { id: 2 }]
})
```
### Execution (`apophis verify`)
1. Sends a request to `GET /api/users`
2. `auth-required` plugin: `onRequest.requires``request_headers(this).authorization != null` → APOPHIS auto-injects `authorization: test-value` → passes
3. Route-level ensures: `status:200` → passes; `response_body(this) is Array` → passes
4. `request-id` plugin: `onSend.ensures``response_headers(this).x-request-id != null` → if the handler doesn't set `x-request-id`, this **fails** as a `plugin:request-id` contract violation
```text
Plugin contract violation (plugin:request-id)
GET /api/users
Phase: onSend
Expected
response_headers(this).x-request-id != null
Observed
response_headers(this).x-request-id was null
```
## Limitations
- **`appliesTo` matches by route path only** by default. Use method-prefixed syntax (`"POST /api/**"`) for HTTP-method-specific rules.
- **Header auto-injection only handles simple patterns** (`!= null` and `== "value"`). Complex preconditions like `>= 5` are not auto-injected; the route's ensures will be skipped.
- **No per-route opt-out.** If a plugin contract matches a route, it always applies. Use narrower `appliesTo` patterns or separate route prefixes to exclude routes.
- **Extension resolution is lazy.** Extensions are checked at route evaluation time, not at registration time. A missing extension is only discovered when a matching route is evaluated.
- **At least one of `requires` or `ensures` must be present per phase.** A phase with neither is silently ignored.
- **Runtime hook ordering follows Fastify's lifecycle.** Plugin contracts in `onResponse` cannot inspect the response body (it has already been sent to the client).
@@ -0,0 +1,4 @@
export default {
profiles: { quick: { preset: "safe-ci" } },
presets: { "safe-ci": { runs: 3 } },
}
@@ -0,0 +1,34 @@
import Fastify from "fastify";
const app = Fastify({ logger: false });
await app.register(import("@fastify/swagger"), {
openapi: { info: { title: "Plugin Contracts Fixture", version: "1.0.0" } },
});
let apophisPlugin;
try {
({ default: apophisPlugin } = await import("../../../index.js"));
} catch {
({ default: apophisPlugin } = await import("../../../../dist/index.js"));
}
await app.register(apophisPlugin, {
runtime: "off",
pluginContracts: {
"required-header": {
appliesTo: "/api/**",
hooks: { onRequest: { requires: ["request_headers(this).x-custom != null"] } },
},
},
});
app.get("/api/data", {
schema: {
"x-category": "observer",
"x-ensures": ["status:200"],
response: { 200: { type: "object", properties: { ok: { type: "boolean" } } } },
},
}, async () => ({ ok: true }));
export default app;
@@ -0,0 +1 @@
{"name":"plugin-contracts-fixture","type":"module","private":true}
+19 -1
View File
@@ -63,8 +63,26 @@ export async function runChaosOnRoute(
const nonDelayEvents = events.filter(e => e.type !== 'inbound-delay') const nonDelayEvents = events.filter(e => e.type !== 'inbound-delay')
const application = applyChaosToExecution(ctx, nonDelayEvents) const application = applyChaosToExecution(ctx, nonDelayEvents)
// 5. Validate post-chaos contracts: did the route still satisfy its contracts? // Validate route contracts after chaos. Compose plugin contracts if available.
let contractsPassed = true let contractsPassed = true
if (deps.pluginContractRegistry) {
const composed = deps.pluginContractRegistry.composeContracts(route)
if (!route.formulaSources) route.formulaSources = {}
for (const phase of Object.values(composed.phases)) {
for (const req of phase.requires) {
if (!route.requires.includes(req.formula)) {
route.requires.push(req.formula)
route.formulaSources[req.formula] = req.source
}
}
for (const ens of phase.ensures) {
if (!route.ensures.includes(ens.formula)) {
route.ensures.push(ens.formula)
route.formulaSources[ens.formula] = ens.source
}
}
}
}
if (application.applied && route.ensures.length > 0) { if (application.applied && route.ensures.length > 0) {
const { validatePostconditionsAsync } = await import('../../../domain/contract-validation.js') const { validatePostconditionsAsync } = await import('../../../domain/contract-validation.js')
const result = await validatePostconditionsAsync( const result = await validatePostconditionsAsync(
+55 -4
View File
@@ -30,6 +30,7 @@ import {
type StepTrace, type StepTrace,
type CleanupFailure, type CleanupFailure,
} from './runner.js' } from './runner.js'
import { isGitRepo, filterChangedRoutes } from '../verify/runner.js'
import { SeededRng } from '../../../infrastructure/seeded-rng.js' import { SeededRng } from '../../../infrastructure/seeded-rng.js'
import type { ScenarioConfig, TestConfig, RouteContract, ChaosConfig } from '../../../types.js' import type { ScenarioConfig, TestConfig, RouteContract, ChaosConfig } from '../../../types.js'
import { renderHumanArtifact } from '../../renderers/human.js' import { renderHumanArtifact } from '../../renderers/human.js'
@@ -64,6 +65,7 @@ function coerceTimeout(value: unknown): number | undefined {
export interface QualifyOptions { export interface QualifyOptions {
profile?: string profile?: string
seed?: number seed?: number
changed?: boolean
config?: string config?: string
cwd?: string cwd?: string
format?: 'human' | 'json' | 'ndjson' format?: 'human' | 'json' | 'ndjson'
@@ -252,7 +254,8 @@ export function buildArtifact(
const totalTests = const totalTests =
runResult.scenarioResults.reduce((sum, s) => sum + s.steps.length, 0) + runResult.scenarioResults.reduce((sum, s) => sum + s.steps.length, 0) +
(runResult.statefulResult?.tests.length ?? 0) (runResult.statefulResult?.summary.passed ?? 0) +
(runResult.statefulResult?.summary.failed ?? 0)
const passedTests = const passedTests =
runResult.scenarioResults.reduce((sum, s) => sum + s.summary.passed, 0) + runResult.scenarioResults.reduce((sum, s) => sum + s.summary.passed, 0) +
@@ -282,7 +285,11 @@ export function buildArtifact(
.filter(Boolean) .filter(Boolean)
)] )]
const statefulRoutes = runResult.statefulResult const statefulRoutes = runResult.statefulResult
? [...new Set(runResult.statefulResult.tests.map(t => normalizeRouteIdentity(t.name)))] ? [...new Set(
runResult.statefulResult.tests
.filter(t => !t.directive)
.map(t => normalizeRouteIdentity(t.name))
)]
: [] : []
const chaosRoutesPlanned: string[] = [] const chaosRoutesPlanned: string[] = []
const chaosRoutesExecuted: string[] = [] const chaosRoutesExecuted: string[] = []
@@ -343,6 +350,7 @@ export function buildArtifact(
total: totalTests, total: totalTests,
passed: passedTests, passed: passedTests,
failed: failures.length, failed: failures.length,
skipped: (runResult.statefulResult?.summary.skipped ?? 0),
}, },
executionSummary, executionSummary,
coverageBreakdown, coverageBreakdown,
@@ -437,6 +445,7 @@ export async function qualifyCommand(
const { const {
profile, profile,
seed: explicitSeed, seed: explicitSeed,
changed,
config: configPath, config: configPath,
cwd, cwd,
artifactDir, artifactDir,
@@ -566,7 +575,31 @@ export async function qualifyCommand(
try { try {
// 9. Discover routes for chaos (with discovery metadata) // 9. Discover routes for chaos (with discovery metadata)
const appRoutes = await discoverAppRoutes(fastify) const allRoutes = await discoverAppRoutes(fastify)
let appRoutes = allRoutes
// 9a. Apply --changed filter
if (changed) {
const inGit = await isGitRepo(workingDir)
if (!inGit) {
return {
exitCode: USAGE_ERROR,
message: '--changed requires a git repository. Current directory is not inside a git repo.',
}
}
appRoutes = await filterChangedRoutes(allRoutes, workingDir)
const matched = appRoutes.length
const total = allRoutes.length
if (!ctx.options.quiet && format === 'human') {
console.log(`${matched}/${total} routes match --changed filter`)
}
if (matched === 0) {
return {
exitCode: SUCCESS,
message: `No routes changed since HEAD. All ${total} routes skipped.`,
}
}
}
// Collect discovery warnings // Collect discovery warnings
const { discoverRouteDetails } = await import('../../../domain/discovery.js') const { discoverRouteDetails } = await import('../../../domain/discovery.js')
@@ -586,6 +619,8 @@ export async function qualifyCommand(
fastify: fastify as any, fastify: fastify as any,
seed, seed,
timeout: presetTimeout, timeout: presetTimeout,
pluginContractRegistry: (fastify as unknown as Record<string, unknown>)?._apophisPluginContractRegistry as
import('../../../domain/plugin-contracts.js').PluginContractRegistry | undefined,
} }
const runResult = await runQualify(deps, gates, scenarios, statefulConfig, chaosConfig, appRoutes) const runResult = await runQualify(deps, gates, scenarios, statefulConfig, chaosConfig, appRoutes)
@@ -600,6 +635,15 @@ export async function qualifyCommand(
seed, seed,
}) })
// Drain plugin contract registry warnings if app registered APOPHIS
const pcr = (fastify as any)?._apophisPluginContractRegistry
if (pcr?.drainWarnings) {
const pcrWarnings = pcr.drainWarnings()
if (pcrWarnings.length > 0) {
artifact.warnings.push(...pcrWarnings)
}
}
// 12. Signal quality guardrails — fail if zero checks executed // 12. Signal quality guardrails — fail if zero checks executed
const execSummary = runResult.executionSummary const execSummary = runResult.executionSummary
const warnings: string[] = [...artifact.warnings, ...discoveryWarnings] const warnings: string[] = [...artifact.warnings, ...discoveryWarnings]
@@ -624,7 +668,9 @@ export async function qualifyCommand(
if (gates.scenario && execSummary.scenariosRun === 0) { if (gates.scenario && execSummary.scenariosRun === 0) {
warnings.push('WARNING: scenario gate enabled but zero scenarios executed. Check route configuration.') warnings.push('WARNING: scenario gate enabled but zero scenarios executed. Check route configuration.')
} }
if (gates.stateful && execSummary.statefulTestsRun === 0) { if (gates.stateful && execSummary.statefulTestsRun === 0 && execSummary.statefulSkipped > 0) {
warnings.push(`WARNING: stateful gate enabled but all ${execSummary.statefulSkipped} commands were skipped (preconditions not met). Check required headers, scopes, or plugin contract settings.`)
} else if (gates.stateful && execSummary.statefulTestsRun === 0) {
warnings.push('WARNING: stateful gate enabled but zero stateful tests executed. Check app routes and schema.') warnings.push('WARNING: stateful gate enabled but zero stateful tests executed. Check app routes and schema.')
} }
if (gates.chaos && execSummary.chaosRunsRun === 0) { if (gates.chaos && execSummary.chaosRunsRun === 0) {
@@ -706,6 +752,7 @@ export async function handleQualify(
const options: QualifyOptions = { const options: QualifyOptions = {
profile: ctx.options.profile || undefined, profile: ctx.options.profile || undefined,
seed: undefined, seed: undefined,
changed: false,
config: ctx.options.config || undefined, config: ctx.options.config || undefined,
cwd: ctx.cwd, cwd: ctx.cwd,
format: ctx.options.format as QualifyOptions['format'], format: ctx.options.format as QualifyOptions['format'],
@@ -722,6 +769,10 @@ export async function handleQualify(
} }
} }
if (args.includes('--changed')) {
options.changed = true
}
const result = await qualifyCommand(options, ctx) const result = await qualifyCommand(options, ctx)
const format = options.format || ctx.options.format || 'human' const format = options.format || ctx.options.format || 'human'
const machineMode = format === 'json' || format === 'ndjson' || format === 'json-summary' || format === 'ndjson-summary' const machineMode = format === 'json' || format === 'ndjson' || format === 'json-summary' || format === 'ndjson-summary'
+11 -4
View File
@@ -29,6 +29,7 @@ import type {
FastifyInjectInstance, FastifyInjectInstance,
} from '../../../types.js' } from '../../../types.js'
import type { ExtensionRegistry } from '../../../extension/types.js' import type { ExtensionRegistry } from '../../../extension/types.js'
import type { PluginContractRegistry } from '../../../domain/plugin-contracts.js'
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Types // Types
@@ -60,6 +61,7 @@ export interface QualifyRunResult {
totalFailed: number totalFailed: number
scenariosRun: number scenariosRun: number
statefulTestsRun: number statefulTestsRun: number
statefulSkipped: number
chaosRunsRun: number chaosRunsRun: number
chaosRoutesPlanned: number chaosRoutesPlanned: number
chaosRoutesExecuted: number chaosRoutesExecuted: number
@@ -85,6 +87,7 @@ export interface CleanupFailure {
export interface QualifyRunnerDeps { export interface QualifyRunnerDeps {
fastify: FastifyInjectInstance fastify: FastifyInjectInstance
extensionRegistry?: ExtensionRegistry extensionRegistry?: ExtensionRegistry
pluginContractRegistry?: PluginContractRegistry
seed: number seed: number
timeout?: number timeout?: number
} }
@@ -237,17 +240,20 @@ export async function runQualify(
const statefulPassed = !statefulResult || statefulResult.summary.failed === 0 const statefulPassed = !statefulResult || statefulResult.summary.failed === 0
const chaosPassed = chaosResults.every(r => !r.applied || r.contractsPassed) const chaosPassed = chaosResults.every(r => !r.applied || r.contractsPassed)
// Count execution metrics // Count execution metrics — distinguish executed from skipped
const scenariosRun = scenarioResults.length const scenariosRun = scenarioResults.length
const statefulTestsRun = statefulResult?.tests.length ?? 0 const statefulPassedCount = statefulResult?.summary.passed ?? 0
const statefulFailedCount = statefulResult?.summary.failed ?? 0
const statefulSkippedCount = statefulResult?.summary.skipped ?? 0
const statefulTestsRun = statefulPassedCount + statefulFailedCount
const chaosRunsRun = chaosResults.length const chaosRunsRun = chaosResults.length
const totalSteps = allTraces.length const totalSteps = allTraces.length
const totalExecuted = scenariosRun + statefulTestsRun + chaosRunsRun const totalExecuted = scenariosRun + statefulTestsRun + chaosRunsRun
const totalPassed = scenarioResults.reduce((sum, r) => sum + r.summary.passed, 0) + const totalPassed = scenarioResults.reduce((sum, r) => sum + r.summary.passed, 0) +
(statefulResult?.summary.passed ?? 0) + statefulPassedCount +
chaosResults.reduce((sum, r) => sum + (r.applied && r.contractsPassed ? 1 : 0), 0) chaosResults.reduce((sum, r) => sum + (r.applied && r.contractsPassed ? 1 : 0), 0)
const totalFailed = scenarioResults.reduce((sum, r) => sum + r.summary.failed, 0) + const totalFailed = scenarioResults.reduce((sum, r) => sum + r.summary.failed, 0) +
(statefulResult?.summary.failed ?? 0) statefulFailedCount
// Track executed and skipped routes for transparency // Track executed and skipped routes for transparency
const executedRoutes: string[] = [] const executedRoutes: string[] = []
@@ -316,6 +322,7 @@ export async function runQualify(
totalFailed, totalFailed,
scenariosRun, scenariosRun,
statefulTestsRun, statefulTestsRun,
statefulSkipped: statefulSkippedCount,
chaosRunsRun, chaosRunsRun,
chaosRoutesPlanned, chaosRoutesPlanned,
chaosRoutesExecuted, chaosRoutesExecuted,
+1 -1
View File
@@ -27,7 +27,7 @@ export async function runScenarioWithTraces(
): Promise<{ result: ScenarioResult; traces: StepTrace[] }> { ): Promise<{ result: ScenarioResult; traces: StepTrace[] }> {
const scopeHeaders: Record<string, string> = {} const scopeHeaders: Record<string, string> = {}
const result = await runScenario(deps.fastify, config, scopeHeaders, deps.extensionRegistry) const result = await runScenario(deps.fastify, config, scopeHeaders, deps.extensionRegistry, deps.pluginContractRegistry)
const traces: StepTrace[] = result.steps.map((step, idx) => { const traces: StepTrace[] = result.steps.map((step, idx) => {
const trace: StepTrace = { const trace: StepTrace = {
+1 -1
View File
@@ -45,7 +45,7 @@ export async function runStatefulWithTraces(
cleanupManager, cleanupManager,
minimalScopeRegistry, minimalScopeRegistry,
deps.extensionRegistry, deps.extensionRegistry,
undefined, // pluginContractRegistry — will be passed from runner when available deps.pluginContractRegistry,
undefined, // outboundContractRegistry undefined, // outboundContractRegistry
) )
+29
View File
@@ -172,9 +172,36 @@ async function executeContractDirect(
validateRuntime: false, validateRuntime: false,
} }
// Compose plugin contracts if the app registered the APOPHIS plugin
const pluginContractRegistry = (fastify as any)?._apophisPluginContractRegistry
if (pluginContractRegistry) {
const composed = pluginContractRegistry.composeContracts(routeContract)
if (!routeContract.formulaSources) routeContract.formulaSources = {}
for (const phase of Object.values(composed.phases) as Array<{ requires: Array<{ formula: string; source: string }>; ensures: Array<{ formula: string; source: string }> }>) {
for (const req of phase.requires) {
if (!routeContract.requires.includes(req.formula)) {
routeContract.requires.push(req.formula)
routeContract.formulaSources[req.formula] = req.source as 'route' | `plugin:${string}`
}
}
for (const ens of phase.ensures) {
if (!routeContract.ensures.includes(ens.formula)) {
routeContract.ensures.push(ens.formula)
routeContract.formulaSources[ens.formula] = ens.source as 'route' | `plugin:${string}`
}
}
}
}
// Build request // Build request
const headers: Record<string, string> = {} const headers: Record<string, string> = {}
// Auto-inject headers required by plugin contract preconditions
for (const f of routeContract.requires) {
const m = f.match(/request_headers\(this\)\.([\w-]+)\s*(!=\s*null|==\s*"([^"]+)")/)
if (m) headers[m[1]!] = m[3] || 'test-value'
}
// Execute request // Execute request
try { try {
const ctx = await executeHttp(fastify, routeContract, { const ctx = await executeHttp(fastify, routeContract, {
@@ -264,10 +291,12 @@ async function executeReplay(
} }
// Try to run verify first (works if app has APOPHIS plugin) // Try to run verify first (works if app has APOPHIS plugin)
const pluginContractRegistry = (fastify as any)?._apophisPluginContractRegistry
let runResult = await runVerify({ let runResult = await runVerify({
fastify: fastify as any, fastify: fastify as any,
seed: artifact.seed || 42, seed: artifact.seed || 42,
routeFilters: [failure.route], routeFilters: [failure.route],
pluginContractRegistry,
}) })
// If no routes matched, or route found but no contracts (plugin not registered before routes), // If no routes matched, or route found but no contracts (plugin not registered before routes),
+19 -35
View File
@@ -170,6 +170,7 @@ function buildArtifact(
total: runResult.total, total: runResult.total,
passed: runResult.passedCount, passed: runResult.passedCount,
failed: runResult.failed, failed: runResult.failed,
skipped: runResult.skipped,
}, },
deterministicParams: { deterministicParams: {
seed: options.seed, seed: options.seed,
@@ -390,6 +391,7 @@ export async function verifyCommand(
runs: typeof config.presets?.[loadResult.presetName || '']?.runs === 'number' runs: typeof config.presets?.[loadResult.presetName || '']?.runs === 'number'
? (config.presets[loadResult.presetName || ''] as { runs?: number }).runs ? (config.presets[loadResult.presetName || ''] as { runs?: number }).runs
: undefined, : undefined,
pluginContractRegistry: (fastify as any)?._apophisPluginContractRegistry,
}) })
// 7. Build artifact // 7. Build artifact
@@ -403,6 +405,15 @@ export async function verifyCommand(
routeFilters, routeFilters,
}) })
// Drain plugin contract registry warnings if app registered APOPHIS
const pcr = (fastify as any)?._apophisPluginContractRegistry
if (pcr?.drainWarnings) {
const pcrWarnings = pcr.drainWarnings()
if (pcrWarnings.length > 0) {
artifact.warnings.push(...pcrWarnings)
}
}
// 8. Write artifact if configured or on failure // 8. Write artifact if configured or on failure
const shouldEmitArtifact = Boolean(artifactDir || config.artifactDir || !runResult.passed) const shouldEmitArtifact = Boolean(artifactDir || config.artifactDir || !runResult.passed)
await emitArtifact(artifact, { await emitArtifact(artifact, {
@@ -638,55 +649,28 @@ export async function handleVerify(
const format = options.format || ctx.options.format || 'human' const format = options.format || ctx.options.format || 'human'
const machineMode = format === 'json' || format === 'ndjson' || format === 'json-summary' || format === 'ndjson-summary' const machineMode = format === 'json' || format === 'ndjson' || format === 'json-summary' || format === 'ndjson-summary'
if (!ctx.options.quiet) { // Machine-readable formats always emit regardless of --quiet
if (machineMode) {
if (format === 'json') { if (format === 'json') {
if (result.artifact) { console.log(result.artifact ? renderJsonArtifact(result.artifact) : renderJson({ exitCode: result.exitCode, message: result.message, warnings: result.warnings }))
console.log(renderJsonArtifact(result.artifact))
} else {
console.log(renderJson({
exitCode: result.exitCode,
message: result.message,
warnings: result.warnings,
}))
}
} else if (format === 'json-summary') { } else if (format === 'json-summary') {
if (result.artifact) { console.log(result.artifact ? renderJsonSummaryArtifact(result.artifact) : renderJson({ exitCode: result.exitCode, message: result.message, warnings: result.warnings }))
console.log(renderJsonSummaryArtifact(result.artifact))
} else {
console.log(renderJson({
exitCode: result.exitCode,
message: result.message,
warnings: result.warnings,
}))
}
} else if (format === 'ndjson') { } else if (format === 'ndjson') {
if (result.artifact) { if (result.artifact) {
renderNdjsonArtifact(result.artifact) renderNdjsonArtifact(result.artifact)
} else { } else {
process.stdout.write(JSON.stringify({ process.stdout.write(JSON.stringify({ type: 'run.completed', command: 'verify', exitCode: result.exitCode, message: result.message, warnings: result.warnings }) + '\n')
type: 'run.completed',
command: 'verify',
exitCode: result.exitCode,
message: result.message,
warnings: result.warnings,
}) + '\n')
} }
} else if (format === 'ndjson-summary') { } else if (format === 'ndjson-summary') {
if (result.artifact) { if (result.artifact) {
renderNdjsonSummaryArtifact(result.artifact) renderNdjsonSummaryArtifact(result.artifact)
} else { } else {
process.stdout.write(JSON.stringify({ process.stdout.write(JSON.stringify({ type: 'run.completed', command: 'verify', exitCode: result.exitCode, message: result.message, warnings: result.warnings }) + '\n')
type: 'run.completed',
command: 'verify',
exitCode: result.exitCode,
message: result.message,
warnings: result.warnings,
}) + '\n')
} }
} else if (result.message) { }
} else if (!ctx.options.quiet && result.message) {
console.log(result.message) console.log(result.message)
} }
}
// Print warnings in human mode only // Print warnings in human mode only
if (!machineMode && result.warnings && result.warnings.length > 0 && !ctx.options.quiet) { if (!machineMode && result.warnings && result.warnings.length > 0 && !ctx.options.quiet) {
+126 -12
View File
@@ -26,6 +26,7 @@ import * as fc from 'fast-check'
import { resolveRuns } from '../../../types.js' import { resolveRuns } from '../../../types.js'
import type { EvalContext, RouteContract, FastifyInjectInstance } from '../../../types.js' import type { EvalContext, RouteContract, FastifyInjectInstance } from '../../../types.js'
import type { RouteResult } from '../../core/types.js' import type { RouteResult } from '../../core/types.js'
import type { PluginContractRegistry } from '../../../domain/plugin-contracts.js'
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Types // Types
@@ -46,6 +47,7 @@ export interface VerifyRunResult {
total: number total: number
passedCount: number passedCount: number
failed: number failed: number
skipped: number
failures: VerifyFailure[] failures: VerifyFailure[]
durationMs: number durationMs: number
noRoutesMatched: boolean noRoutesMatched: boolean
@@ -66,6 +68,7 @@ export interface VerifyRunnerDeps {
changed?: boolean changed?: boolean
profileRoutes?: string[] profileRoutes?: string[]
runs?: number runs?: number
pluginContractRegistry?: PluginContractRegistry
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -161,7 +164,7 @@ function filterRoutesByPatterns(routes: RouteContract[], patterns: string[]): Ro
/** /**
* Check if cwd is inside a git repository. * Check if cwd is inside a git repository.
*/ */
async function isGitRepo(cwd: string): Promise<boolean> { export async function isGitRepo(cwd: string): Promise<boolean> {
try { try {
const { execSync } = await import('node:child_process') const { execSync } = await import('node:child_process')
execSync('git rev-parse --git-dir', { cwd, encoding: 'utf-8', stdio: 'pipe' }) execSync('git rev-parse --git-dir', { cwd, encoding: 'utf-8', stdio: 'pipe' })
@@ -174,7 +177,7 @@ async function isGitRepo(cwd: string): Promise<boolean> {
/** /**
* Get git-modified files for --changed filtering. * Get git-modified files for --changed filtering.
*/ */
async function getGitChangedFiles(cwd: string): Promise<string[]> { export async function getGitChangedFiles(cwd: string): Promise<string[]> {
try { try {
const { execSync } = await import('node:child_process') const { execSync } = await import('node:child_process')
const output = execSync('git diff --name-only HEAD', { cwd, encoding: 'utf-8' }) const output = execSync('git diff --name-only HEAD', { cwd, encoding: 'utf-8' })
@@ -187,7 +190,7 @@ async function getGitChangedFiles(cwd: string): Promise<string[]> {
/** /**
* Filter routes to only those modified in git. * Filter routes to only those modified in git.
*/ */
async function filterChangedRoutes( export async function filterChangedRoutes(
routes: RouteContract[], routes: RouteContract[],
cwd: string, cwd: string,
): Promise<RouteContract[]> { ): Promise<RouteContract[]> {
@@ -481,6 +484,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
total: 0, total: 0,
passedCount: 0, passedCount: 0,
failed: 0, failed: 0,
skipped: 0,
failures: [], failures: [],
durationMs: Date.now() - started, durationMs: Date.now() - started,
noRoutesMatched: false, noRoutesMatched: false,
@@ -501,6 +505,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
total: 0, total: 0,
passedCount: 0, passedCount: 0,
failed: 0, failed: 0,
skipped: 0,
failures: [], failures: [],
durationMs: Date.now() - started, durationMs: Date.now() - started,
noRoutesMatched: true, noRoutesMatched: true,
@@ -511,6 +516,28 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
} }
} }
// 3a. Compose plugin contracts into route contracts
if (deps.pluginContractRegistry) {
for (const route of routes) {
const composed = deps.pluginContractRegistry.composeContracts(route)
if (!route.formulaSources) route.formulaSources = {}
for (const phase of Object.values(composed.phases)) {
for (const req of phase.requires) {
if (!route.requires.includes(req.formula)) {
route.requires.push(req.formula)
route.formulaSources[req.formula] = req.source
}
}
for (const ens of phase.ensures) {
if (!route.ensures.includes(ens.formula)) {
route.ensures.push(ens.formula)
route.formulaSources[ens.formula] = ens.source
}
}
}
}
}
// 3. Check for behavioral contracts // 3. Check for behavioral contracts
const routesWithContracts = routes.filter(route => const routesWithContracts = routes.filter(route =>
route.ensures.length > 0 || route.requires.length > 0 route.ensures.length > 0 || route.requires.length > 0
@@ -522,6 +549,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
total: 0, total: 0,
passedCount: 0, passedCount: 0,
failed: 0, failed: 0,
skipped: 0,
failures: [], failures: [],
durationMs: Date.now() - started, durationMs: Date.now() - started,
noRoutesMatched: false, noRoutesMatched: false,
@@ -533,32 +561,117 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
} }
// 4. Execute contracts (with variant expansion) // 4. Execute contracts (with variant expansion)
// Execute one HTTP request per (route, variant, runIndex), then validate
// all contracts against the same request context.
// Requires (preconditions) that fail → SKIP the ensures and count as skipped.
// Ensures (postconditions) that fail → count as failures.
const runConfig = resolveRuns(deps.runs) const runConfig = resolveRuns(deps.runs)
const runs = runConfig.contractRuns const runs = runConfig.contractRuns
const failures: VerifyFailure[] = [] const failures: VerifyFailure[] = []
let total = 0 let total = 0
let passedCount = 0 let passedCount = 0
let skippedCount = 0
// Extract required headers from precondition formulas and auto-inject them
// so that plugin contracts (e.g., x-tenant-id != null) are satisfied.
function extractRequiredHeaders(formulas: string[]): Record<string, string> {
const headers: Record<string, string> = {}
for (const f of formulas) {
const nullCheck = f.match(/request_headers\(this\)\.([\w-]+)\s*!=\s*null/i)
if (nullCheck) { headers[nullCheck[1]!] = 'test-value'; continue }
const eqCheck = f.match(/request_headers\(this\)\.([\w-]+)\s*==\s*"([^"]+)"/i)
if (eqCheck) { headers[eqCheck[1]!] = eqCheck[2]!; continue }
}
return headers
}
for (const route of routesWithContracts) { for (const route of routesWithContracts) {
const contracts = [...route.requires, ...route.ensures] const defaultHeaders = extractRequiredHeaders(route.requires)
const variants = route.variants && route.variants.length > 0 const variants = route.variants && route.variants.length > 0
? route.variants ? route.variants
: [{ name: 'default' }] : [{ name: 'default' }]
for (const variant of variants) { for (const variant of variants) {
for (const contract of contracts) {
for (let runIndex = 0; runIndex < runs; runIndex++) { for (let runIndex = 0; runIndex < runs; runIndex++) {
total++ // Execute one HTTP request per run
const result = await executeContract( const request = buildRouteRequest(route, deps.seed, runIndex, variant)
fastify, route, contract, deps.timeout, const headers = {
variant, deps.seed, runIndex, ...defaultHeaders,
) ...request.headers,
...(variant.headers ?? {}),
}
const ctx = await executeHttp(fastify, route, {
method: request.method,
url: request.url,
body: request.body,
headers,
query: {},
}, undefined, deps.timeout)
const evalCtx: EvalContext = {
...ctx,
operationResolver: createOperationResolver(fastify, headers, ctx),
}
if (result.failure) { // Evaluate requires first. If any fail, skip ensures for this run.
failures.push(result.failure) let requiresPassed = true
for (const formula of route.requires) {
total++
try {
const parsed = parse(formula)
const result = await evaluateAsync(parsed.ast, evalCtx)
if (!result.success || !result.value) {
requiresPassed = false
skippedCount++
} else { } else {
passedCount++ passedCount++
} }
} catch {
requiresPassed = false
skippedCount++
}
}
// Only evaluate ensures if all requires passed
if (requiresPassed) {
for (const formula of route.ensures) {
total++
try {
const parsed = parse(formula)
const result = await evaluateAsync(parsed.ast, evalCtx)
if (!result.success || !result.value) {
const diagnostic = buildFailureDiagnostic(formula, evalCtx, result.success ? undefined : result.error)
failures.push({
route: variant.name && variant.name !== 'default'
? `[variant:${variant.name}] ${route.method} ${route.path}`
: `${route.method} ${route.path}`,
contract: formula,
expected: diagnostic.expected,
observed: diagnostic.observed,
formula: diagnostic.formula,
category: diagnostic.category,
})
} else {
passedCount++
}
} catch (error) {
const diagnostic = buildFailureDiagnostic(formula, evalCtx, error instanceof Error ? error.message : String(error))
failures.push({
route: variant.name && variant.name !== 'default'
? `[variant:${variant.name}] ${route.method} ${route.path}`
: `${route.method} ${route.path}`,
contract: formula,
expected: diagnostic.expected,
observed: diagnostic.observed,
formula: diagnostic.formula,
category: diagnostic.category,
})
}
}
} else {
// Preconditions failed: ensures are skipped but not evaluated.
// We still count each ensure as "total" and "skipped" for transparency.
total += route.ensures.length
skippedCount += route.ensures.length
} }
} }
} }
@@ -578,6 +691,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
total, total,
passedCount, passedCount,
failed: failures.length, failed: failures.length,
skipped: skippedCount,
failures: sortedFailures, failures: sortedFailures,
durationMs, durationMs,
noRoutesMatched: false, noRoutesMatched: false,
+1 -1
View File
@@ -140,7 +140,7 @@ const CONFIG_SCHEMA: Record<string, SchemaField> = {
optional: true, optional: true,
items: { type: 'object' }, items: { type: 'object' },
}, },
chaos: { pluginContracts: {
type: 'object', type: 'object',
optional: true, optional: true,
properties: {}, properties: {},
+1 -1
View File
@@ -93,7 +93,7 @@ export function createContext(options: Record<string, unknown> = {}): CliContext
const packageManager = detectPackageManager(cwd); const packageManager = detectPackageManager(cwd);
// Normalize options // Normalize options
const format = options.format === 'json' || options.format === 'ndjson' const format = options.format === 'json' || options.format === 'ndjson' || options.format === 'json-summary' || options.format === 'ndjson-summary'
? options.format ? options.format
: 'human'; : 'human';
+7 -4
View File
@@ -102,9 +102,11 @@ function getCommandHelp(command: string): string {
${pc.dim('Options:')} ${pc.dim('Options:')}
--profile <name> Profile name from config --profile <name> Profile name from config
--seed <number> Deterministic seed --seed <number> Deterministic seed
--changed Filter to git-modified routes
${pc.dim('Examples:')} ${pc.dim('Examples:')}
apophis qualify --profile oauth-nightly --seed 42 apophis qualify --profile oauth-nightly --seed 42
apophis qualify --changed
`, `,
replay: ` replay: `
${pc.bold('apophis replay')} Replay a failure using seed and stored trace ${pc.bold('apophis replay')} Replay a failure using seed and stored trace
@@ -167,18 +169,18 @@ function printInternalError(error: unknown): void {
console.error(); console.error();
} }
function resolveRequestedFormat(argv: string[]): 'human' | 'json' | 'ndjson' { function resolveRequestedFormat(argv: string[]): 'human' | 'json' | 'ndjson' | 'json-summary' | 'ndjson-summary' {
for (let i = 0; i < argv.length; i++) { for (let i = 0; i < argv.length; i++) {
const arg = argv[i]; const arg = argv[i];
if (!arg) continue; if (!arg) continue;
if (arg === '--format' && argv[i + 1]) { if (arg === '--format' && argv[i + 1]) {
const value = argv[i + 1]; const value = argv[i + 1];
if (value === 'json' || value === 'ndjson') return value; if (value === 'json' || value === 'ndjson' || value === 'json-summary' || value === 'ndjson-summary') return value as 'json' | 'ndjson' | 'json-summary' | 'ndjson-summary';
return 'human'; return 'human';
} }
if (arg.startsWith('--format=')) { if (arg.startsWith('--format=')) {
const value = arg.slice('--format='.length); const value = arg.slice('--format='.length);
if (value === 'json' || value === 'ndjson') return value; if (value === 'json' || value === 'ndjson' || value === 'json-summary' || value === 'ndjson-summary') return value as 'json' | 'ndjson' | 'json-summary' | 'ndjson-summary';
return 'human'; return 'human';
} }
} }
@@ -277,6 +279,7 @@ export async function main(argv: string[] = process.argv.slice(2)): Promise<numb
case 'qualify': case 'qualify':
cmd.option('--profile <name>', 'Profile name from config'); cmd.option('--profile <name>', 'Profile name from config');
cmd.option('--seed <number>', 'Deterministic seed'); cmd.option('--seed <number>', 'Deterministic seed');
cmd.option('--changed', 'Filter to git-modified routes');
break; break;
case 'replay': case 'replay':
cmd.option('--artifact <path>', 'Path to failure artifact'); cmd.option('--artifact <path>', 'Path to failure artifact');
@@ -375,7 +378,7 @@ export async function main(argv: string[] = process.argv.slice(2)): Promise<numb
init: new Set(['--preset', '-p', '--force', '-f', '--noninteractive']), init: new Set(['--preset', '-p', '--force', '-f', '--noninteractive']),
verify: new Set(['--profile', '--routes', '--seed', '--changed', '--workspace']), verify: new Set(['--profile', '--routes', '--seed', '--changed', '--workspace']),
observe: new Set(['--profile', '--check-config', '--workspace']), observe: new Set(['--profile', '--check-config', '--workspace']),
qualify: new Set(['--profile', '--seed', '--workspace']), qualify: new Set(['--profile', '--seed', '--workspace', '--changed']),
replay: new Set(['--artifact', '--route']), replay: new Set(['--artifact', '--route']),
doctor: new Set(['--mode', '--strict', '--workspace']), doctor: new Set(['--mode', '--strict', '--workspace']),
migrate: new Set(['--check', '--dry-run', '--write']), migrate: new Set(['--check', '--dry-run', '--write']),
+1
View File
@@ -297,6 +297,7 @@ export interface Artifact {
total: number; total: number;
passed: number; passed: number;
failed: number; failed: number;
skipped?: number;
}; };
executionSummary?: ExecutionSummary; executionSummary?: ExecutionSummary;
coverageBreakdown?: { coverageBreakdown?: {
+20
View File
@@ -202,6 +202,10 @@ function renderSummary(
lines.push(` Failed: ${summary.failed}`); lines.push(` Failed: ${summary.failed}`);
} }
if (summary.skipped !== undefined && summary.skipped > 0) {
lines.push(` ${c.yellow('Skipped:')} ${summary.skipped} (preconditions not met)`);
}
lines.push(` Duration: ${formatDuration(artifact.durationMs)}`); lines.push(` Duration: ${formatDuration(artifact.durationMs)}`);
if (artifact.seed !== undefined) { if (artifact.seed !== undefined) {
@@ -404,15 +408,31 @@ export function renderHumanArtifact(
lines.push(c.bold('Next steps')); lines.push(c.bold('Next steps'));
if (artifact.command === 'verify') { if (artifact.command === 'verify') {
if (artifact.summary.failed === 0) { if (artifact.summary.failed === 0) {
if (artifact.summary.skipped && artifact.summary.skipped > 0) {
lines.push(` ${c.yellow('!')} No contract violations, but ${artifact.summary.skipped} precondition(s) were unmet and skipped.`);
lines.push(` ${c.dim('→ Preconditions (x-requires + plugin contract requires) must be satisfied for tests to run.')}`);
lines.push(` ${c.dim('→ Send required headers via --variant or scope configuration.')}`);
lines.push(` ${c.dim('→ Add more behavioral contracts with')} x-ensures ${c.dim('and')} x-requires ${c.dim('to cover more routes.')}`);
} else {
lines.push(` ${c.green('✓')} All contracts passed.`); lines.push(` ${c.green('✓')} All contracts passed.`);
lines.push(` ${c.dim('→ Add more behavioral contracts with')} x-ensures ${c.dim('and')} x-requires ${c.dim('to cover more routes.')}`); lines.push(` ${c.dim('→ Add more behavioral contracts with')} x-ensures ${c.dim('and')} x-requires ${c.dim('to cover more routes.')}`);
lines.push(` ${c.dim('→ Run')} apophis observe ${c.dim('to enable runtime contract monitoring in production.')}`); lines.push(` ${c.dim('→ Run')} apophis observe ${c.dim('to enable runtime contract monitoring in production.')}`);
lines.push(` ${c.dim('→ Run')} apophis qualify --profile standard ${c.dim('for stateful/chaos testing.')}`); lines.push(` ${c.dim('→ Run')} apophis qualify --profile standard ${c.dim('for stateful/chaos testing.')}`);
}
} else { } else {
lines.push(` ${c.yellow('!')} Fix failing contracts and rerun with:`); lines.push(` ${c.yellow('!')} Fix failing contracts and rerun with:`);
lines.push(` ${c.dim('→')} apophis verify --seed ${artifact.seed} ${artifact.profile ? `--profile ${artifact.profile}` : ''}`); lines.push(` ${c.dim('→')} apophis verify --seed ${artifact.seed} ${artifact.profile ? `--profile ${artifact.profile}` : ''}`);
lines.push(` ${c.dim('→ Or replay the artifact:')} apophis replay --artifact <path>`); lines.push(` ${c.dim('→ Or replay the artifact:')} apophis replay --artifact <path>`);
} }
} else if (artifact.command === 'qualify') {
if (artifact.summary.skipped && artifact.summary.skipped > 0) {
lines.push(` ${c.yellow('!')} ${artifact.summary.skipped} stateful command(s) skipped — preconditions not met.`);
lines.push(` ${c.dim('→ Check plugin contracts and route headers. Qualify needs preconditions satisfied.')}`);
}
if (artifact.summary.failed === 0 && (!artifact.summary.skipped || artifact.summary.skipped === 0)) {
lines.push(` ${c.green('✓')} All checks passed.`);
}
lines.push(` ${c.dim('→ Review')} reports/apophis/ ${c.dim('for detailed artifact output.')}`);
} }
return lines.join('\n'); return lines.join('\n');
+4
View File
@@ -179,11 +179,13 @@ const makeConditionFailure = (
const expected = fieldExpectation?.expected ?? statusExpectation.expected const expected = fieldExpectation?.expected ?? statusExpectation.expected
const actual = fieldExpectation?.actual ?? statusExpectation.actual const actual = fieldExpectation?.actual ?? statusExpectation.actual
const diff = buildDiff(formula, ctx.response.body) const diff = buildDiff(formula, ctx.response.body)
const source = (route as RouteContract | undefined)?.formulaSources?.[formula]
return makeViolation({ return makeViolation({
route: route ?? { method: '', path: '' }, route: route ?? { method: '', path: '' },
formula, formula,
kind, kind,
...(source ? { source, phase: 'postcondition' } : {}),
request: { request: {
body: ctx.request.body, body: ctx.request.body,
headers: ctx.request.headers, headers: ctx.request.headers,
@@ -206,10 +208,12 @@ const makeFormulaError = (
route: RouteContract | { method: string; path: string } | undefined, route: RouteContract | { method: string; path: string } | undefined,
message: string message: string
): ContractViolation => { ): ContractViolation => {
const source = (route as RouteContract | undefined)?.formulaSources?.[formula]
return makeViolation({ return makeViolation({
route: route ?? { method: '', path: '' }, route: route ?? { method: '', path: '' },
formula, formula,
kind, kind,
...(source ? { source, phase: 'postcondition' } : {}),
request: { request: {
body: ctx.request.body, body: ctx.request.body,
headers: ctx.request.headers, headers: ctx.request.headers,
+12 -1
View File
@@ -39,6 +39,7 @@ function matchPattern(pattern: string, path: string): boolean {
export class PluginContractRegistry { export class PluginContractRegistry {
private contracts = new Map<string, PluginContractSpec>() private contracts = new Map<string, PluginContractSpec>()
private availableExtensions = new Set<string>() private availableExtensions = new Set<string>()
private _warnings: string[] = []
/** /**
* Register a plugin's contract specification. * Register a plugin's contract specification.
* Idempotent: registering the same plugin twice updates the spec. * Idempotent: registering the same plugin twice updates the spec.
@@ -75,7 +76,7 @@ export class PluginContractRegistry {
if (!matchPattern(spec.appliesTo, route.path)) continue if (!matchPattern(spec.appliesTo, route.path)) continue
const extCheck = this.checkExtensions(spec) const extCheck = this.checkExtensions(spec)
if (!extCheck.available) { if (!extCheck.available) {
console.warn( this._warnings.push(
`Plugin '${plugin}' requires extensions [${extCheck.missing.join(', ')}] which are not registered. Skipping its contracts.` `Plugin '${plugin}' requires extensions [${extCheck.missing.join(', ')}] which are not registered. Skipping its contracts.`
) )
continue continue
@@ -141,6 +142,16 @@ export class PluginContractRegistry {
getAvailableExtensions(): string[] { getAvailableExtensions(): string[] {
return Array.from(this.availableExtensions) return Array.from(this.availableExtensions)
} }
/**
* Collect and clear warnings (e.g., missing extension dependencies).
* Callers should surface these via their own output channels instead of
* letting console.warn bleed into CLI formatted output.
*/
drainWarnings(): string[] {
const w = this._warnings
this._warnings = []
return w
}
} }
// ============================================================================ // ============================================================================
// Factory // Factory
+15 -2
View File
@@ -1,11 +1,17 @@
import type { FastifyInstance, FastifyServerOptions } from 'fastify' import type { FastifyInstance, FastifyServerOptions } from 'fastify'
import Fastify from 'fastify' import Fastify from 'fastify'
import { installRouteDiscovery } from './infrastructure/discovery-hook.js' import { installRouteDiscovery } from './infrastructure/discovery-hook.js'
import type { ObserveSink } from './types.js'
export interface CreateFastifyOptions extends FastifyServerOptions { export interface CreateFastifyOptions extends FastifyServerOptions {
apophis?: { apophis?: {
runtime?: 'off' | 'warn' | 'error' runtime?: 'off' | 'warn' | 'error'
discoveryOnly?: boolean discoveryOnly?: boolean
observe?: {
enabled?: boolean
sampling?: number
sinks?: ObserveSink[]
}
} }
} }
@@ -16,9 +22,16 @@ export async function createFastify(
const fastify = Fastify(fastifyOpts) const fastify = Fastify(fastifyOpts)
installRouteDiscovery(fastify as unknown as { addHook: Function }) installRouteDiscovery(fastify as unknown as { addHook: Function })
if (apophisOpts?.discoveryOnly !== true && apophisOpts?.runtime && apophisOpts.runtime !== 'off') { const shouldRegisterApophis = apophisOpts?.discoveryOnly !== true && (
(apophisOpts?.runtime !== undefined && apophisOpts.runtime !== 'off') ||
apophisOpts?.observe?.enabled === true
)
if (shouldRegisterApophis) {
const { apophisPlugin } = await import('./plugin/index.js') const { apophisPlugin } = await import('./plugin/index.js')
await fastify.register(apophisPlugin, { runtime: apophisOpts.runtime }) await fastify.register(apophisPlugin, {
runtime: apophisOpts.runtime,
observe: apophisOpts.observe,
})
} }
return fastify return fastify
+31 -4
View File
@@ -727,6 +727,8 @@ function parseVariable(input: string, pos: number): { node: FormulaNode; pos: nu
} }
} }
const name = input.substring(p, i) const name = input.substring(p, i)
// F/T shorthands: treat as boolean literals, not variables.
// Only apply when there's no accessor (e.g., F.x is a variable, not literal).
let accessor: string[] | undefined let accessor: string[] | undefined
const dotPos = skipWs(input, i) const dotPos = skipWs(input, i)
if (input[dotPos] === '.') { if (input[dotPos] === '.') {
@@ -734,6 +736,12 @@ function parseVariable(input: string, pos: number): { node: FormulaNode; pos: nu
accessor = [acc.id] accessor = [acc.id]
i = acc.pos i = acc.pos
} }
if ((name === 'F' || name === 'T') && accessor === undefined) {
return {
node: { type: 'literal', value: name === 'T' },
pos: i
}
}
return { return {
node: { type: 'variable', name, accessor }, node: { type: 'variable', name, accessor },
@@ -842,12 +850,31 @@ function isWordChar(c: number): boolean {
function parseClause(input: string, pos: number, extensionHeaders: string[] = []): { node: FormulaNode; pos: number } { function parseClause(input: string, pos: number, extensionHeaders: string[] = []): { node: FormulaNode; pos: number } {
const p = skipWs(input, pos) const p = skipWs(input, pos)
const len = input.length const len = input.length
if (p < len && input[p] === 'T' && (p + 1 >= len || !isWordChar(input.charCodeAt(p + 1)))) {
return { node: { type: 'literal', value: true }, pos: p + 1 } // T/F shorthand: only consume as standalone literal if NOT followed by a
// comparison operator (==, !=, <, >, <=, >=, matches).
function isTfLiteral(): boolean {
if (p >= len) return false
const ch = input[p]!
if (ch !== 'T' && ch !== 'F') return false
const afterP = p + 1
if (afterP >= len || !isWordChar(input.charCodeAt(afterP))) {
const next = skipWs(input, afterP)
const c0 = next < len ? input.charCodeAt(next) : -1
const c1 = next + 1 < len ? input.charCodeAt(next + 1) : -1
// If followed by a comparator, T/F is a term, not a standalone literal
if (c0 === 61 || c0 === 33 || c0 === 60 || c0 === 62) return false // ==, !=, <=, <, >=, >
if (c0 === 109) return false // matches
return true
} }
if (p < len && input[p] === 'F' && (p + 1 >= len || !isWordChar(input.charCodeAt(p + 1)))) { return false
return { node: { type: 'literal', value: false }, pos: p + 1 }
} }
if (isTfLiteral()) {
const ch = input[p]!
return { node: { type: 'literal', value: ch === 'T' }, pos: p + 1 }
}
const cmp = parseComparison(input, pos, extensionHeaders) const cmp = parseComparison(input, pos, extensionHeaders)
if (cmp) return cmp if (cmp) return cmp
// If not a comparison, it must be a term (operation or literal) // If not a comparison, it must be a term (operation or literal)
+3 -2
View File
@@ -67,12 +67,13 @@ export const buildCleanup = (cleanupManager: CleanupManager) =>
export const buildScenario = ( export const buildScenario = (
fastify: FastifyInstance, fastify: FastifyInstance,
scope: ScopeRegistry, scope: ScopeRegistry,
extensionRegistry: ExtensionRegistry extensionRegistry: ExtensionRegistry,
pluginContractRegistry?: import('../domain/plugin-contracts.js').PluginContractRegistry
) => async (opts: ScenarioConfig): Promise<ScenarioResult> => { ) => async (opts: ScenarioConfig): Promise<ScenarioResult> => {
assertNonProduction('scenario') assertNonProduction('scenario')
const scopeHeaders = scope.getHeaders(opts.scope ?? null) const scopeHeaders = scope.getHeaders(opts.scope ?? null)
const injectInstance = fastify as unknown as import('../types.js').FastifyInjectInstance const injectInstance = fastify as unknown as import('../types.js').FastifyInjectInstance
return runScenario(injectInstance, opts, scopeHeaders, extensionRegistry) return runScenario(injectInstance, opts, scopeHeaders, extensionRegistry, pluginContractRegistry)
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
+15 -7
View File
@@ -96,7 +96,7 @@ export const apophisPlugin = async (fastify: FastifyInstance, opts: ApophisOptio
contract: buildContract(fastify, scope, extensionRegistry, pluginContractRegistry, outboundContractRegistry), contract: buildContract(fastify, scope, extensionRegistry, pluginContractRegistry, outboundContractRegistry),
stateful: buildStateful(fastify, scope, cleanupManager, extensionRegistry, pluginContractRegistry, outboundContractRegistry), stateful: buildStateful(fastify, scope, cleanupManager, extensionRegistry, pluginContractRegistry, outboundContractRegistry),
check: buildCheck(fastify, scope, extensionRegistry, pluginContractRegistry), check: buildCheck(fastify, scope, extensionRegistry, pluginContractRegistry),
scenario: buildScenario(fastify, scope, extensionRegistry), scenario: buildScenario(fastify, scope, extensionRegistry, pluginContractRegistry),
cleanup: buildCleanup(cleanupManager), cleanup: buildCleanup(cleanupManager),
spec: buildSpec(fastify), spec: buildSpec(fastify),
test: { test: {
@@ -147,17 +147,25 @@ export const apophisPlugin = async (fastify: FastifyInstance, opts: ApophisOptio
}, },
} }
fastify.decorate('apophis', decorations) fastify.decorate('apophis', decorations)
// Runtime validation: never register hooks in production // Expose plugin contract registry for CLI qualify access
;(fastify as unknown as Record<string, unknown>)._apophisPluginContractRegistry = pluginContractRegistry
// Runtime validation may block requests, so it is disabled in production.
// Observe sinks are explicitly non-blocking and are allowed in production.
const isProd = process.env.NODE_ENV === 'production' || process.env.NODE_ENV === 'prod' const isProd = process.env.NODE_ENV === 'production' || process.env.NODE_ENV === 'prod'
if (opts.runtime && opts.runtime !== 'off' && !isProd) { const observeSinks = opts.observe?.enabled && opts.observe.sinks && opts.observe.sinks.length > 0
? opts.observe.sinks
: undefined
const enableBlockingRuntime = opts.runtime && opts.runtime !== 'off' && !isProd
if (enableBlockingRuntime || observeSinks) {
const observeSampling = opts.observe?.sampling ?? 1
const hookOpts: { validateRuntime: boolean; runtimeLevel?: 'warn' | 'error'; observers?: { sinks: import('../types.js').ObserveSink[]; sampling: number } } = { const hookOpts: { validateRuntime: boolean; runtimeLevel?: 'warn' | 'error'; observers?: { sinks: import('../types.js').ObserveSink[]; sampling: number } } = {
validateRuntime: true, validateRuntime: true,
runtimeLevel: opts.runtime, runtimeLevel: enableBlockingRuntime ? opts.runtime : 'warn',
} }
if (opts.observe?.enabled && opts.observe.sinks && opts.observe.sinks.length > 0) { if (observeSinks) {
hookOpts.observers = { hookOpts.observers = {
sinks: opts.observe.sinks, sinks: observeSinks,
sampling: opts.observe.sampling ?? 1, sampling: observeSampling,
} }
} }
registerValidationHooks(fastify, hookOpts as Parameters<typeof registerValidationHooks>[1]) registerValidationHooks(fastify, hookOpts as Parameters<typeof registerValidationHooks>[1])
+13 -1
View File
@@ -118,15 +118,18 @@ export const runPetitTests = async (
if (pluginContractRegistry) { if (pluginContractRegistry) {
for (const route of routes) { for (const route of routes) {
const composed = pluginContractRegistry.composeContracts(route) const composed = pluginContractRegistry.composeContracts(route)
if (!route.formulaSources) route.formulaSources = {}
for (const phase of Object.values(composed.phases)) { for (const phase of Object.values(composed.phases)) {
for (const req of phase.requires) { for (const req of phase.requires) {
if (!route.requires.includes(req.formula)) { if (!route.requires.includes(req.formula)) {
route.requires.push(req.formula) route.requires.push(req.formula)
route.formulaSources[req.formula] = req.source
} }
} }
for (const ens of phase.ensures) { for (const ens of phase.ensures) {
if (!route.ensures.includes(ens.formula)) { if (!route.ensures.includes(ens.formula)) {
route.ensures.push(ens.formula) route.ensures.push(ens.formula)
route.formulaSources[ens.formula] = ens.source
} }
} }
} }
@@ -276,7 +279,16 @@ export const runPetitTests = async (
flushCache() flushCache()
const deduped = deduplicateTestFailures(results).results const deduped = deduplicateTestFailures(results).results
const suite: TestSuite = buildPetitSuite(allRoutes, routes, skippedRoutes, deduped, cacheHits, cacheMisses, startTime) const { formatSuite } = await import('./formatters.js')
const suite: TestSuite = formatSuite(deduped, {
startTime,
cacheHits,
cacheMisses,
allRoutes,
testedRoutes: routes,
skippedRoutes,
pluginContractRegistry,
})
if (suiteMockRuntime) suiteMockRuntime.restore() if (suiteMockRuntime) suiteMockRuntime.restore()
if (extensionRegistry) await extensionRegistry.runSuiteEndHooks(suite) if (extensionRegistry) await extensionRegistry.runSuiteEndHooks(suite)
+36 -2
View File
@@ -5,6 +5,7 @@ import { executeHttp } from '../infrastructure/http-executor.js'
import { validatePostconditionsAsync } from '../domain/contract-validation.js' import { validatePostconditionsAsync } from '../domain/contract-validation.js'
import { CONTENT_TYPE } from '../infrastructure/http-executor.js' import { CONTENT_TYPE } from '../infrastructure/http-executor.js'
import type { ExtensionRegistry } from '../extension/types.js' import type { ExtensionRegistry } from '../extension/types.js'
import type { PluginContractRegistry } from '../domain/plugin-contracts.js'
import type { import type {
EvalContext, EvalContext,
FastifyInjectInstance, FastifyInjectInstance,
@@ -123,7 +124,8 @@ export const runScenario = async (
fastify: FastifyInjectInstance, fastify: FastifyInjectInstance,
config: ScenarioConfig, config: ScenarioConfig,
scopeHeaders: Record<string, string>, scopeHeaders: Record<string, string>,
extensionRegistry?: ExtensionRegistry extensionRegistry?: ExtensionRegistry,
pluginContractRegistry?: PluginContractRegistry
): Promise<ScenarioResult> => { ): Promise<ScenarioResult> => {
const started = Date.now() const started = Date.now()
const steps: ScenarioStepResult[] = [] const steps: ScenarioStepResult[] = []
@@ -162,6 +164,37 @@ export const runScenario = async (
} }
const route = buildScenarioRoute({ ...interpolated, method }) const route = buildScenarioRoute({ ...interpolated, method })
// Merge plugin contracts into the synthetic route
if (pluginContractRegistry) {
const composed = pluginContractRegistry.composeContracts(route)
if (!route.formulaSources) route.formulaSources = {}
for (const phase of Object.values(composed.phases)) {
for (const req of phase.requires) {
if (!route.requires.includes(req.formula)) {
route.requires.push(req.formula)
route.formulaSources[req.formula] = req.source
}
}
for (const ens of phase.ensures) {
if (!route.ensures.includes(ens.formula)) {
route.ensures.push(ens.formula)
route.formulaSources[ens.formula] = ens.source
}
}
}
}
// Auto-inject headers required by plugin contract preconditions
for (const f of route.requires) {
const m = f.match(/request_headers\(this\)\.([\w-]+)\s*(!=\s*null|==\s*"([^"]+)")/)
if (m) {
if (headers[m[1]!] === undefined) {
headers[m[1]!] = m[3] || 'test-value'
}
}
}
const preContext: EvalContext = { const preContext: EvalContext = {
request: { request: {
body, body,
@@ -204,7 +237,8 @@ export const runScenario = async (
operationResolver: createOperationResolver(fastify, headers, preContext), operationResolver: createOperationResolver(fastify, headers, preContext),
} }
const validation = await validatePostconditionsAsync(step.expect.slice(), evalCtx, route, extensionRegistry) const mergedEnsures = [...step.expect, ...route.ensures]
const validation = await validatePostconditionsAsync(mergedEnsures, evalCtx, route, extensionRegistry)
if (!validation.success) { if (!validation.success) {
steps.push({ steps.push({
name: step.name, name: step.name,
+11
View File
@@ -49,6 +49,17 @@ const runBuildAndPrecondition = async (
const { fastify, scopeHeaders, rng, extensionRegistry } = runtime const { fastify, scopeHeaders, rng, extensionRegistry } = runtime
let request = buildRequest(command.route, command.params, scopeHeaders, modelState, rng) let request = buildRequest(command.route, command.params, scopeHeaders, modelState, rng)
// Auto-inject headers required by plugin contract preconditions
if (command.route.requires.length > 0) {
const merged = { ...request.headers }
for (const f of command.route.requires) {
const m = f.match(/request_headers\(this\)\.([\w-]+)\s*(!=\s*null|==\s*"([^"]+)")/)
if (m) {
merged[m[1]!] = m[3] || 'test-value'
}
}
request.headers = merged
}
if (extensionRegistry) { if (extensionRegistry) {
request = await extensionRegistry.runBuildRequestHooks({ request = await extensionRegistry.runBuildRequestHooks({
route: command.route, route: command.route,
+32 -2
View File
@@ -22,7 +22,7 @@ import type { OutboundContractRegistry } from '../domain/outbound-contracts.js'
import * as fc from 'fast-check' import * as fc from 'fast-check'
import type { ModelState } from '../domain/stateful.js' import type { ModelState } from '../domain/stateful.js'
import type { CleanupManager } from '../infrastructure/cleanup-manager.js' import type { CleanupManager } from '../infrastructure/cleanup-manager.js'
import type { EvalContext, FastifyInjectInstance, RouteContract, ScopeRegistry, TestConfig, TestResult, TestSuite } from '../types.js' import type { EvalContext, FastifyInjectInstance, RouteContract, ScopeRegistry, TestConfig, TestResult, TestSuite, ContractViolation } from '../types.js'
// Pure: hash helpers for deterministic sub-seeds // Pure: hash helpers for deterministic sub-seeds
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -105,15 +105,18 @@ export const runStatefulTests = async (
if (pluginContractRegistry) { if (pluginContractRegistry) {
for (const route of routes) { for (const route of routes) {
const composed = pluginContractRegistry.composeContracts(route) const composed = pluginContractRegistry.composeContracts(route)
if (!route.formulaSources) route.formulaSources = {}
for (const phase of Object.values(composed.phases)) { for (const phase of Object.values(composed.phases)) {
for (const req of phase.requires) { for (const req of phase.requires) {
if (!route.requires.includes(req.formula)) { if (!route.requires.includes(req.formula)) {
route.requires.push(req.formula) route.requires.push(req.formula)
route.formulaSources[req.formula] = req.source
} }
} }
for (const ens of phase.ensures) { for (const ens of phase.ensures) {
if (!route.ensures.includes(ens.formula)) { if (!route.ensures.includes(ens.formula)) {
route.ensures.push(ens.formula) route.ensures.push(ens.formula)
route.formulaSources[ens.formula] = ens.source
} }
} }
} }
@@ -303,9 +306,36 @@ export const runStatefulTests = async (
const passed = dedupedResults.filter((r) => r.ok && r.directive === undefined).length const passed = dedupedResults.filter((r) => r.ok && r.directive === undefined).length
const failed = dedupedResults.filter((r) => !r.ok).length const failed = dedupedResults.filter((r) => !r.ok).length
const skipped = dedupedResults.filter((r) => r.directive !== undefined).length const skipped = dedupedResults.filter((r) => r.directive !== undefined).length
// Compute plugin contract counts
let pluginContractsApplied = 0
let pluginContractsFailed = 0
if (pluginContractRegistry) {
for (const route of routes) {
const composed = pluginContractRegistry.composeContracts(route)
for (const phase of Object.values(composed.phases)) {
pluginContractsApplied += phase.ensures.length + phase.requires.length
}
}
for (const result of dedupedResults) {
if (!result.ok && result.diagnostics?.violation) {
const violation = result.diagnostics.violation as ContractViolation
if (violation.source?.startsWith('plugin:')) pluginContractsFailed++
}
}
}
const suite: TestSuite = { const suite: TestSuite = {
tests: dedupedResults, tests: dedupedResults,
summary: { passed, failed, skipped, timeMs: Date.now() - startTime, cacheHits, cacheMisses, counterexample: counterexampleOutput }, summary: {
passed,
failed,
skipped,
timeMs: Date.now() - startTime,
cacheHits,
cacheMisses,
counterexample: counterexampleOutput,
pluginContractsApplied,
pluginContractsFailed,
},
routes: allRoutes.map(r => ({ routes: allRoutes.map(r => ({
path: r.path, path: r.path,
method: r.method, method: r.method,
@@ -1,6 +1,8 @@
import { test } from 'node:test'; import { test } from 'node:test';
import assert from 'node:assert'; import assert from 'node:assert';
import { spawnSync } from 'node:child_process'; import { spawnSync } from 'node:child_process';
import fs from 'node:fs';
import path from 'node:path';
function runCli(args: string[]): { status: number; stdout: string; stderr: string } { function runCli(args: string[]): { status: number; stdout: string; stderr: string } {
const result = spawnSync( const result = spawnSync(
@@ -84,3 +86,158 @@ test('global help in machine mode emits parseable JSON', () => {
assert.ok(typeof parsed.help === 'string'); assert.ok(typeof parsed.help === 'string');
assert.ok(parsed.help.includes('Usage:')); assert.ok(parsed.help.includes('Usage:'));
}); });
// ---------------------------------------------------------------------------
// json-summary and ndjson-summary format tests
// ---------------------------------------------------------------------------
test('verify --format json-summary emits parseable JSON with summary fields', () => {
const { status, stdout, stderr } = runCli([
'verify', '--cwd', 'src/cli/__fixtures__/tiny-fastify',
'--profile', 'quick', '--seed', '42', '--format', 'json-summary',
]);
assert.strictEqual(status, 0);
assert.strictEqual(stderr.trim(), '');
assert.ok(!stdout.includes('Seed:'), 'machine output must not include human seed prelude');
const parsed = JSON.parse(stdout);
assert.strictEqual(parsed.command, 'verify');
assert.strictEqual(typeof parsed.summary.total, 'number');
assert.strictEqual(typeof parsed.summary.passed, 'number');
assert.strictEqual(typeof parsed.summary.failed, 'number');
assert.strictEqual(typeof parsed.seed, 'number');
assert.strictEqual(typeof parsed.exitReason, 'string');
});
test('verify --format ndjson-summary emits only NDJSON records', () => {
const { status, stdout, stderr } = runCli([
'verify', '--cwd', 'src/cli/__fixtures__/tiny-fastify',
'--profile', 'quick', '--seed', '42', '--format', 'ndjson-summary',
]);
assert.strictEqual(status, 0);
assert.strictEqual(stderr.trim(), '');
assert.ok(!stdout.includes('Seed:'), 'machine output must not include human seed prelude');
const lines = stdout.split('\n').map((line) => line.trim()).filter(Boolean);
assert.ok(lines.length >= 1, 'ndjson-summary should emit at least one record');
for (const line of lines) {
const parsed = JSON.parse(line);
assert.ok(typeof parsed.type === 'string' || typeof parsed.summary === 'object' || typeof parsed.command === 'string');
}
});
test('verify --format json-summary --quiet still emits JSON', () => {
const { status, stdout, stderr } = runCli([
'verify', '--cwd', 'src/cli/__fixtures__/tiny-fastify',
'--profile', 'quick', '--seed', '42', '--format', 'json-summary', '--quiet',
]);
assert.strictEqual(status, 0);
assert.strictEqual(stderr.trim(), '');
assert.ok(stdout.trim().length > 0, '--quiet must not suppress machine-readable JSON');
const parsed = JSON.parse(stdout);
assert.strictEqual(parsed.command, 'verify');
});
// ---------------------------------------------------------------------------
// skipped field in artifact
// ---------------------------------------------------------------------------
test('verify --format json-summary includes skipped field when preconditions cannot be met', () => {
const { status, stdout, stderr } = runCli([
'verify', '--cwd', 'src/cli/__fixtures__/plugin-contracts',
'--profile', 'quick', '--seed', '42', '--format', 'json-summary',
]);
// Plugin contract requires x-custom header which the generated request won't send.
// Auto-injection should add it, so preconditions pass.
// If not, skipped > 0 and failed === 0.
const parsed = JSON.parse(stdout);
assert.strictEqual(parsed.command, 'verify');
assert.ok(typeof parsed.summary.skipped === 'number', 'skipped field must be present');
assert.strictEqual(parsed.summary.failed, 0, 'no postcondition failures expected');
});
test('verify --format json-summary exit code 0 when all contracts pass', () => {
const { status, stdout } = runCli([
'verify', '--cwd', 'src/cli/__fixtures__/tiny-fastify',
'--seed', '42', '--format', 'json-summary', '--quiet',
]);
assert.strictEqual(status, 0, 'exit code must be 0 when all contracts pass');
const parsed = JSON.parse(stdout);
assert.strictEqual(parsed.summary.failed, 0);
});
// ---------------------------------------------------------------------------
// qualify json-summary and round-trip
// ---------------------------------------------------------------------------
test('qualify --format json-summary emits parseable JSON', () => {
const { status, stdout, stderr } = runCli([
'qualify', '--cwd', 'src/cli/__fixtures__/plugin-contracts',
'--seed', '42', '--format', 'json-summary',
]);
const parsed = JSON.parse(stdout);
assert.strictEqual(parsed.command, 'qualify');
assert.strictEqual(typeof parsed.summary.total, 'number');
assert.strictEqual(typeof parsed.summary.passed, 'number');
assert.strictEqual(typeof parsed.summary.failed, 'number');
assert.ok(typeof parsed.executionSummary === 'object', 'qualify artifact must include execution summary');
});
test('verify + replay round-trip with plugin contracts', () => {
const artifactDir = '/tmp/apophis-replay-test-' + Date.now();
const { status: verifyStatus } = runCli([
'verify', '--cwd', 'src/cli/__fixtures__/plugin-contracts',
'--profile', 'quick', '--seed', '42', '--artifact-dir', artifactDir,
'--format', 'human',
]);
const files = fs.readdirSync(artifactDir);
const artifactFile = files.find((f) => f.endsWith('.json'));
assert.ok(artifactFile, 'artifact file should exist');
const artifactPath = path.join(artifactDir, artifactFile);
const { status: replayStatus, stdout: replayOut } = runCli([
'replay', '--artifact', artifactPath,
]);
fs.rmSync(artifactDir, { recursive: true, force: true });
assert.strictEqual(verifyStatus, 0, 'verify should exit 0');
// Replay exits 2 when artifact has no failures (nothing to reproduce).
// This proves the round-trip: artifact written, artifact loaded successfully.
assert.strictEqual(replayStatus, 2, 'replay exits 2 for no-failure artifact');
assert.ok(replayOut.includes('no failures to replay'), 'replay reports no failures');
});
test('replay preserves seed from artifact', () => {
const artifactDir = '/tmp/apophis-seed-test-' + Date.now();
runCli([
'verify', '--cwd', 'src/cli/__fixtures__/plugin-contracts',
'--profile', 'quick', '--seed', '99', '--artifact-dir', artifactDir,
'--format', 'human',
]);
const files = fs.readdirSync(artifactDir);
const artifactFile = files.find((f) => f.endsWith('.json'));
assert.ok(artifactFile, 'artifact file should exist');
const artifactPath = path.join(artifactDir, artifactFile);
const artifactContent = fs.readFileSync(artifactPath, 'utf8');
const artifact = JSON.parse(artifactContent);
assert.strictEqual(artifact.seed, 99, 'artifact must preserve seed 99');
const { status: replayStatus, stdout: replayOut } = runCli([
'replay', '--artifact', artifactPath,
]);
fs.rmSync(artifactDir, { recursive: true, force: true });
assert.strictEqual(replayStatus, 2, 'replay exits 2 for no-failure artifact');
assert.ok(replayOut.includes('no failures to replay'), 'replay loads artifact with seed 99');
});
+32
View File
@@ -222,6 +222,38 @@ test('parse: F shorthand for false', () => {
const result = parse('F') const result = parse('F')
assert.deepStrictEqual(result.ast, { type: 'literal', value: false }) assert.deepStrictEqual(result.ast, { type: 'literal', value: false })
}) })
test('parse: F == true parses as comparison', () => {
const result = parse('F == true')
assert.deepStrictEqual(result.ast, {
type: 'comparison', op: '==',
left: { type: 'literal', value: false },
right: { type: 'literal', value: true },
})
})
test('parse: T == false parses as comparison', () => {
const result = parse('T == false')
assert.deepStrictEqual(result.ast, {
type: 'comparison', op: '==',
left: { type: 'literal', value: true },
right: { type: 'literal', value: false },
})
})
test('parse: F != true parses as comparison', () => {
const result = parse('F != true')
assert.deepStrictEqual(result.ast, {
type: 'comparison', op: '!=',
left: { type: 'literal', value: false },
right: { type: 'literal', value: true },
})
})
test('parse: F < T parses as comparison', () => {
const result = parse('F < T')
assert.deepStrictEqual(result.ast, {
type: 'comparison', op: '<',
left: { type: 'literal', value: false },
right: { type: 'literal', value: true },
})
})
test('parse: throws on empty formula', () => { test('parse: throws on empty formula', () => {
assert.throws(() => parse(''), /Empty formula/) assert.throws(() => parse(''), /Empty formula/)
}) })
+196
View File
@@ -7,6 +7,7 @@ import assert from 'node:assert'
import Fastify from 'fastify' import Fastify from 'fastify'
import type { FastifyInstance } from 'fastify' import type { FastifyInstance } from 'fastify'
import apophisPlugin from '../index.js' import apophisPlugin from '../index.js'
import { createFastify } from '../fastify-factory.js'
import { runPetitTests } from '../quality/petit-runner.js' import { runPetitTests } from '../quality/petit-runner.js'
import { CleanupManager } from '../infrastructure/cleanup-manager.js' import { CleanupManager } from '../infrastructure/cleanup-manager.js'
import { ScopeRegistry } from '../infrastructure/scope-registry.js' import { ScopeRegistry } from '../infrastructure/scope-registry.js'
@@ -924,3 +925,198 @@ test('observe: violation event emitted but response still succeeds in observe mo
await fastify.close() await fastify.close()
} }
}) })
test('observe: production mode still emits non-blocking sink events', async () => {
const originalEnv = process.env.NODE_ENV
process.env.NODE_ENV = 'production'
const fastify = Fastify()
try {
await fastify.register(swagger, {})
const emitted: ObserveEvent[] = []
const sink: ObserveSink = { emit: (event) => { emitted.push(event) } }
await fastify.register(apophisPlugin, {
runtime: 'off',
observe: { enabled: true, sinks: [sink], sampling: 1 },
})
fastify.get('/prod-observe', {
schema: {
'x-category': 'observer',
'x-ensures': ['status:201'],
} as Record<string, unknown>
}, async () => ({ ok: true }))
await fastify.ready()
const response = await fastify.inject({ method: 'GET', url: '/prod-observe' })
assert.strictEqual(response.statusCode, 200, 'production observe must not block responses')
const violationEvents = emitted.filter(e => e.type === 'contract.violation')
assert.ok(violationEvents.length > 0, 'production observe should emit contract violations')
assert.strictEqual(violationEvents[0]!.formula, 'status:201')
} finally {
await fastify.close()
if (originalEnv === undefined) delete process.env.NODE_ENV
else process.env.NODE_ENV = originalEnv
}
})
test('createFastify: observe options activate sink-backed runtime observation', async () => {
const emitted: ObserveEvent[] = []
const sink: ObserveSink = { emit: (event) => { emitted.push(event) } }
const fastify = await createFastify({
apophis: {
observe: { enabled: true, sinks: [sink], sampling: 1 },
},
})
try {
fastify.get('/factory-observe', {
schema: {
'x-category': 'observer',
'x-ensures': ['status:200'],
} as Record<string, unknown>
}, async () => ({ ok: true }))
await fastify.ready()
const response = await fastify.inject({ method: 'GET', url: '/factory-observe' })
assert.strictEqual(response.statusCode, 200)
const passEvents = emitted.filter(e => e.type === 'contract.pass')
assert.ok(passEvents.length > 0, 'createFastify observe options should wire runtime sink events')
} finally {
await fastify.close()
}
})
test('createFastify: runtime off alone keeps APOPHIS plugin inactive', async () => {
const fastify = await createFastify({
apophis: { runtime: 'off' },
})
try {
assert.strictEqual(fastify.hasDecorator('apophis'), false)
} finally {
await fastify.close()
}
})
// ---------------------------------------------------------------------------
// Plugin composition: plugin contracts merged into route contracts
// ---------------------------------------------------------------------------
test('plugin composition: contracts merge into route and source is tracked on violation', async () => {
const fastify = Fastify() as unknown as TestFastifyInstance
try {
await fastify.register(swagger, {})
await fastify.register(apophisPlugin, {
pluginContracts: {
'auth-plugin': {
appliesTo: '/api/**',
hooks: {
preHandler: { ensures: ['1 == 2'] },
},
meta: { name: 'auth-plugin' },
},
},
})
fastify.get('/api/resource', {
schema: {
'x-category': 'observer',
response: { 200: { type: 'object' } },
} as Record<string, unknown>,
}, async () => ({ ok: true }))
await fastify.ready()
const suite = await fastify.apophis.contract({ runs: 5 })
assert.ok(suite.summary.pluginContractsApplied! > 0, 'plugin contracts should be counted as applied')
assert.ok(suite.summary.pluginContractsFailed! > 0, 'always-false ensures should cause at least one plugin contract failure')
} finally {
await fastify.close()
}
})
test('plugin composition: missing required extension skips the plugin silently', async () => {
const fastify = Fastify() as unknown as TestFastifyInstance
try {
await fastify.register(swagger, {})
await fastify.register(apophisPlugin, {
pluginContracts: {
'requires-missing-ext': {
appliesTo: '/**',
extensions: [{ name: 'nonexistent', required: true }],
hooks: {
preHandler: { ensures: ['1 == 2'] },
},
},
},
})
fastify.get('/plain', {
schema: {
'x-category': 'observer',
'x-ensures': ['status:200'],
response: { 200: { type: 'object' } },
} as Record<string, unknown>,
}, async () => ({ ok: true }))
await fastify.ready()
const suite = await fastify.apophis.contract({ runs: 5 })
assert.strictEqual(suite.summary.failed, 0, 'route contract should pass without the skipped plugin formula')
assert.strictEqual(
suite.summary.pluginContractsApplied ?? 0,
1,
'only the route-level formula counts when plugin is skipped',
)
assert.strictEqual(
suite.summary.pluginContractsFailed ?? 0,
0,
'no plugin failures when plugin is skipped',
)
} finally {
await fastify.close()
}
})
test('plugin composition: formula from plugin with extension passes when condition met', async () => {
const fastify = Fastify() as unknown as TestFastifyInstance
try {
await fastify.register(swagger, {})
await fastify.register(apophisPlugin, {
extensions: [
{
name: 'tenant-gate',
headers: ['check_tenant'],
predicates: {
check_tenant: (ctx: { evalContext: { request: { headers: Record<string, string> } } }) => ({
value: ctx.evalContext.request.headers['x-tenant'] === 'acme',
success: true,
}),
},
} as import('../extension/types.js').ApophisExtension,
],
pluginContracts: {
'tenant-validator': {
appliesTo: '/**',
extensions: [{ name: 'tenant-gate', required: true }],
hooks: {
onRequest: { requires: ['check_tenant(this) == true'] },
},
},
},
})
fastify.get('/tenanted', {
schema: {
'x-category': 'observer',
'x-ensures': ['status:200'],
response: { 200: { type: 'object' } },
} as Record<string, unknown>,
}, async () => ({ ok: true }))
await fastify.ready()
const suite = await fastify.apophis.contract({
runs: 5,
variants: [{ name: 'acme', headers: { 'x-tenant': 'acme' } }],
})
assert.ok(suite.summary.pluginContractsApplied! > 0, 'tenant validator should be applied')
assert.strictEqual(suite.summary.pluginContractsFailed ?? 0, 0, 'tenant check should pass for acme')
assert.strictEqual(suite.summary.failed, 0, 'all tests should pass')
} finally {
await fastify.close()
}
})
// ---------------------------------------------------------------------------
// Plugin composition + scenarios (integration)
// ---------------------------------------------------------------------------
// NOTE: scenario + plugin contract integration test resides in
// _smoke-scenario-plugin.ts for isolation from cross-test state.
// The programmatic path is verified by running that smoke script directly.
+2
View File
@@ -45,6 +45,8 @@ export interface RouteContract {
outbound?: readonly OutboundBinding[] outbound?: readonly OutboundBinding[]
/** Route-level variants for negotiated content-type or feature testing. Extracted from schema['x-variants']. */ /** Route-level variants for negotiated content-type or feature testing. Extracted from schema['x-variants']. */
variants?: RouteVariant[] variants?: RouteVariant[]
/** Formula → source mapping for plugin contract attribution. Populated at test time. */
formulaSources?: Record<string, 'route' | `plugin:${string}`>
} }
// ============================================================================ // ============================================================================