feat: plugin contract e2e, qualify --changed, production observe, regressions
This commit is contained in:
John Dvorak
+23
-14
@@ -8,11 +8,11 @@ This audit is based on code inspection plus command verification, not documentat
|
||||
|
||||
## Executive Summary
|
||||
|
||||
APOPHIS has real product value. It is not just a schema wrapper: it gives Fastify teams a way to express and verify behavioral API promises that OpenAPI/JSON Schema cannot cover, especially cross-route invariants such as create/read consistency, delete semantics, auth/session flows, state transitions, idempotency, outbound dependency expectations, and replayable counterexamples.
|
||||
APOPHIS has real product value. It is not just a schema wrapper: it gives Fastify teams a way to express, verify, and observe behavioral API promises that OpenAPI/JSON Schema cannot cover, especially cross-route invariants such as create/read consistency, delete semantics, auth/session flows, state transitions, idempotency, outbound dependency expectations, and replayable counterexamples.
|
||||
|
||||
I would adopt APOPHIS today as a focused behavioral verification tool for Fastify v5 ESM services. I would start with CI `verify` and a small number of high-value contracts, then expand into `qualify` and runtime observation once the team has clear operating guidance.
|
||||
|
||||
I would not yet treat it as a complete production observability platform or a turnkey organization-wide release gate. The core implementation is strong, but the remaining value gap is mostly around operational maturity: standalone observe activation, deeper tests around recent CLI behavior, richer scenario authoring, and clearer release-gate recommendations.
|
||||
I would not yet treat it as a complete production observability platform or a turnkey organization-wide release gate. The core implementation is strong, but the remaining value gap is mostly around operational maturity: standalone observe process management, richer scenario authoring, and organization-specific release-gate policy.
|
||||
|
||||
Adoption verdict: strong team pilot candidate, credible standardization candidate after the remaining gaps below are addressed.
|
||||
|
||||
@@ -34,10 +34,10 @@ Observed results:
|
||||
|---|---:|
|
||||
| Typecheck | pass |
|
||||
| Build | pass |
|
||||
| Source tests | 587 pass, 0 fail |
|
||||
| CLI tests | 311 pass, 0 fail |
|
||||
| Source tests | 590 pass, 0 fail |
|
||||
| CLI tests | 320 pass, 0 fail |
|
||||
| Docs smoke tests | 4 pass, 0 fail |
|
||||
| Total tests | 902 pass, 0 fail |
|
||||
| Total tests | 921 pass, 0 fail |
|
||||
|
||||
The working tree contains many broader project changes unrelated to this audit. This document evaluates the current working tree state.
|
||||
|
||||
@@ -51,7 +51,7 @@ Mostly yes for behavioral verification. Partially for production observation and
|
||||
| Deterministic CI verification | Yes, materially. CLI `verify` now honors configured `runs`, uses seeded request generation, emits artifacts, supports route filters, replay metadata, and machine-readable output. |
|
||||
| Cross-route behavior | Yes for supported formula operations and route-call semantics. This is the most differentiated value. |
|
||||
| Runtime validation | Yes when the plugin is explicitly configured outside production. Production enforcement is intentionally blocked. |
|
||||
| Runtime observation | Partially. Programmatic plugin observation exists and emits non-blocking sink events with sampling. The CLI validates/report readiness but does not attach to or run a service. |
|
||||
| Runtime observation | Yes for programmatic production-safe hooks. APOPHIS emits non-blocking sink events with sampling in production when `observe.enabled` and sinks are configured. The CLI validates/report readiness but does not attach to or run a service. |
|
||||
| Stateful/scenario/chaos qualification | Partially. The runner and artifacts are useful, route discovery is now shared with verify, and config supports scenarios/chaos knobs. Scenario authoring is still young and needs more real-world examples/tests. |
|
||||
| Outbound dependency mocking | Useful but intentionally process-global. The misleading scoped `undici-mock-agent` option has been removed. Teams still need careful test isolation. |
|
||||
| Team-safe onboarding | Good. The package has CLI help, init/doctor/replay/verify/qualify/observe, config validation, machine output, docs smoke tests, packaging tests, and production safety checks. |
|
||||
@@ -102,12 +102,17 @@ The following earlier adoption risks have been addressed in the current working
|
||||
|---|---|
|
||||
| CLI `verify` runs | `VerifyRunnerDeps` accepts `runs`; `verifyCommand()` passes resolved config; `runVerify()` executes contracts for `contractRuns`. |
|
||||
| Observe sampling | `hook-validator.ts` gates sink emission using `opts.observe.sampling` before emitting pass/violation/error events. |
|
||||
| Production observe activation | `apophisPlugin` now keeps blocking runtime validation disabled in production while allowing non-blocking observe sinks to emit pass/violation/error events. |
|
||||
| Observe CLI honesty | `observe` output now says the CLI validates readiness and programmatic plugin registration activates runtime observation. |
|
||||
| Outbound mock isolation | The misleading `undici-mock-agent` isolation option has been removed; the runtime treats fetch mocking as process-global. |
|
||||
| Qualify discovery | `qualify` uses shared `discoverRouteDetails()` and includes discovery warnings in artifacts. |
|
||||
| Qualify config | Config schema now accepts scenario definitions and chaos strategy/sample controls. |
|
||||
| Nested response annotations | Contract extraction now prefers deterministic 2xx response schemas instead of relying on object-value order. |
|
||||
| `--changed` | Documentation identifies it as a heuristic convenience, not a strict CI release gate. |
|
||||
| Plugin contracts (end-to-end) | Full pipeline: config schema, plugin registration, compose+merge in all runners, precondition→skip, auto-inject headers, source attribution (`formulaSources`), failure counting, `drainWarnings()` collection, production safety. Wired through verify, qualify (scenario/stateful/chaos), and replay. |
|
||||
| Artifact pipeline CI/CD | 6 CI-facing regression tests: json-summary parseable, ndjson-summary parseable, `--quiet` persistence, skipped field presence, exit code 0 on pass, qualify json-summary. Verify→replay round-trip test with plugin contracts. |
|
||||
| CLI output hygiene | Console.warn bleeding fixed (`drainWarnings`); `json-summary`→`human` format normalization bug fixed; `--quiet` no longer suppresses machine format output. |
|
||||
| Qualify --changed | Qualify now supports `--changed` flag with same git-diff heuristic as verify. Prints match count, exits 0 when no changed routes. |
|
||||
|
||||
## Remaining Adoption Gaps
|
||||
|
||||
@@ -121,7 +126,7 @@ The implementation supports runtime observation only when the application explic
|
||||
- Integration tests prove sink sync failures and async rejections never change route responses.
|
||||
- Integration tests prove sampling: 0 suppresses all events; sampling: 1 emits expected `contract.pass`/`contract.violation` events.
|
||||
|
||||
**Still open:** A future `apophis observe --app ./app.ts` mode that activates a running service observer.
|
||||
**Still open:** A future `apophis observe --app ./app.ts` mode that imports and starts a service for local/staging smoke observation. Production observation itself is now programmatic and active through plugin registration.
|
||||
|
||||
### P1: Recent `verify` Runs Behavior Now Has Regression Tests
|
||||
|
||||
@@ -130,7 +135,7 @@ The implementation supports runtime observation only when the application explic
|
||||
- Regression test proves `runs: 5` scales multiplicatively from `runs: 1`.
|
||||
- Regression test proves `runs: 10` is deterministic at the same seed.
|
||||
|
||||
**Still open:** Variant-aware runs test (verifying run budget is per-variant or shared).
|
||||
**Completed:** Variant-aware runs regression proves the run budget is applied per variant.
|
||||
|
||||
### P1: Qualify Product Shape Improved
|
||||
|
||||
@@ -139,7 +144,7 @@ The implementation supports runtime observation only when the application explic
|
||||
- Configured-scenario qualify test added (independent of OAuth fixture routes).
|
||||
- `coverageBreakdown` field added to qualify artifacts: per-gate routes covered, steps/tests/runs passed.
|
||||
|
||||
**Still open:** Clear guidance for nightly/staging use versus pull-request gating in qualify docs.
|
||||
**Completed:** `docs/qualify.md` now documents pull-request versus nightly/staging gate guidance.
|
||||
|
||||
### P1: Outbound Mocks Process-Global, Honestly Documented
|
||||
|
||||
@@ -210,14 +215,14 @@ High-value first contracts:
|
||||
| Fastify fit | 8/10 | Strong plugin/inject/decorator alignment; discovery order still matters. |
|
||||
| Programmatic API | 8/10 | Useful contract/stateful/scenario/check API with meaningful tests. |
|
||||
| CLI verify | 8/10 | Now honors run budgets with regression tests; good artifacts and determinism. |
|
||||
| Observe | 7/10 | Runtime sink primitives, sampling, and sink-failure-resilience exist with tests. Production-style docs added. Standalone operational story not complete. |
|
||||
| Observe | 8/10 | Production-safe non-blocking sink emission, sampling, and sink-failure-resilience exist with tests. Standalone process-management story is still future work. |
|
||||
| Qualify | 7/10 | Improved discovery/config/scenarios. Coverage breakdown in artifacts. Needs richer scenario examples and gating guidance. |
|
||||
| Outbound mocking | 7/10 | Useful and honest about process-global behavior. Docs and README explicit. True scoped mocking remains future work. |
|
||||
| Docs | 8/10 | Broad and increasingly precise. Observe and qualify docs expanded with real code examples. |
|
||||
| Packaging | 9/10 | Strong for a Node/Fastify package. |
|
||||
| Team readiness | 8/10 | Ready for pilot and selective CI use with regression-locked verification behavior. |
|
||||
|
||||
Overall: 8/10 for real team pilot use. Potential 9/10 if observe gains a clearer production story and qualify gets first-class CI workflow guidance.
|
||||
Overall: 8.5/10 for real team pilot use. Potential 9/10 if standalone observe process management and richer scenario libraries become first-class.
|
||||
|
||||
## Highest-Impact Next Work
|
||||
|
||||
@@ -226,9 +231,13 @@ Overall: 8/10 for real team pilot use. Potential 9/10 if observe gains a clearer
|
||||
3. ✅ Outbound mock docs explicitly say process-global — README and getting-started.md updated.
|
||||
4. ✅ Qualify scenario config documented with full examples in qualify.md.
|
||||
5. ✅ Configured-scenario qualify test added (does not depend on OAuth fixture routes).
|
||||
6. Add full production-style observe example with a real collector sink implementation.
|
||||
7. Improve qualify artifact coverage summaries to distinguish route-contract, scenario, stateful, and chaos coverage more clearly.
|
||||
8. Consider true scoped outbound mocking (undici dispatcher) only if concurrent in-process dependency tests become a core promise.
|
||||
6. ✅ Full production-style observe example with real collector sink implementation added to docs/observe.md.
|
||||
7. ✅ Plugin contract support end-to-end: docs, tests, all runners wired.
|
||||
8. ✅ Artifact pipeline CI/CD regression tests: json-summary, ndjson-summary, --quiet, skipped field, exit codes.
|
||||
9. ✅ Qualify --changed implemented.
|
||||
10. Add standalone observe process management (`apophis observe --app ./app.ts`) for local/staging observation.
|
||||
11. Add route ownership / file-to-route maps for precise `--changed` filtering.
|
||||
12. Consider true scoped outbound mocking (undici dispatcher) only if concurrent in-process dependency tests become a core promise.
|
||||
|
||||
## Bottom Line
|
||||
|
||||
|
||||
@@ -136,7 +136,12 @@ import { createFastify } from '@apophis/fastify'
|
||||
|
||||
const app = await createFastify({
|
||||
logger: true,
|
||||
apophis: { runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off' },
|
||||
apophis: {
|
||||
runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off',
|
||||
observe: process.env.NODE_ENV === 'production'
|
||||
? { enabled: true, sampling: 0.1, sinks: [metricsSink] } // your ObserveSink
|
||||
: undefined,
|
||||
},
|
||||
})
|
||||
// Register swagger, auth, plugins, and routes after app creation.
|
||||
```
|
||||
|
||||
+36
-15
@@ -162,30 +162,19 @@ export default {
|
||||
## Programmatic Runtime Activation
|
||||
|
||||
The CLI only validates configuration. To activate runtime observation, register
|
||||
APOPHIS with observe options in your application:
|
||||
APOPHIS with observe options in your application before routes are registered.
|
||||
Observe remains active in production because it is non-blocking; blocking
|
||||
runtime validation still stays disabled in production.
|
||||
|
||||
```typescript
|
||||
import Fastify from 'fastify'
|
||||
import apophisPlugin from '@apophis/fastify'
|
||||
import type { ObserveSink, ObserveEvent } from '@apophis/fastify'
|
||||
|
||||
const app = Fastify({ logger: true })
|
||||
|
||||
// Register APOPHIS with observe enabled.
|
||||
// This emits non-blocking contract pass/violation/error events
|
||||
// for every covered request, gated by sampling.
|
||||
await app.register(apophisPlugin, {
|
||||
runtime: 'warn',
|
||||
observe: {
|
||||
enabled: true,
|
||||
sampling: 0.1, // observe 10% of requests
|
||||
sinks: [metricsSink],
|
||||
},
|
||||
})
|
||||
|
||||
// Implement the ObserveSink interface.
|
||||
// Capture events to your preferred observability backend.
|
||||
import type { ObserveSink, ObserveEvent } from '@apophis/fastify'
|
||||
|
||||
const metricsSink: ObserveSink = {
|
||||
emit(event: ObserveEvent) {
|
||||
// Emit a counter for each contract evaluation
|
||||
@@ -205,11 +194,43 @@ const metricsSink: ObserveSink = {
|
||||
}
|
||||
},
|
||||
}
|
||||
|
||||
// Register APOPHIS with observe enabled.
|
||||
// This emits non-blocking contract pass/violation/error events
|
||||
// for every covered request, gated by sampling.
|
||||
await app.register(apophisPlugin, {
|
||||
runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off',
|
||||
observe: {
|
||||
enabled: true,
|
||||
sampling: 0.1, // observe 10% of requests
|
||||
sinks: [metricsSink],
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
For new services, `createFastify()` wires discovery and APOPHIS before your
|
||||
routes, which avoids the most common ordering mistake:
|
||||
|
||||
```typescript
|
||||
import { createFastify } from '@apophis/fastify'
|
||||
|
||||
const app = await createFastify({
|
||||
logger: true,
|
||||
apophis: {
|
||||
runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off',
|
||||
observe: {
|
||||
enabled: true,
|
||||
sampling: 0.1,
|
||||
sinks: [metricsSink],
|
||||
},
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
Key constraints:
|
||||
- Sink `emit()` can be sync or async (returns `void | Promise<void>`).
|
||||
- Sink rejections and thrown errors are silently caught — they never affect the route response or status code.
|
||||
- In production, observe hooks still run when `observe.enabled` and `observe.sinks` are configured; blocking runtime validation does not.
|
||||
- Sampling is applied per-formula evaluation via `Math.random() < sampling`.
|
||||
At `sampling: 1` every formula is emitted. At `sampling: 0` nothing is emitted.
|
||||
- Only routes with APOPHIS annotations (`x-ensures`, `x-requires`) produce events.
|
||||
|
||||
@@ -0,0 +1,431 @@
|
||||
# Plugin Contracts
|
||||
|
||||
Define behavioral rules that apply across all matching routes and are automatically injected into every matching route's contract evaluation.
|
||||
|
||||
Plugin contracts let teams codify cross-cutting expectations — "all API routes must carry an auth header," "all POST handlers must close the database connection," "all responses must include a request ID." They're defined once in config and evaluated alongside route-level contracts without per-route boilerplate.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [When to Use It](#when-to-use-it)
|
||||
- [Config Format](#config-format)
|
||||
- [PluginContractSpec Reference](#plugincontractspec-reference)
|
||||
- [Route Pattern Matching](#route-pattern-matching)
|
||||
- [Precondition → Skip Semantics](#precondition--skip-semantics)
|
||||
- [Auto-Injected Headers](#auto-injected-headers)
|
||||
- [Failure Attribution](#failure-attribution)
|
||||
- [Extension Dependencies](#extension-dependencies)
|
||||
- [Production Safety](#production-safety)
|
||||
- [Complete Example](#complete-example)
|
||||
- [Limitations](#limitations)
|
||||
|
||||
## When to Use It
|
||||
|
||||
Plugin contracts are ideal for rules that span many routes and don't belong on a single handler:
|
||||
|
||||
- Auth header presence across all API routes
|
||||
- Response metadata (request IDs, timing headers, version stamps)
|
||||
- Tenant isolation checks that apply to every route in a prefix
|
||||
- Database connection lifecycle guards on write endpoints
|
||||
- Content-type enforcement on all response paths
|
||||
|
||||
Don't use plugin contracts for route-specific behavioral rules — those belong on the route schema via `x-requires` and `x-ensures`.
|
||||
|
||||
## Config Format
|
||||
|
||||
### apophis.config.js
|
||||
|
||||
```javascript
|
||||
export default {
|
||||
pluginContracts: {
|
||||
"auth-presence": {
|
||||
appliesTo: "/api/**",
|
||||
hooks: {
|
||||
onRequest: {
|
||||
requires: ["request_headers(this).authorization != null"]
|
||||
}
|
||||
}
|
||||
},
|
||||
"rate-limit-respected": {
|
||||
appliesTo: "POST /api/**",
|
||||
hooks: {
|
||||
onResponse: {
|
||||
ensures: ["status != 429"]
|
||||
}
|
||||
},
|
||||
meta: { name: "rate-limit-respected", description: "Rate limit should not be hit under normal CI load" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Programmatic (Fastify plugin registration)
|
||||
|
||||
```javascript
|
||||
await app.register(apophisPlugin, {
|
||||
pluginContracts: {
|
||||
"auth-presence": {
|
||||
appliesTo: "/api/**",
|
||||
hooks: {
|
||||
onRequest: { requires: ["request_headers(this).authorization != null"] }
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
Contracts registered programmatically take the same path through the plugin contract registry as config-defined contracts. Both sources are merged into the composed contract at evaluation time.
|
||||
|
||||
## PluginContractSpec Reference
|
||||
|
||||
| Field | Required | Type | Description |
|
||||
|---|---|---|---|
|
||||
| `appliesTo` | yes | `string` | Route pattern (see [Route Pattern Matching](#route-pattern-matching)) |
|
||||
| `hooks` | yes | `object` | Hook-phase contracts keyed by Fastify lifecycle hook name |
|
||||
| `hooks.<phase>.requires` | no | `string[]` | APOSTL formulas that must be `true` **before** this phase executes |
|
||||
| `hooks.<phase>.ensures` | no | `string[]` | APOSTL formulas that must be `true` **after** this phase executes |
|
||||
| `meta` | no | `object` | Display metadata |
|
||||
| `meta.name` | no | `string` | Human-readable plugin name |
|
||||
| `meta.version` | no | `string` | Plugin version |
|
||||
| `meta.description` | no | `string` | Description of what the plugin enforces |
|
||||
| `extensions` | no | `array` | APOPHIS extensions this plugin depends on |
|
||||
|
||||
### Supported hook phases
|
||||
|
||||
Keys in `hooks` correspond to Fastify lifecycle hook names:
|
||||
|
||||
- `onRequest` — evaluated before the route handler runs
|
||||
- `preHandler` — evaluated before the handler, after shared hooks
|
||||
- `preSerialization` — evaluated before the response body is serialized
|
||||
- `onSend` — evaluated before the response is sent
|
||||
- `onResponse` — evaluated after the response is sent (body already consumed)
|
||||
|
||||
At least one of `requires` or `ensures` must be present per phase. Empty phases are ignored.
|
||||
|
||||
### Extension dependencies
|
||||
|
||||
```javascript
|
||||
pluginContracts: {
|
||||
"jwt-auth": {
|
||||
appliesTo: "/api/**",
|
||||
extensions: [{ name: "jwt-decoder", required: true }],
|
||||
hooks: {
|
||||
onRequest: { requires: ["decode_jwt(this).valid == true"] }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
If a required extension is missing at registration time, the plugin's contracts are skipped with a warning. Extensions are resolved from the APOPHIS extension registry at evaluation time, not at route discovery time.
|
||||
|
||||
## Route Pattern Matching
|
||||
|
||||
The `appliesTo` field controls which routes a plugin contract covers.
|
||||
|
||||
### Pattern syntax
|
||||
|
||||
| Pattern | Matches | Does NOT match |
|
||||
|---|---|---|
|
||||
| `"/api/users"` | `/api/users` (exact) | `/api/users/42`, `/api/v2/users` |
|
||||
| `"/api/**"` | `/api/users`, `/api/users/42`, `/api/users/:id/posts` | `/v2/api/users` |
|
||||
| `"/api/*"` | `/api/users`, `/api/status` | `/api/users/42`, `/api/users/:id/posts` |
|
||||
| `"**"` | Everything | — |
|
||||
| `"POST /api/**"` | `POST /api/users`, `POST /api/orders/42` | `GET /api/users`, `POST /v2/api/users` |
|
||||
|
||||
The `/**` suffix matches any path at any depth below the prefix. The `/*` suffix matches exactly one segment below the prefix.
|
||||
|
||||
### Method-prefixed patterns
|
||||
|
||||
Prepend the HTTP method to restrict a pattern to a specific verb:
|
||||
|
||||
```javascript
|
||||
{
|
||||
appliesTo: "POST /api/**",
|
||||
hooks: {
|
||||
preHandler: { requires: ["request_body(this) != null"] }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
When a method is specified, the pattern only matches routes registered with that HTTP method. The method must be a valid HTTP method in uppercase.
|
||||
|
||||
Pattern matching is evaluated against the route's path string only (the raw argument passed to `fastify.get()`, `fastify.post()`, etc.), not against the dereferenced URL with prefix applied. Fastify prefix application happens before pattern matching, so a route registered as `fastify.get("/users")` under a `prefix: "/api"` is matched against `/api/users`.
|
||||
|
||||
## Precondition → Skip Semantics
|
||||
|
||||
When a plugin contract defines `requires` formulas, the runner evaluates them before `ensures`. If any require formula fails:
|
||||
|
||||
1. Require failure is **not** counted as a contract failure
|
||||
2. All `ensures` formulas for that route+run are **skipped**
|
||||
3. The skip is counted separately from failures in the artifact summary
|
||||
|
||||
This distinguishes "the plugin's prerequisites aren't met" from "the route's behavior violated the contract."
|
||||
|
||||
### Example
|
||||
|
||||
```javascript
|
||||
pluginContracts: {
|
||||
"tenant-check": {
|
||||
appliesTo: "/api/**",
|
||||
hooks: {
|
||||
onRequest: {
|
||||
requires: ["request_headers(this).x-tenant-id != null"],
|
||||
ensures: ["status != 500"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For a route without the `x-tenant-id` header:
|
||||
- The `requires` check fails → the `ensures` (`status != 500`) is **skipped**
|
||||
- `skippedCount` increments; `failedCount` does not
|
||||
|
||||
For a route with the `x-tenant-id` header:
|
||||
- The `requires` check passes → the `ensures` is evaluated normally
|
||||
- If `status == 500`, it's counted as a plugin contract **failure**
|
||||
|
||||
## Auto-Injected Headers
|
||||
|
||||
To avoid every test being skipped when a plugin contract checks for header presence, APOPHIS automatically injects headers into test requests when the precondition pattern is simple enough.
|
||||
|
||||
### Supported patterns
|
||||
|
||||
| Formula pattern | Injected header |
|
||||
|---|---|
|
||||
| `request_headers(this).<name> != null` | `<name>: test-value` |
|
||||
| `request_headers(this).<name> == "value"` | `<name>: value` |
|
||||
|
||||
### Example
|
||||
|
||||
```javascript
|
||||
pluginContracts: {
|
||||
"auth-presence": {
|
||||
appliesTo: "/api/**",
|
||||
hooks: {
|
||||
onRequest: {
|
||||
requires: ["request_headers(this).authorization != null"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
APOPHIS detects the `!= null` pattern on `authorization` and injects `authorization: test-value` into every test request. The precondition passes automatically without user intervention.
|
||||
|
||||
Auto-injection works in all runners: verify, qualify (scenario, stateful, chaos), and replay.
|
||||
|
||||
### Unsupported patterns
|
||||
|
||||
Complex preconditions are not auto-injected. For these, the requires check will fail and the route's ensures will be skipped:
|
||||
|
||||
```javascript
|
||||
// NOT auto-injected — the route's ensures will be skipped
|
||||
requires: ["request_headers(this).x-count >= 5"]
|
||||
```
|
||||
|
||||
For these cases, provide the header explicitly via route variants:
|
||||
|
||||
```javascript
|
||||
await fastify.apophis.contract({
|
||||
runs: 5,
|
||||
variants: [{ name: "with-count", headers: { "x-count": "10" } }]
|
||||
})
|
||||
```
|
||||
|
||||
## Failure Attribution
|
||||
|
||||
When a plugin contract formula fails, the violation carries metadata that distinguishes it from route-level failures.
|
||||
|
||||
### ContractViolation fields
|
||||
|
||||
| Field | Value | Meaning |
|
||||
|---|---|---|
|
||||
| `source` | `"plugin:auth-presence"` | Which plugin added the formula |
|
||||
| `phase` | `"onRequest"` | Which hook phase the formula belongs to |
|
||||
|
||||
In human-readable output, plugin failures appear separately from route contract failures:
|
||||
|
||||
```text
|
||||
Plugin contract violation (plugin:auth-presence)
|
||||
GET /api/users
|
||||
Phase: onRequest
|
||||
|
||||
Expected
|
||||
response_headers(this).x-request-id != null
|
||||
|
||||
Observed
|
||||
response_headers(this).x-request-id was null
|
||||
```
|
||||
|
||||
The artifact summary tracks plugin contract outcomes independently:
|
||||
|
||||
```json
|
||||
{
|
||||
"summary": {
|
||||
"pluginContractsApplied": 12,
|
||||
"pluginContractsFailed": 2,
|
||||
"failed": 1,
|
||||
"passed": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `pluginContractsApplied` — total plugin contract formulas evaluated across all routes
|
||||
- `pluginContractsFailed` — plugin contract formulas that failed
|
||||
- `failed` — total failures (route + plugin)
|
||||
- `passed` — route-level contract passes (plugin-only formulas not counted here)
|
||||
|
||||
A route can pass its own contracts and still have plugin contract failures. The CLI exit code reflects the aggregate: any failure (route or plugin) produces exit code 1.
|
||||
|
||||
## Extension Dependencies
|
||||
|
||||
Plugin contracts can declare dependencies on APOPHIS extensions. Extension resolution happens at evaluation time via the extension registry.
|
||||
|
||||
### Declaring dependencies
|
||||
|
||||
```javascript
|
||||
pluginContracts: {
|
||||
"jwt-auth": {
|
||||
appliesTo: "/api/**",
|
||||
extensions: [{ name: "jwt-decoder", required: true }],
|
||||
hooks: {
|
||||
onRequest: { requires: ["decode_jwt(this).valid == true"] }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Each extension reference has:
|
||||
- `name` (required) — matches the extension's registered name
|
||||
- `required` (optional, default `true`) — if `false`, missing extension emits a warning but does not skip the plugin
|
||||
|
||||
### Missing extension behavior
|
||||
|
||||
If a required extension is not registered, the plugin's contracts are **skipped** for all matching routes. Warnings are collected in the registry and surfaced via `drainWarnings()`:
|
||||
|
||||
```text
|
||||
Plugin 'jwt-auth' requires extensions [jwt-decoder] which are not registered. Skipping its contracts.
|
||||
```
|
||||
|
||||
Warnings appear in the CLI output and the artifact's `warnings` array.
|
||||
|
||||
### Extension registration
|
||||
|
||||
Extensions are registered alongside plugin contracts:
|
||||
|
||||
```javascript
|
||||
await app.register(apophisPlugin, {
|
||||
extensions: [
|
||||
{
|
||||
name: "jwt-decoder",
|
||||
headers: ["decode_jwt"],
|
||||
predicates: {
|
||||
decode_jwt: (ctx) => ({ value: true, success: true })
|
||||
}
|
||||
}
|
||||
],
|
||||
pluginContracts: {
|
||||
"jwt-auth": {
|
||||
appliesTo: "/api/**",
|
||||
extensions: [{ name: "jwt-decoder", required: true }],
|
||||
hooks: {
|
||||
onRequest: { requires: ["decode_jwt(this).valid == true"] }
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Production Safety
|
||||
|
||||
`pluginContracts` is a **test-only** feature. If `pluginContracts` is non-empty in a production environment (`NODE_ENV=production`), APOPHIS throws at plugin registration:
|
||||
|
||||
```text
|
||||
APOPHIS: Unsafe options detected in production: pluginContracts.
|
||||
These features are test-only and must not be enabled in production.
|
||||
Remove them from ApophisOptions or set NODE_ENV=test.
|
||||
```
|
||||
|
||||
This is enforced by `validateProductionSafety()` before any contract evaluation runs. There is no escape hatch.
|
||||
|
||||
## Complete Example
|
||||
|
||||
### Config (`apophis.config.js`)
|
||||
|
||||
```javascript
|
||||
export default {
|
||||
profiles: {
|
||||
quick: { preset: "safe-ci" },
|
||||
},
|
||||
presets: {
|
||||
"safe-ci": { runs: 10 },
|
||||
},
|
||||
pluginContracts: {
|
||||
"auth-required": {
|
||||
appliesTo: "/api/**",
|
||||
hooks: {
|
||||
onRequest: {
|
||||
requires: ["request_headers(this).authorization != null"],
|
||||
},
|
||||
},
|
||||
meta: { name: "auth-required", description: "All API routes require an Authorization header" },
|
||||
},
|
||||
"request-id": {
|
||||
appliesTo: "/api/**",
|
||||
hooks: {
|
||||
onSend: {
|
||||
ensures: ["response_headers(this).x-request-id != null"],
|
||||
},
|
||||
},
|
||||
meta: { name: "request-id", description: "All API responses must include x-request-id" },
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### Route (`routes/users.js`)
|
||||
|
||||
```javascript
|
||||
app.get("/api/users", {
|
||||
schema: {
|
||||
"x-category": "observer",
|
||||
"x-ensures": ["status:200", "response_body(this) is Array"],
|
||||
response: {
|
||||
200: {
|
||||
type: "array",
|
||||
items: { type: "object", properties: { id: { type: "number" } } },
|
||||
},
|
||||
},
|
||||
},
|
||||
}, async (request, reply) => {
|
||||
return [{ id: 1 }, { id: 2 }]
|
||||
})
|
||||
```
|
||||
|
||||
### Execution (`apophis verify`)
|
||||
|
||||
1. Sends a request to `GET /api/users`
|
||||
2. `auth-required` plugin: `onRequest.requires` → `request_headers(this).authorization != null` → APOPHIS auto-injects `authorization: test-value` → passes
|
||||
3. Route-level ensures: `status:200` → passes; `response_body(this) is Array` → passes
|
||||
4. `request-id` plugin: `onSend.ensures` → `response_headers(this).x-request-id != null` → if the handler doesn't set `x-request-id`, this **fails** as a `plugin:request-id` contract violation
|
||||
|
||||
```text
|
||||
Plugin contract violation (plugin:request-id)
|
||||
GET /api/users
|
||||
Phase: onSend
|
||||
|
||||
Expected
|
||||
response_headers(this).x-request-id != null
|
||||
|
||||
Observed
|
||||
response_headers(this).x-request-id was null
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
- **`appliesTo` matches by route path only** by default. Use method-prefixed syntax (`"POST /api/**"`) for HTTP-method-specific rules.
|
||||
- **Header auto-injection only handles simple patterns** (`!= null` and `== "value"`). Complex preconditions like `>= 5` are not auto-injected; the route's ensures will be skipped.
|
||||
- **No per-route opt-out.** If a plugin contract matches a route, it always applies. Use narrower `appliesTo` patterns or separate route prefixes to exclude routes.
|
||||
- **Extension resolution is lazy.** Extensions are checked at route evaluation time, not at registration time. A missing extension is only discovered when a matching route is evaluated.
|
||||
- **At least one of `requires` or `ensures` must be present per phase.** A phase with neither is silently ignored.
|
||||
- **Runtime hook ordering follows Fastify's lifecycle.** Plugin contracts in `onResponse` cannot inspect the response body (it has already been sent to the client).
|
||||
@@ -0,0 +1,4 @@
|
||||
export default {
|
||||
profiles: { quick: { preset: "safe-ci" } },
|
||||
presets: { "safe-ci": { runs: 3 } },
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
import Fastify from "fastify";
|
||||
|
||||
const app = Fastify({ logger: false });
|
||||
|
||||
await app.register(import("@fastify/swagger"), {
|
||||
openapi: { info: { title: "Plugin Contracts Fixture", version: "1.0.0" } },
|
||||
});
|
||||
|
||||
let apophisPlugin;
|
||||
try {
|
||||
({ default: apophisPlugin } = await import("../../../index.js"));
|
||||
} catch {
|
||||
({ default: apophisPlugin } = await import("../../../../dist/index.js"));
|
||||
}
|
||||
|
||||
await app.register(apophisPlugin, {
|
||||
runtime: "off",
|
||||
pluginContracts: {
|
||||
"required-header": {
|
||||
appliesTo: "/api/**",
|
||||
hooks: { onRequest: { requires: ["request_headers(this).x-custom != null"] } },
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
app.get("/api/data", {
|
||||
schema: {
|
||||
"x-category": "observer",
|
||||
"x-ensures": ["status:200"],
|
||||
response: { 200: { type: "object", properties: { ok: { type: "boolean" } } } },
|
||||
},
|
||||
}, async () => ({ ok: true }));
|
||||
|
||||
export default app;
|
||||
@@ -0,0 +1 @@
|
||||
{"name":"plugin-contracts-fixture","type":"module","private":true}
|
||||
@@ -63,8 +63,26 @@ export async function runChaosOnRoute(
|
||||
const nonDelayEvents = events.filter(e => e.type !== 'inbound-delay')
|
||||
const application = applyChaosToExecution(ctx, nonDelayEvents)
|
||||
|
||||
// 5. Validate post-chaos contracts: did the route still satisfy its contracts?
|
||||
// Validate route contracts after chaos. Compose plugin contracts if available.
|
||||
let contractsPassed = true
|
||||
if (deps.pluginContractRegistry) {
|
||||
const composed = deps.pluginContractRegistry.composeContracts(route)
|
||||
if (!route.formulaSources) route.formulaSources = {}
|
||||
for (const phase of Object.values(composed.phases)) {
|
||||
for (const req of phase.requires) {
|
||||
if (!route.requires.includes(req.formula)) {
|
||||
route.requires.push(req.formula)
|
||||
route.formulaSources[req.formula] = req.source
|
||||
}
|
||||
}
|
||||
for (const ens of phase.ensures) {
|
||||
if (!route.ensures.includes(ens.formula)) {
|
||||
route.ensures.push(ens.formula)
|
||||
route.formulaSources[ens.formula] = ens.source
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if (application.applied && route.ensures.length > 0) {
|
||||
const { validatePostconditionsAsync } = await import('../../../domain/contract-validation.js')
|
||||
const result = await validatePostconditionsAsync(
|
||||
|
||||
@@ -30,6 +30,7 @@ import {
|
||||
type StepTrace,
|
||||
type CleanupFailure,
|
||||
} from './runner.js'
|
||||
import { isGitRepo, filterChangedRoutes } from '../verify/runner.js'
|
||||
import { SeededRng } from '../../../infrastructure/seeded-rng.js'
|
||||
import type { ScenarioConfig, TestConfig, RouteContract, ChaosConfig } from '../../../types.js'
|
||||
import { renderHumanArtifact } from '../../renderers/human.js'
|
||||
@@ -64,6 +65,7 @@ function coerceTimeout(value: unknown): number | undefined {
|
||||
export interface QualifyOptions {
|
||||
profile?: string
|
||||
seed?: number
|
||||
changed?: boolean
|
||||
config?: string
|
||||
cwd?: string
|
||||
format?: 'human' | 'json' | 'ndjson'
|
||||
@@ -252,7 +254,8 @@ export function buildArtifact(
|
||||
|
||||
const totalTests =
|
||||
runResult.scenarioResults.reduce((sum, s) => sum + s.steps.length, 0) +
|
||||
(runResult.statefulResult?.tests.length ?? 0)
|
||||
(runResult.statefulResult?.summary.passed ?? 0) +
|
||||
(runResult.statefulResult?.summary.failed ?? 0)
|
||||
|
||||
const passedTests =
|
||||
runResult.scenarioResults.reduce((sum, s) => sum + s.summary.passed, 0) +
|
||||
@@ -282,7 +285,11 @@ export function buildArtifact(
|
||||
.filter(Boolean)
|
||||
)]
|
||||
const statefulRoutes = runResult.statefulResult
|
||||
? [...new Set(runResult.statefulResult.tests.map(t => normalizeRouteIdentity(t.name)))]
|
||||
? [...new Set(
|
||||
runResult.statefulResult.tests
|
||||
.filter(t => !t.directive)
|
||||
.map(t => normalizeRouteIdentity(t.name))
|
||||
)]
|
||||
: []
|
||||
const chaosRoutesPlanned: string[] = []
|
||||
const chaosRoutesExecuted: string[] = []
|
||||
@@ -343,6 +350,7 @@ export function buildArtifact(
|
||||
total: totalTests,
|
||||
passed: passedTests,
|
||||
failed: failures.length,
|
||||
skipped: (runResult.statefulResult?.summary.skipped ?? 0),
|
||||
},
|
||||
executionSummary,
|
||||
coverageBreakdown,
|
||||
@@ -437,6 +445,7 @@ export async function qualifyCommand(
|
||||
const {
|
||||
profile,
|
||||
seed: explicitSeed,
|
||||
changed,
|
||||
config: configPath,
|
||||
cwd,
|
||||
artifactDir,
|
||||
@@ -566,7 +575,31 @@ export async function qualifyCommand(
|
||||
|
||||
try {
|
||||
// 9. Discover routes for chaos (with discovery metadata)
|
||||
const appRoutes = await discoverAppRoutes(fastify)
|
||||
const allRoutes = await discoverAppRoutes(fastify)
|
||||
let appRoutes = allRoutes
|
||||
|
||||
// 9a. Apply --changed filter
|
||||
if (changed) {
|
||||
const inGit = await isGitRepo(workingDir)
|
||||
if (!inGit) {
|
||||
return {
|
||||
exitCode: USAGE_ERROR,
|
||||
message: '--changed requires a git repository. Current directory is not inside a git repo.',
|
||||
}
|
||||
}
|
||||
appRoutes = await filterChangedRoutes(allRoutes, workingDir)
|
||||
const matched = appRoutes.length
|
||||
const total = allRoutes.length
|
||||
if (!ctx.options.quiet && format === 'human') {
|
||||
console.log(`${matched}/${total} routes match --changed filter`)
|
||||
}
|
||||
if (matched === 0) {
|
||||
return {
|
||||
exitCode: SUCCESS,
|
||||
message: `No routes changed since HEAD. All ${total} routes skipped.`,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Collect discovery warnings
|
||||
const { discoverRouteDetails } = await import('../../../domain/discovery.js')
|
||||
@@ -586,6 +619,8 @@ export async function qualifyCommand(
|
||||
fastify: fastify as any,
|
||||
seed,
|
||||
timeout: presetTimeout,
|
||||
pluginContractRegistry: (fastify as unknown as Record<string, unknown>)?._apophisPluginContractRegistry as
|
||||
import('../../../domain/plugin-contracts.js').PluginContractRegistry | undefined,
|
||||
}
|
||||
|
||||
const runResult = await runQualify(deps, gates, scenarios, statefulConfig, chaosConfig, appRoutes)
|
||||
@@ -600,6 +635,15 @@ export async function qualifyCommand(
|
||||
seed,
|
||||
})
|
||||
|
||||
// Drain plugin contract registry warnings if app registered APOPHIS
|
||||
const pcr = (fastify as any)?._apophisPluginContractRegistry
|
||||
if (pcr?.drainWarnings) {
|
||||
const pcrWarnings = pcr.drainWarnings()
|
||||
if (pcrWarnings.length > 0) {
|
||||
artifact.warnings.push(...pcrWarnings)
|
||||
}
|
||||
}
|
||||
|
||||
// 12. Signal quality guardrails — fail if zero checks executed
|
||||
const execSummary = runResult.executionSummary
|
||||
const warnings: string[] = [...artifact.warnings, ...discoveryWarnings]
|
||||
@@ -624,7 +668,9 @@ export async function qualifyCommand(
|
||||
if (gates.scenario && execSummary.scenariosRun === 0) {
|
||||
warnings.push('WARNING: scenario gate enabled but zero scenarios executed. Check route configuration.')
|
||||
}
|
||||
if (gates.stateful && execSummary.statefulTestsRun === 0) {
|
||||
if (gates.stateful && execSummary.statefulTestsRun === 0 && execSummary.statefulSkipped > 0) {
|
||||
warnings.push(`WARNING: stateful gate enabled but all ${execSummary.statefulSkipped} commands were skipped (preconditions not met). Check required headers, scopes, or plugin contract settings.`)
|
||||
} else if (gates.stateful && execSummary.statefulTestsRun === 0) {
|
||||
warnings.push('WARNING: stateful gate enabled but zero stateful tests executed. Check app routes and schema.')
|
||||
}
|
||||
if (gates.chaos && execSummary.chaosRunsRun === 0) {
|
||||
@@ -706,6 +752,7 @@ export async function handleQualify(
|
||||
const options: QualifyOptions = {
|
||||
profile: ctx.options.profile || undefined,
|
||||
seed: undefined,
|
||||
changed: false,
|
||||
config: ctx.options.config || undefined,
|
||||
cwd: ctx.cwd,
|
||||
format: ctx.options.format as QualifyOptions['format'],
|
||||
@@ -722,6 +769,10 @@ export async function handleQualify(
|
||||
}
|
||||
}
|
||||
|
||||
if (args.includes('--changed')) {
|
||||
options.changed = true
|
||||
}
|
||||
|
||||
const result = await qualifyCommand(options, ctx)
|
||||
const format = options.format || ctx.options.format || 'human'
|
||||
const machineMode = format === 'json' || format === 'ndjson' || format === 'json-summary' || format === 'ndjson-summary'
|
||||
|
||||
@@ -29,6 +29,7 @@ import type {
|
||||
FastifyInjectInstance,
|
||||
} from '../../../types.js'
|
||||
import type { ExtensionRegistry } from '../../../extension/types.js'
|
||||
import type { PluginContractRegistry } from '../../../domain/plugin-contracts.js'
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
@@ -60,6 +61,7 @@ export interface QualifyRunResult {
|
||||
totalFailed: number
|
||||
scenariosRun: number
|
||||
statefulTestsRun: number
|
||||
statefulSkipped: number
|
||||
chaosRunsRun: number
|
||||
chaosRoutesPlanned: number
|
||||
chaosRoutesExecuted: number
|
||||
@@ -85,6 +87,7 @@ export interface CleanupFailure {
|
||||
export interface QualifyRunnerDeps {
|
||||
fastify: FastifyInjectInstance
|
||||
extensionRegistry?: ExtensionRegistry
|
||||
pluginContractRegistry?: PluginContractRegistry
|
||||
seed: number
|
||||
timeout?: number
|
||||
}
|
||||
@@ -237,17 +240,20 @@ export async function runQualify(
|
||||
const statefulPassed = !statefulResult || statefulResult.summary.failed === 0
|
||||
const chaosPassed = chaosResults.every(r => !r.applied || r.contractsPassed)
|
||||
|
||||
// Count execution metrics
|
||||
// Count execution metrics — distinguish executed from skipped
|
||||
const scenariosRun = scenarioResults.length
|
||||
const statefulTestsRun = statefulResult?.tests.length ?? 0
|
||||
const statefulPassedCount = statefulResult?.summary.passed ?? 0
|
||||
const statefulFailedCount = statefulResult?.summary.failed ?? 0
|
||||
const statefulSkippedCount = statefulResult?.summary.skipped ?? 0
|
||||
const statefulTestsRun = statefulPassedCount + statefulFailedCount
|
||||
const chaosRunsRun = chaosResults.length
|
||||
const totalSteps = allTraces.length
|
||||
const totalExecuted = scenariosRun + statefulTestsRun + chaosRunsRun
|
||||
const totalPassed = scenarioResults.reduce((sum, r) => sum + r.summary.passed, 0) +
|
||||
(statefulResult?.summary.passed ?? 0) +
|
||||
statefulPassedCount +
|
||||
chaosResults.reduce((sum, r) => sum + (r.applied && r.contractsPassed ? 1 : 0), 0)
|
||||
const totalFailed = scenarioResults.reduce((sum, r) => sum + r.summary.failed, 0) +
|
||||
(statefulResult?.summary.failed ?? 0)
|
||||
statefulFailedCount
|
||||
|
||||
// Track executed and skipped routes for transparency
|
||||
const executedRoutes: string[] = []
|
||||
@@ -316,6 +322,7 @@ export async function runQualify(
|
||||
totalFailed,
|
||||
scenariosRun,
|
||||
statefulTestsRun,
|
||||
statefulSkipped: statefulSkippedCount,
|
||||
chaosRunsRun,
|
||||
chaosRoutesPlanned,
|
||||
chaosRoutesExecuted,
|
||||
|
||||
@@ -27,7 +27,7 @@ export async function runScenarioWithTraces(
|
||||
): Promise<{ result: ScenarioResult; traces: StepTrace[] }> {
|
||||
const scopeHeaders: Record<string, string> = {}
|
||||
|
||||
const result = await runScenario(deps.fastify, config, scopeHeaders, deps.extensionRegistry)
|
||||
const result = await runScenario(deps.fastify, config, scopeHeaders, deps.extensionRegistry, deps.pluginContractRegistry)
|
||||
|
||||
const traces: StepTrace[] = result.steps.map((step, idx) => {
|
||||
const trace: StepTrace = {
|
||||
|
||||
@@ -45,7 +45,7 @@ export async function runStatefulWithTraces(
|
||||
cleanupManager,
|
||||
minimalScopeRegistry,
|
||||
deps.extensionRegistry,
|
||||
undefined, // pluginContractRegistry — will be passed from runner when available
|
||||
deps.pluginContractRegistry,
|
||||
undefined, // outboundContractRegistry
|
||||
)
|
||||
|
||||
|
||||
@@ -172,9 +172,36 @@ async function executeContractDirect(
|
||||
validateRuntime: false,
|
||||
}
|
||||
|
||||
// Compose plugin contracts if the app registered the APOPHIS plugin
|
||||
const pluginContractRegistry = (fastify as any)?._apophisPluginContractRegistry
|
||||
if (pluginContractRegistry) {
|
||||
const composed = pluginContractRegistry.composeContracts(routeContract)
|
||||
if (!routeContract.formulaSources) routeContract.formulaSources = {}
|
||||
for (const phase of Object.values(composed.phases) as Array<{ requires: Array<{ formula: string; source: string }>; ensures: Array<{ formula: string; source: string }> }>) {
|
||||
for (const req of phase.requires) {
|
||||
if (!routeContract.requires.includes(req.formula)) {
|
||||
routeContract.requires.push(req.formula)
|
||||
routeContract.formulaSources[req.formula] = req.source as 'route' | `plugin:${string}`
|
||||
}
|
||||
}
|
||||
for (const ens of phase.ensures) {
|
||||
if (!routeContract.ensures.includes(ens.formula)) {
|
||||
routeContract.ensures.push(ens.formula)
|
||||
routeContract.formulaSources[ens.formula] = ens.source as 'route' | `plugin:${string}`
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Build request
|
||||
const headers: Record<string, string> = {}
|
||||
|
||||
// Auto-inject headers required by plugin contract preconditions
|
||||
for (const f of routeContract.requires) {
|
||||
const m = f.match(/request_headers\(this\)\.([\w-]+)\s*(!=\s*null|==\s*"([^"]+)")/)
|
||||
if (m) headers[m[1]!] = m[3] || 'test-value'
|
||||
}
|
||||
|
||||
// Execute request
|
||||
try {
|
||||
const ctx = await executeHttp(fastify, routeContract, {
|
||||
@@ -264,10 +291,12 @@ async function executeReplay(
|
||||
}
|
||||
|
||||
// Try to run verify first (works if app has APOPHIS plugin)
|
||||
const pluginContractRegistry = (fastify as any)?._apophisPluginContractRegistry
|
||||
let runResult = await runVerify({
|
||||
fastify: fastify as any,
|
||||
seed: artifact.seed || 42,
|
||||
routeFilters: [failure.route],
|
||||
pluginContractRegistry,
|
||||
})
|
||||
|
||||
// If no routes matched, or route found but no contracts (plugin not registered before routes),
|
||||
|
||||
@@ -170,6 +170,7 @@ function buildArtifact(
|
||||
total: runResult.total,
|
||||
passed: runResult.passedCount,
|
||||
failed: runResult.failed,
|
||||
skipped: runResult.skipped,
|
||||
},
|
||||
deterministicParams: {
|
||||
seed: options.seed,
|
||||
@@ -390,6 +391,7 @@ export async function verifyCommand(
|
||||
runs: typeof config.presets?.[loadResult.presetName || '']?.runs === 'number'
|
||||
? (config.presets[loadResult.presetName || ''] as { runs?: number }).runs
|
||||
: undefined,
|
||||
pluginContractRegistry: (fastify as any)?._apophisPluginContractRegistry,
|
||||
})
|
||||
|
||||
// 7. Build artifact
|
||||
@@ -403,6 +405,15 @@ export async function verifyCommand(
|
||||
routeFilters,
|
||||
})
|
||||
|
||||
// Drain plugin contract registry warnings if app registered APOPHIS
|
||||
const pcr = (fastify as any)?._apophisPluginContractRegistry
|
||||
if (pcr?.drainWarnings) {
|
||||
const pcrWarnings = pcr.drainWarnings()
|
||||
if (pcrWarnings.length > 0) {
|
||||
artifact.warnings.push(...pcrWarnings)
|
||||
}
|
||||
}
|
||||
|
||||
// 8. Write artifact if configured or on failure
|
||||
const shouldEmitArtifact = Boolean(artifactDir || config.artifactDir || !runResult.passed)
|
||||
await emitArtifact(artifact, {
|
||||
@@ -638,55 +649,28 @@ export async function handleVerify(
|
||||
const format = options.format || ctx.options.format || 'human'
|
||||
const machineMode = format === 'json' || format === 'ndjson' || format === 'json-summary' || format === 'ndjson-summary'
|
||||
|
||||
if (!ctx.options.quiet) {
|
||||
// Machine-readable formats always emit regardless of --quiet
|
||||
if (machineMode) {
|
||||
if (format === 'json') {
|
||||
if (result.artifact) {
|
||||
console.log(renderJsonArtifact(result.artifact))
|
||||
} else {
|
||||
console.log(renderJson({
|
||||
exitCode: result.exitCode,
|
||||
message: result.message,
|
||||
warnings: result.warnings,
|
||||
}))
|
||||
}
|
||||
console.log(result.artifact ? renderJsonArtifact(result.artifact) : renderJson({ exitCode: result.exitCode, message: result.message, warnings: result.warnings }))
|
||||
} else if (format === 'json-summary') {
|
||||
if (result.artifact) {
|
||||
console.log(renderJsonSummaryArtifact(result.artifact))
|
||||
} else {
|
||||
console.log(renderJson({
|
||||
exitCode: result.exitCode,
|
||||
message: result.message,
|
||||
warnings: result.warnings,
|
||||
}))
|
||||
}
|
||||
console.log(result.artifact ? renderJsonSummaryArtifact(result.artifact) : renderJson({ exitCode: result.exitCode, message: result.message, warnings: result.warnings }))
|
||||
} else if (format === 'ndjson') {
|
||||
if (result.artifact) {
|
||||
renderNdjsonArtifact(result.artifact)
|
||||
} else {
|
||||
process.stdout.write(JSON.stringify({
|
||||
type: 'run.completed',
|
||||
command: 'verify',
|
||||
exitCode: result.exitCode,
|
||||
message: result.message,
|
||||
warnings: result.warnings,
|
||||
}) + '\n')
|
||||
process.stdout.write(JSON.stringify({ type: 'run.completed', command: 'verify', exitCode: result.exitCode, message: result.message, warnings: result.warnings }) + '\n')
|
||||
}
|
||||
} else if (format === 'ndjson-summary') {
|
||||
if (result.artifact) {
|
||||
renderNdjsonSummaryArtifact(result.artifact)
|
||||
} else {
|
||||
process.stdout.write(JSON.stringify({
|
||||
type: 'run.completed',
|
||||
command: 'verify',
|
||||
exitCode: result.exitCode,
|
||||
message: result.message,
|
||||
warnings: result.warnings,
|
||||
}) + '\n')
|
||||
process.stdout.write(JSON.stringify({ type: 'run.completed', command: 'verify', exitCode: result.exitCode, message: result.message, warnings: result.warnings }) + '\n')
|
||||
}
|
||||
} else if (result.message) {
|
||||
}
|
||||
} else if (!ctx.options.quiet && result.message) {
|
||||
console.log(result.message)
|
||||
}
|
||||
}
|
||||
|
||||
// Print warnings in human mode only
|
||||
if (!machineMode && result.warnings && result.warnings.length > 0 && !ctx.options.quiet) {
|
||||
|
||||
@@ -26,6 +26,7 @@ import * as fc from 'fast-check'
|
||||
import { resolveRuns } from '../../../types.js'
|
||||
import type { EvalContext, RouteContract, FastifyInjectInstance } from '../../../types.js'
|
||||
import type { RouteResult } from '../../core/types.js'
|
||||
import type { PluginContractRegistry } from '../../../domain/plugin-contracts.js'
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
@@ -46,6 +47,7 @@ export interface VerifyRunResult {
|
||||
total: number
|
||||
passedCount: number
|
||||
failed: number
|
||||
skipped: number
|
||||
failures: VerifyFailure[]
|
||||
durationMs: number
|
||||
noRoutesMatched: boolean
|
||||
@@ -66,6 +68,7 @@ export interface VerifyRunnerDeps {
|
||||
changed?: boolean
|
||||
profileRoutes?: string[]
|
||||
runs?: number
|
||||
pluginContractRegistry?: PluginContractRegistry
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -161,7 +164,7 @@ function filterRoutesByPatterns(routes: RouteContract[], patterns: string[]): Ro
|
||||
/**
|
||||
* Check if cwd is inside a git repository.
|
||||
*/
|
||||
async function isGitRepo(cwd: string): Promise<boolean> {
|
||||
export async function isGitRepo(cwd: string): Promise<boolean> {
|
||||
try {
|
||||
const { execSync } = await import('node:child_process')
|
||||
execSync('git rev-parse --git-dir', { cwd, encoding: 'utf-8', stdio: 'pipe' })
|
||||
@@ -174,7 +177,7 @@ async function isGitRepo(cwd: string): Promise<boolean> {
|
||||
/**
|
||||
* Get git-modified files for --changed filtering.
|
||||
*/
|
||||
async function getGitChangedFiles(cwd: string): Promise<string[]> {
|
||||
export async function getGitChangedFiles(cwd: string): Promise<string[]> {
|
||||
try {
|
||||
const { execSync } = await import('node:child_process')
|
||||
const output = execSync('git diff --name-only HEAD', { cwd, encoding: 'utf-8' })
|
||||
@@ -187,7 +190,7 @@ async function getGitChangedFiles(cwd: string): Promise<string[]> {
|
||||
/**
|
||||
* Filter routes to only those modified in git.
|
||||
*/
|
||||
async function filterChangedRoutes(
|
||||
export async function filterChangedRoutes(
|
||||
routes: RouteContract[],
|
||||
cwd: string,
|
||||
): Promise<RouteContract[]> {
|
||||
@@ -481,6 +484,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
|
||||
total: 0,
|
||||
passedCount: 0,
|
||||
failed: 0,
|
||||
skipped: 0,
|
||||
failures: [],
|
||||
durationMs: Date.now() - started,
|
||||
noRoutesMatched: false,
|
||||
@@ -501,6 +505,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
|
||||
total: 0,
|
||||
passedCount: 0,
|
||||
failed: 0,
|
||||
skipped: 0,
|
||||
failures: [],
|
||||
durationMs: Date.now() - started,
|
||||
noRoutesMatched: true,
|
||||
@@ -511,6 +516,28 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
|
||||
}
|
||||
}
|
||||
|
||||
// 3a. Compose plugin contracts into route contracts
|
||||
if (deps.pluginContractRegistry) {
|
||||
for (const route of routes) {
|
||||
const composed = deps.pluginContractRegistry.composeContracts(route)
|
||||
if (!route.formulaSources) route.formulaSources = {}
|
||||
for (const phase of Object.values(composed.phases)) {
|
||||
for (const req of phase.requires) {
|
||||
if (!route.requires.includes(req.formula)) {
|
||||
route.requires.push(req.formula)
|
||||
route.formulaSources[req.formula] = req.source
|
||||
}
|
||||
}
|
||||
for (const ens of phase.ensures) {
|
||||
if (!route.ensures.includes(ens.formula)) {
|
||||
route.ensures.push(ens.formula)
|
||||
route.formulaSources[ens.formula] = ens.source
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Check for behavioral contracts
|
||||
const routesWithContracts = routes.filter(route =>
|
||||
route.ensures.length > 0 || route.requires.length > 0
|
||||
@@ -522,6 +549,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
|
||||
total: 0,
|
||||
passedCount: 0,
|
||||
failed: 0,
|
||||
skipped: 0,
|
||||
failures: [],
|
||||
durationMs: Date.now() - started,
|
||||
noRoutesMatched: false,
|
||||
@@ -533,32 +561,117 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
|
||||
}
|
||||
|
||||
// 4. Execute contracts (with variant expansion)
|
||||
// Execute one HTTP request per (route, variant, runIndex), then validate
|
||||
// all contracts against the same request context.
|
||||
// Requires (preconditions) that fail → SKIP the ensures and count as skipped.
|
||||
// Ensures (postconditions) that fail → count as failures.
|
||||
const runConfig = resolveRuns(deps.runs)
|
||||
const runs = runConfig.contractRuns
|
||||
const failures: VerifyFailure[] = []
|
||||
let total = 0
|
||||
let passedCount = 0
|
||||
let skippedCount = 0
|
||||
|
||||
// Extract required headers from precondition formulas and auto-inject them
|
||||
// so that plugin contracts (e.g., x-tenant-id != null) are satisfied.
|
||||
function extractRequiredHeaders(formulas: string[]): Record<string, string> {
|
||||
const headers: Record<string, string> = {}
|
||||
for (const f of formulas) {
|
||||
const nullCheck = f.match(/request_headers\(this\)\.([\w-]+)\s*!=\s*null/i)
|
||||
if (nullCheck) { headers[nullCheck[1]!] = 'test-value'; continue }
|
||||
const eqCheck = f.match(/request_headers\(this\)\.([\w-]+)\s*==\s*"([^"]+)"/i)
|
||||
if (eqCheck) { headers[eqCheck[1]!] = eqCheck[2]!; continue }
|
||||
}
|
||||
return headers
|
||||
}
|
||||
|
||||
for (const route of routesWithContracts) {
|
||||
const contracts = [...route.requires, ...route.ensures]
|
||||
const defaultHeaders = extractRequiredHeaders(route.requires)
|
||||
const variants = route.variants && route.variants.length > 0
|
||||
? route.variants
|
||||
: [{ name: 'default' }]
|
||||
|
||||
for (const variant of variants) {
|
||||
for (const contract of contracts) {
|
||||
for (let runIndex = 0; runIndex < runs; runIndex++) {
|
||||
total++
|
||||
const result = await executeContract(
|
||||
fastify, route, contract, deps.timeout,
|
||||
variant, deps.seed, runIndex,
|
||||
)
|
||||
// Execute one HTTP request per run
|
||||
const request = buildRouteRequest(route, deps.seed, runIndex, variant)
|
||||
const headers = {
|
||||
...defaultHeaders,
|
||||
...request.headers,
|
||||
...(variant.headers ?? {}),
|
||||
}
|
||||
const ctx = await executeHttp(fastify, route, {
|
||||
method: request.method,
|
||||
url: request.url,
|
||||
body: request.body,
|
||||
headers,
|
||||
query: {},
|
||||
}, undefined, deps.timeout)
|
||||
const evalCtx: EvalContext = {
|
||||
...ctx,
|
||||
operationResolver: createOperationResolver(fastify, headers, ctx),
|
||||
}
|
||||
|
||||
if (result.failure) {
|
||||
failures.push(result.failure)
|
||||
// Evaluate requires first. If any fail, skip ensures for this run.
|
||||
let requiresPassed = true
|
||||
for (const formula of route.requires) {
|
||||
total++
|
||||
try {
|
||||
const parsed = parse(formula)
|
||||
const result = await evaluateAsync(parsed.ast, evalCtx)
|
||||
if (!result.success || !result.value) {
|
||||
requiresPassed = false
|
||||
skippedCount++
|
||||
} else {
|
||||
passedCount++
|
||||
}
|
||||
} catch {
|
||||
requiresPassed = false
|
||||
skippedCount++
|
||||
}
|
||||
}
|
||||
|
||||
// Only evaluate ensures if all requires passed
|
||||
if (requiresPassed) {
|
||||
for (const formula of route.ensures) {
|
||||
total++
|
||||
try {
|
||||
const parsed = parse(formula)
|
||||
const result = await evaluateAsync(parsed.ast, evalCtx)
|
||||
if (!result.success || !result.value) {
|
||||
const diagnostic = buildFailureDiagnostic(formula, evalCtx, result.success ? undefined : result.error)
|
||||
failures.push({
|
||||
route: variant.name && variant.name !== 'default'
|
||||
? `[variant:${variant.name}] ${route.method} ${route.path}`
|
||||
: `${route.method} ${route.path}`,
|
||||
contract: formula,
|
||||
expected: diagnostic.expected,
|
||||
observed: diagnostic.observed,
|
||||
formula: diagnostic.formula,
|
||||
category: diagnostic.category,
|
||||
})
|
||||
} else {
|
||||
passedCount++
|
||||
}
|
||||
} catch (error) {
|
||||
const diagnostic = buildFailureDiagnostic(formula, evalCtx, error instanceof Error ? error.message : String(error))
|
||||
failures.push({
|
||||
route: variant.name && variant.name !== 'default'
|
||||
? `[variant:${variant.name}] ${route.method} ${route.path}`
|
||||
: `${route.method} ${route.path}`,
|
||||
contract: formula,
|
||||
expected: diagnostic.expected,
|
||||
observed: diagnostic.observed,
|
||||
formula: diagnostic.formula,
|
||||
category: diagnostic.category,
|
||||
})
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Preconditions failed: ensures are skipped but not evaluated.
|
||||
// We still count each ensure as "total" and "skipped" for transparency.
|
||||
total += route.ensures.length
|
||||
skippedCount += route.ensures.length
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -578,6 +691,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
|
||||
total,
|
||||
passedCount,
|
||||
failed: failures.length,
|
||||
skipped: skippedCount,
|
||||
failures: sortedFailures,
|
||||
durationMs,
|
||||
noRoutesMatched: false,
|
||||
|
||||
@@ -140,7 +140,7 @@ const CONFIG_SCHEMA: Record<string, SchemaField> = {
|
||||
optional: true,
|
||||
items: { type: 'object' },
|
||||
},
|
||||
chaos: {
|
||||
pluginContracts: {
|
||||
type: 'object',
|
||||
optional: true,
|
||||
properties: {},
|
||||
|
||||
@@ -93,7 +93,7 @@ export function createContext(options: Record<string, unknown> = {}): CliContext
|
||||
const packageManager = detectPackageManager(cwd);
|
||||
|
||||
// Normalize options
|
||||
const format = options.format === 'json' || options.format === 'ndjson'
|
||||
const format = options.format === 'json' || options.format === 'ndjson' || options.format === 'json-summary' || options.format === 'ndjson-summary'
|
||||
? options.format
|
||||
: 'human';
|
||||
|
||||
|
||||
@@ -102,9 +102,11 @@ function getCommandHelp(command: string): string {
|
||||
${pc.dim('Options:')}
|
||||
--profile <name> Profile name from config
|
||||
--seed <number> Deterministic seed
|
||||
--changed Filter to git-modified routes
|
||||
|
||||
${pc.dim('Examples:')}
|
||||
apophis qualify --profile oauth-nightly --seed 42
|
||||
apophis qualify --changed
|
||||
`,
|
||||
replay: `
|
||||
${pc.bold('apophis replay')} — Replay a failure using seed and stored trace
|
||||
@@ -167,18 +169,18 @@ function printInternalError(error: unknown): void {
|
||||
console.error();
|
||||
}
|
||||
|
||||
function resolveRequestedFormat(argv: string[]): 'human' | 'json' | 'ndjson' {
|
||||
function resolveRequestedFormat(argv: string[]): 'human' | 'json' | 'ndjson' | 'json-summary' | 'ndjson-summary' {
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const arg = argv[i];
|
||||
if (!arg) continue;
|
||||
if (arg === '--format' && argv[i + 1]) {
|
||||
const value = argv[i + 1];
|
||||
if (value === 'json' || value === 'ndjson') return value;
|
||||
if (value === 'json' || value === 'ndjson' || value === 'json-summary' || value === 'ndjson-summary') return value as 'json' | 'ndjson' | 'json-summary' | 'ndjson-summary';
|
||||
return 'human';
|
||||
}
|
||||
if (arg.startsWith('--format=')) {
|
||||
const value = arg.slice('--format='.length);
|
||||
if (value === 'json' || value === 'ndjson') return value;
|
||||
if (value === 'json' || value === 'ndjson' || value === 'json-summary' || value === 'ndjson-summary') return value as 'json' | 'ndjson' | 'json-summary' | 'ndjson-summary';
|
||||
return 'human';
|
||||
}
|
||||
}
|
||||
@@ -277,6 +279,7 @@ export async function main(argv: string[] = process.argv.slice(2)): Promise<numb
|
||||
case 'qualify':
|
||||
cmd.option('--profile <name>', 'Profile name from config');
|
||||
cmd.option('--seed <number>', 'Deterministic seed');
|
||||
cmd.option('--changed', 'Filter to git-modified routes');
|
||||
break;
|
||||
case 'replay':
|
||||
cmd.option('--artifact <path>', 'Path to failure artifact');
|
||||
@@ -375,7 +378,7 @@ export async function main(argv: string[] = process.argv.slice(2)): Promise<numb
|
||||
init: new Set(['--preset', '-p', '--force', '-f', '--noninteractive']),
|
||||
verify: new Set(['--profile', '--routes', '--seed', '--changed', '--workspace']),
|
||||
observe: new Set(['--profile', '--check-config', '--workspace']),
|
||||
qualify: new Set(['--profile', '--seed', '--workspace']),
|
||||
qualify: new Set(['--profile', '--seed', '--workspace', '--changed']),
|
||||
replay: new Set(['--artifact', '--route']),
|
||||
doctor: new Set(['--mode', '--strict', '--workspace']),
|
||||
migrate: new Set(['--check', '--dry-run', '--write']),
|
||||
|
||||
@@ -297,6 +297,7 @@ export interface Artifact {
|
||||
total: number;
|
||||
passed: number;
|
||||
failed: number;
|
||||
skipped?: number;
|
||||
};
|
||||
executionSummary?: ExecutionSummary;
|
||||
coverageBreakdown?: {
|
||||
|
||||
@@ -202,6 +202,10 @@ function renderSummary(
|
||||
lines.push(` Failed: ${summary.failed}`);
|
||||
}
|
||||
|
||||
if (summary.skipped !== undefined && summary.skipped > 0) {
|
||||
lines.push(` ${c.yellow('Skipped:')} ${summary.skipped} (preconditions not met)`);
|
||||
}
|
||||
|
||||
lines.push(` Duration: ${formatDuration(artifact.durationMs)}`);
|
||||
|
||||
if (artifact.seed !== undefined) {
|
||||
@@ -404,15 +408,31 @@ export function renderHumanArtifact(
|
||||
lines.push(c.bold('Next steps'));
|
||||
if (artifact.command === 'verify') {
|
||||
if (artifact.summary.failed === 0) {
|
||||
if (artifact.summary.skipped && artifact.summary.skipped > 0) {
|
||||
lines.push(` ${c.yellow('!')} No contract violations, but ${artifact.summary.skipped} precondition(s) were unmet and skipped.`);
|
||||
lines.push(` ${c.dim('→ Preconditions (x-requires + plugin contract requires) must be satisfied for tests to run.')}`);
|
||||
lines.push(` ${c.dim('→ Send required headers via --variant or scope configuration.')}`);
|
||||
lines.push(` ${c.dim('→ Add more behavioral contracts with')} x-ensures ${c.dim('and')} x-requires ${c.dim('to cover more routes.')}`);
|
||||
} else {
|
||||
lines.push(` ${c.green('✓')} All contracts passed.`);
|
||||
lines.push(` ${c.dim('→ Add more behavioral contracts with')} x-ensures ${c.dim('and')} x-requires ${c.dim('to cover more routes.')}`);
|
||||
lines.push(` ${c.dim('→ Run')} apophis observe ${c.dim('to enable runtime contract monitoring in production.')}`);
|
||||
lines.push(` ${c.dim('→ Run')} apophis qualify --profile standard ${c.dim('for stateful/chaos testing.')}`);
|
||||
}
|
||||
} else {
|
||||
lines.push(` ${c.yellow('!')} Fix failing contracts and rerun with:`);
|
||||
lines.push(` ${c.dim('→')} apophis verify --seed ${artifact.seed} ${artifact.profile ? `--profile ${artifact.profile}` : ''}`);
|
||||
lines.push(` ${c.dim('→ Or replay the artifact:')} apophis replay --artifact <path>`);
|
||||
}
|
||||
} else if (artifact.command === 'qualify') {
|
||||
if (artifact.summary.skipped && artifact.summary.skipped > 0) {
|
||||
lines.push(` ${c.yellow('!')} ${artifact.summary.skipped} stateful command(s) skipped — preconditions not met.`);
|
||||
lines.push(` ${c.dim('→ Check plugin contracts and route headers. Qualify needs preconditions satisfied.')}`);
|
||||
}
|
||||
if (artifact.summary.failed === 0 && (!artifact.summary.skipped || artifact.summary.skipped === 0)) {
|
||||
lines.push(` ${c.green('✓')} All checks passed.`);
|
||||
}
|
||||
lines.push(` ${c.dim('→ Review')} reports/apophis/ ${c.dim('for detailed artifact output.')}`);
|
||||
}
|
||||
|
||||
return lines.join('\n');
|
||||
|
||||
@@ -179,11 +179,13 @@ const makeConditionFailure = (
|
||||
const expected = fieldExpectation?.expected ?? statusExpectation.expected
|
||||
const actual = fieldExpectation?.actual ?? statusExpectation.actual
|
||||
const diff = buildDiff(formula, ctx.response.body)
|
||||
const source = (route as RouteContract | undefined)?.formulaSources?.[formula]
|
||||
|
||||
return makeViolation({
|
||||
route: route ?? { method: '', path: '' },
|
||||
formula,
|
||||
kind,
|
||||
...(source ? { source, phase: 'postcondition' } : {}),
|
||||
request: {
|
||||
body: ctx.request.body,
|
||||
headers: ctx.request.headers,
|
||||
@@ -206,10 +208,12 @@ const makeFormulaError = (
|
||||
route: RouteContract | { method: string; path: string } | undefined,
|
||||
message: string
|
||||
): ContractViolation => {
|
||||
const source = (route as RouteContract | undefined)?.formulaSources?.[formula]
|
||||
return makeViolation({
|
||||
route: route ?? { method: '', path: '' },
|
||||
formula,
|
||||
kind,
|
||||
...(source ? { source, phase: 'postcondition' } : {}),
|
||||
request: {
|
||||
body: ctx.request.body,
|
||||
headers: ctx.request.headers,
|
||||
|
||||
@@ -39,6 +39,7 @@ function matchPattern(pattern: string, path: string): boolean {
|
||||
export class PluginContractRegistry {
|
||||
private contracts = new Map<string, PluginContractSpec>()
|
||||
private availableExtensions = new Set<string>()
|
||||
private _warnings: string[] = []
|
||||
/**
|
||||
* Register a plugin's contract specification.
|
||||
* Idempotent: registering the same plugin twice updates the spec.
|
||||
@@ -75,7 +76,7 @@ export class PluginContractRegistry {
|
||||
if (!matchPattern(spec.appliesTo, route.path)) continue
|
||||
const extCheck = this.checkExtensions(spec)
|
||||
if (!extCheck.available) {
|
||||
console.warn(
|
||||
this._warnings.push(
|
||||
`Plugin '${plugin}' requires extensions [${extCheck.missing.join(', ')}] which are not registered. Skipping its contracts.`
|
||||
)
|
||||
continue
|
||||
@@ -141,6 +142,16 @@ export class PluginContractRegistry {
|
||||
getAvailableExtensions(): string[] {
|
||||
return Array.from(this.availableExtensions)
|
||||
}
|
||||
/**
|
||||
* Collect and clear warnings (e.g., missing extension dependencies).
|
||||
* Callers should surface these via their own output channels instead of
|
||||
* letting console.warn bleed into CLI formatted output.
|
||||
*/
|
||||
drainWarnings(): string[] {
|
||||
const w = this._warnings
|
||||
this._warnings = []
|
||||
return w
|
||||
}
|
||||
}
|
||||
// ============================================================================
|
||||
// Factory
|
||||
|
||||
+15
-2
@@ -1,11 +1,17 @@
|
||||
import type { FastifyInstance, FastifyServerOptions } from 'fastify'
|
||||
import Fastify from 'fastify'
|
||||
import { installRouteDiscovery } from './infrastructure/discovery-hook.js'
|
||||
import type { ObserveSink } from './types.js'
|
||||
|
||||
export interface CreateFastifyOptions extends FastifyServerOptions {
|
||||
apophis?: {
|
||||
runtime?: 'off' | 'warn' | 'error'
|
||||
discoveryOnly?: boolean
|
||||
observe?: {
|
||||
enabled?: boolean
|
||||
sampling?: number
|
||||
sinks?: ObserveSink[]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -16,9 +22,16 @@ export async function createFastify(
|
||||
const fastify = Fastify(fastifyOpts)
|
||||
installRouteDiscovery(fastify as unknown as { addHook: Function })
|
||||
|
||||
if (apophisOpts?.discoveryOnly !== true && apophisOpts?.runtime && apophisOpts.runtime !== 'off') {
|
||||
const shouldRegisterApophis = apophisOpts?.discoveryOnly !== true && (
|
||||
(apophisOpts?.runtime !== undefined && apophisOpts.runtime !== 'off') ||
|
||||
apophisOpts?.observe?.enabled === true
|
||||
)
|
||||
if (shouldRegisterApophis) {
|
||||
const { apophisPlugin } = await import('./plugin/index.js')
|
||||
await fastify.register(apophisPlugin, { runtime: apophisOpts.runtime })
|
||||
await fastify.register(apophisPlugin, {
|
||||
runtime: apophisOpts.runtime,
|
||||
observe: apophisOpts.observe,
|
||||
})
|
||||
}
|
||||
|
||||
return fastify
|
||||
|
||||
+31
-4
@@ -727,6 +727,8 @@ function parseVariable(input: string, pos: number): { node: FormulaNode; pos: nu
|
||||
}
|
||||
}
|
||||
const name = input.substring(p, i)
|
||||
// F/T shorthands: treat as boolean literals, not variables.
|
||||
// Only apply when there's no accessor (e.g., F.x is a variable, not literal).
|
||||
let accessor: string[] | undefined
|
||||
const dotPos = skipWs(input, i)
|
||||
if (input[dotPos] === '.') {
|
||||
@@ -734,6 +736,12 @@ function parseVariable(input: string, pos: number): { node: FormulaNode; pos: nu
|
||||
accessor = [acc.id]
|
||||
i = acc.pos
|
||||
}
|
||||
if ((name === 'F' || name === 'T') && accessor === undefined) {
|
||||
return {
|
||||
node: { type: 'literal', value: name === 'T' },
|
||||
pos: i
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
node: { type: 'variable', name, accessor },
|
||||
@@ -842,12 +850,31 @@ function isWordChar(c: number): boolean {
|
||||
function parseClause(input: string, pos: number, extensionHeaders: string[] = []): { node: FormulaNode; pos: number } {
|
||||
const p = skipWs(input, pos)
|
||||
const len = input.length
|
||||
if (p < len && input[p] === 'T' && (p + 1 >= len || !isWordChar(input.charCodeAt(p + 1)))) {
|
||||
return { node: { type: 'literal', value: true }, pos: p + 1 }
|
||||
|
||||
// T/F shorthand: only consume as standalone literal if NOT followed by a
|
||||
// comparison operator (==, !=, <, >, <=, >=, matches).
|
||||
function isTfLiteral(): boolean {
|
||||
if (p >= len) return false
|
||||
const ch = input[p]!
|
||||
if (ch !== 'T' && ch !== 'F') return false
|
||||
const afterP = p + 1
|
||||
if (afterP >= len || !isWordChar(input.charCodeAt(afterP))) {
|
||||
const next = skipWs(input, afterP)
|
||||
const c0 = next < len ? input.charCodeAt(next) : -1
|
||||
const c1 = next + 1 < len ? input.charCodeAt(next + 1) : -1
|
||||
// If followed by a comparator, T/F is a term, not a standalone literal
|
||||
if (c0 === 61 || c0 === 33 || c0 === 60 || c0 === 62) return false // ==, !=, <=, <, >=, >
|
||||
if (c0 === 109) return false // matches
|
||||
return true
|
||||
}
|
||||
if (p < len && input[p] === 'F' && (p + 1 >= len || !isWordChar(input.charCodeAt(p + 1)))) {
|
||||
return { node: { type: 'literal', value: false }, pos: p + 1 }
|
||||
return false
|
||||
}
|
||||
|
||||
if (isTfLiteral()) {
|
||||
const ch = input[p]!
|
||||
return { node: { type: 'literal', value: ch === 'T' }, pos: p + 1 }
|
||||
}
|
||||
|
||||
const cmp = parseComparison(input, pos, extensionHeaders)
|
||||
if (cmp) return cmp
|
||||
// If not a comparison, it must be a term (operation or literal)
|
||||
|
||||
@@ -67,12 +67,13 @@ export const buildCleanup = (cleanupManager: CleanupManager) =>
|
||||
export const buildScenario = (
|
||||
fastify: FastifyInstance,
|
||||
scope: ScopeRegistry,
|
||||
extensionRegistry: ExtensionRegistry
|
||||
extensionRegistry: ExtensionRegistry,
|
||||
pluginContractRegistry?: import('../domain/plugin-contracts.js').PluginContractRegistry
|
||||
) => async (opts: ScenarioConfig): Promise<ScenarioResult> => {
|
||||
assertNonProduction('scenario')
|
||||
const scopeHeaders = scope.getHeaders(opts.scope ?? null)
|
||||
const injectInstance = fastify as unknown as import('../types.js').FastifyInjectInstance
|
||||
return runScenario(injectInstance, opts, scopeHeaders, extensionRegistry)
|
||||
return runScenario(injectInstance, opts, scopeHeaders, extensionRegistry, pluginContractRegistry)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
+15
-7
@@ -96,7 +96,7 @@ export const apophisPlugin = async (fastify: FastifyInstance, opts: ApophisOptio
|
||||
contract: buildContract(fastify, scope, extensionRegistry, pluginContractRegistry, outboundContractRegistry),
|
||||
stateful: buildStateful(fastify, scope, cleanupManager, extensionRegistry, pluginContractRegistry, outboundContractRegistry),
|
||||
check: buildCheck(fastify, scope, extensionRegistry, pluginContractRegistry),
|
||||
scenario: buildScenario(fastify, scope, extensionRegistry),
|
||||
scenario: buildScenario(fastify, scope, extensionRegistry, pluginContractRegistry),
|
||||
cleanup: buildCleanup(cleanupManager),
|
||||
spec: buildSpec(fastify),
|
||||
test: {
|
||||
@@ -147,17 +147,25 @@ export const apophisPlugin = async (fastify: FastifyInstance, opts: ApophisOptio
|
||||
},
|
||||
}
|
||||
fastify.decorate('apophis', decorations)
|
||||
// Runtime validation: never register hooks in production
|
||||
// Expose plugin contract registry for CLI qualify access
|
||||
;(fastify as unknown as Record<string, unknown>)._apophisPluginContractRegistry = pluginContractRegistry
|
||||
// Runtime validation may block requests, so it is disabled in production.
|
||||
// Observe sinks are explicitly non-blocking and are allowed in production.
|
||||
const isProd = process.env.NODE_ENV === 'production' || process.env.NODE_ENV === 'prod'
|
||||
if (opts.runtime && opts.runtime !== 'off' && !isProd) {
|
||||
const observeSinks = opts.observe?.enabled && opts.observe.sinks && opts.observe.sinks.length > 0
|
||||
? opts.observe.sinks
|
||||
: undefined
|
||||
const enableBlockingRuntime = opts.runtime && opts.runtime !== 'off' && !isProd
|
||||
if (enableBlockingRuntime || observeSinks) {
|
||||
const observeSampling = opts.observe?.sampling ?? 1
|
||||
const hookOpts: { validateRuntime: boolean; runtimeLevel?: 'warn' | 'error'; observers?: { sinks: import('../types.js').ObserveSink[]; sampling: number } } = {
|
||||
validateRuntime: true,
|
||||
runtimeLevel: opts.runtime,
|
||||
runtimeLevel: enableBlockingRuntime ? opts.runtime : 'warn',
|
||||
}
|
||||
if (opts.observe?.enabled && opts.observe.sinks && opts.observe.sinks.length > 0) {
|
||||
if (observeSinks) {
|
||||
hookOpts.observers = {
|
||||
sinks: opts.observe.sinks,
|
||||
sampling: opts.observe.sampling ?? 1,
|
||||
sinks: observeSinks,
|
||||
sampling: observeSampling,
|
||||
}
|
||||
}
|
||||
registerValidationHooks(fastify, hookOpts as Parameters<typeof registerValidationHooks>[1])
|
||||
|
||||
@@ -118,15 +118,18 @@ export const runPetitTests = async (
|
||||
if (pluginContractRegistry) {
|
||||
for (const route of routes) {
|
||||
const composed = pluginContractRegistry.composeContracts(route)
|
||||
if (!route.formulaSources) route.formulaSources = {}
|
||||
for (const phase of Object.values(composed.phases)) {
|
||||
for (const req of phase.requires) {
|
||||
if (!route.requires.includes(req.formula)) {
|
||||
route.requires.push(req.formula)
|
||||
route.formulaSources[req.formula] = req.source
|
||||
}
|
||||
}
|
||||
for (const ens of phase.ensures) {
|
||||
if (!route.ensures.includes(ens.formula)) {
|
||||
route.ensures.push(ens.formula)
|
||||
route.formulaSources[ens.formula] = ens.source
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -276,7 +279,16 @@ export const runPetitTests = async (
|
||||
|
||||
flushCache()
|
||||
const deduped = deduplicateTestFailures(results).results
|
||||
const suite: TestSuite = buildPetitSuite(allRoutes, routes, skippedRoutes, deduped, cacheHits, cacheMisses, startTime)
|
||||
const { formatSuite } = await import('./formatters.js')
|
||||
const suite: TestSuite = formatSuite(deduped, {
|
||||
startTime,
|
||||
cacheHits,
|
||||
cacheMisses,
|
||||
allRoutes,
|
||||
testedRoutes: routes,
|
||||
skippedRoutes,
|
||||
pluginContractRegistry,
|
||||
})
|
||||
|
||||
if (suiteMockRuntime) suiteMockRuntime.restore()
|
||||
if (extensionRegistry) await extensionRegistry.runSuiteEndHooks(suite)
|
||||
|
||||
@@ -5,6 +5,7 @@ import { executeHttp } from '../infrastructure/http-executor.js'
|
||||
import { validatePostconditionsAsync } from '../domain/contract-validation.js'
|
||||
import { CONTENT_TYPE } from '../infrastructure/http-executor.js'
|
||||
import type { ExtensionRegistry } from '../extension/types.js'
|
||||
import type { PluginContractRegistry } from '../domain/plugin-contracts.js'
|
||||
import type {
|
||||
EvalContext,
|
||||
FastifyInjectInstance,
|
||||
@@ -123,7 +124,8 @@ export const runScenario = async (
|
||||
fastify: FastifyInjectInstance,
|
||||
config: ScenarioConfig,
|
||||
scopeHeaders: Record<string, string>,
|
||||
extensionRegistry?: ExtensionRegistry
|
||||
extensionRegistry?: ExtensionRegistry,
|
||||
pluginContractRegistry?: PluginContractRegistry
|
||||
): Promise<ScenarioResult> => {
|
||||
const started = Date.now()
|
||||
const steps: ScenarioStepResult[] = []
|
||||
@@ -162,6 +164,37 @@ export const runScenario = async (
|
||||
}
|
||||
|
||||
const route = buildScenarioRoute({ ...interpolated, method })
|
||||
|
||||
// Merge plugin contracts into the synthetic route
|
||||
if (pluginContractRegistry) {
|
||||
const composed = pluginContractRegistry.composeContracts(route)
|
||||
if (!route.formulaSources) route.formulaSources = {}
|
||||
for (const phase of Object.values(composed.phases)) {
|
||||
for (const req of phase.requires) {
|
||||
if (!route.requires.includes(req.formula)) {
|
||||
route.requires.push(req.formula)
|
||||
route.formulaSources[req.formula] = req.source
|
||||
}
|
||||
}
|
||||
for (const ens of phase.ensures) {
|
||||
if (!route.ensures.includes(ens.formula)) {
|
||||
route.ensures.push(ens.formula)
|
||||
route.formulaSources[ens.formula] = ens.source
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Auto-inject headers required by plugin contract preconditions
|
||||
for (const f of route.requires) {
|
||||
const m = f.match(/request_headers\(this\)\.([\w-]+)\s*(!=\s*null|==\s*"([^"]+)")/)
|
||||
if (m) {
|
||||
if (headers[m[1]!] === undefined) {
|
||||
headers[m[1]!] = m[3] || 'test-value'
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const preContext: EvalContext = {
|
||||
request: {
|
||||
body,
|
||||
@@ -204,7 +237,8 @@ export const runScenario = async (
|
||||
operationResolver: createOperationResolver(fastify, headers, preContext),
|
||||
}
|
||||
|
||||
const validation = await validatePostconditionsAsync(step.expect.slice(), evalCtx, route, extensionRegistry)
|
||||
const mergedEnsures = [...step.expect, ...route.ensures]
|
||||
const validation = await validatePostconditionsAsync(mergedEnsures, evalCtx, route, extensionRegistry)
|
||||
if (!validation.success) {
|
||||
steps.push({
|
||||
name: step.name,
|
||||
|
||||
@@ -49,6 +49,17 @@ const runBuildAndPrecondition = async (
|
||||
const { fastify, scopeHeaders, rng, extensionRegistry } = runtime
|
||||
|
||||
let request = buildRequest(command.route, command.params, scopeHeaders, modelState, rng)
|
||||
// Auto-inject headers required by plugin contract preconditions
|
||||
if (command.route.requires.length > 0) {
|
||||
const merged = { ...request.headers }
|
||||
for (const f of command.route.requires) {
|
||||
const m = f.match(/request_headers\(this\)\.([\w-]+)\s*(!=\s*null|==\s*"([^"]+)")/)
|
||||
if (m) {
|
||||
merged[m[1]!] = m[3] || 'test-value'
|
||||
}
|
||||
}
|
||||
request.headers = merged
|
||||
}
|
||||
if (extensionRegistry) {
|
||||
request = await extensionRegistry.runBuildRequestHooks({
|
||||
route: command.route,
|
||||
|
||||
@@ -22,7 +22,7 @@ import type { OutboundContractRegistry } from '../domain/outbound-contracts.js'
|
||||
import * as fc from 'fast-check'
|
||||
import type { ModelState } from '../domain/stateful.js'
|
||||
import type { CleanupManager } from '../infrastructure/cleanup-manager.js'
|
||||
import type { EvalContext, FastifyInjectInstance, RouteContract, ScopeRegistry, TestConfig, TestResult, TestSuite } from '../types.js'
|
||||
import type { EvalContext, FastifyInjectInstance, RouteContract, ScopeRegistry, TestConfig, TestResult, TestSuite, ContractViolation } from '../types.js'
|
||||
|
||||
// Pure: hash helpers for deterministic sub-seeds
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -105,15 +105,18 @@ export const runStatefulTests = async (
|
||||
if (pluginContractRegistry) {
|
||||
for (const route of routes) {
|
||||
const composed = pluginContractRegistry.composeContracts(route)
|
||||
if (!route.formulaSources) route.formulaSources = {}
|
||||
for (const phase of Object.values(composed.phases)) {
|
||||
for (const req of phase.requires) {
|
||||
if (!route.requires.includes(req.formula)) {
|
||||
route.requires.push(req.formula)
|
||||
route.formulaSources[req.formula] = req.source
|
||||
}
|
||||
}
|
||||
for (const ens of phase.ensures) {
|
||||
if (!route.ensures.includes(ens.formula)) {
|
||||
route.ensures.push(ens.formula)
|
||||
route.formulaSources[ens.formula] = ens.source
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -303,9 +306,36 @@ export const runStatefulTests = async (
|
||||
const passed = dedupedResults.filter((r) => r.ok && r.directive === undefined).length
|
||||
const failed = dedupedResults.filter((r) => !r.ok).length
|
||||
const skipped = dedupedResults.filter((r) => r.directive !== undefined).length
|
||||
// Compute plugin contract counts
|
||||
let pluginContractsApplied = 0
|
||||
let pluginContractsFailed = 0
|
||||
if (pluginContractRegistry) {
|
||||
for (const route of routes) {
|
||||
const composed = pluginContractRegistry.composeContracts(route)
|
||||
for (const phase of Object.values(composed.phases)) {
|
||||
pluginContractsApplied += phase.ensures.length + phase.requires.length
|
||||
}
|
||||
}
|
||||
for (const result of dedupedResults) {
|
||||
if (!result.ok && result.diagnostics?.violation) {
|
||||
const violation = result.diagnostics.violation as ContractViolation
|
||||
if (violation.source?.startsWith('plugin:')) pluginContractsFailed++
|
||||
}
|
||||
}
|
||||
}
|
||||
const suite: TestSuite = {
|
||||
tests: dedupedResults,
|
||||
summary: { passed, failed, skipped, timeMs: Date.now() - startTime, cacheHits, cacheMisses, counterexample: counterexampleOutput },
|
||||
summary: {
|
||||
passed,
|
||||
failed,
|
||||
skipped,
|
||||
timeMs: Date.now() - startTime,
|
||||
cacheHits,
|
||||
cacheMisses,
|
||||
counterexample: counterexampleOutput,
|
||||
pluginContractsApplied,
|
||||
pluginContractsFailed,
|
||||
},
|
||||
routes: allRoutes.map(r => ({
|
||||
path: r.path,
|
||||
method: r.method,
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
import { test } from 'node:test';
|
||||
import assert from 'node:assert';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
|
||||
function runCli(args: string[]): { status: number; stdout: string; stderr: string } {
|
||||
const result = spawnSync(
|
||||
@@ -84,3 +86,158 @@ test('global help in machine mode emits parseable JSON', () => {
|
||||
assert.ok(typeof parsed.help === 'string');
|
||||
assert.ok(parsed.help.includes('Usage:'));
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// json-summary and ndjson-summary format tests
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
test('verify --format json-summary emits parseable JSON with summary fields', () => {
|
||||
const { status, stdout, stderr } = runCli([
|
||||
'verify', '--cwd', 'src/cli/__fixtures__/tiny-fastify',
|
||||
'--profile', 'quick', '--seed', '42', '--format', 'json-summary',
|
||||
]);
|
||||
|
||||
assert.strictEqual(status, 0);
|
||||
assert.strictEqual(stderr.trim(), '');
|
||||
assert.ok(!stdout.includes('Seed:'), 'machine output must not include human seed prelude');
|
||||
|
||||
const parsed = JSON.parse(stdout);
|
||||
assert.strictEqual(parsed.command, 'verify');
|
||||
assert.strictEqual(typeof parsed.summary.total, 'number');
|
||||
assert.strictEqual(typeof parsed.summary.passed, 'number');
|
||||
assert.strictEqual(typeof parsed.summary.failed, 'number');
|
||||
assert.strictEqual(typeof parsed.seed, 'number');
|
||||
assert.strictEqual(typeof parsed.exitReason, 'string');
|
||||
});
|
||||
|
||||
test('verify --format ndjson-summary emits only NDJSON records', () => {
|
||||
const { status, stdout, stderr } = runCli([
|
||||
'verify', '--cwd', 'src/cli/__fixtures__/tiny-fastify',
|
||||
'--profile', 'quick', '--seed', '42', '--format', 'ndjson-summary',
|
||||
]);
|
||||
|
||||
assert.strictEqual(status, 0);
|
||||
assert.strictEqual(stderr.trim(), '');
|
||||
assert.ok(!stdout.includes('Seed:'), 'machine output must not include human seed prelude');
|
||||
|
||||
const lines = stdout.split('\n').map((line) => line.trim()).filter(Boolean);
|
||||
assert.ok(lines.length >= 1, 'ndjson-summary should emit at least one record');
|
||||
for (const line of lines) {
|
||||
const parsed = JSON.parse(line);
|
||||
assert.ok(typeof parsed.type === 'string' || typeof parsed.summary === 'object' || typeof parsed.command === 'string');
|
||||
}
|
||||
});
|
||||
|
||||
test('verify --format json-summary --quiet still emits JSON', () => {
|
||||
const { status, stdout, stderr } = runCli([
|
||||
'verify', '--cwd', 'src/cli/__fixtures__/tiny-fastify',
|
||||
'--profile', 'quick', '--seed', '42', '--format', 'json-summary', '--quiet',
|
||||
]);
|
||||
|
||||
assert.strictEqual(status, 0);
|
||||
assert.strictEqual(stderr.trim(), '');
|
||||
assert.ok(stdout.trim().length > 0, '--quiet must not suppress machine-readable JSON');
|
||||
const parsed = JSON.parse(stdout);
|
||||
assert.strictEqual(parsed.command, 'verify');
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// skipped field in artifact
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
test('verify --format json-summary includes skipped field when preconditions cannot be met', () => {
|
||||
const { status, stdout, stderr } = runCli([
|
||||
'verify', '--cwd', 'src/cli/__fixtures__/plugin-contracts',
|
||||
'--profile', 'quick', '--seed', '42', '--format', 'json-summary',
|
||||
]);
|
||||
|
||||
// Plugin contract requires x-custom header which the generated request won't send.
|
||||
// Auto-injection should add it, so preconditions pass.
|
||||
// If not, skipped > 0 and failed === 0.
|
||||
const parsed = JSON.parse(stdout);
|
||||
assert.strictEqual(parsed.command, 'verify');
|
||||
assert.ok(typeof parsed.summary.skipped === 'number', 'skipped field must be present');
|
||||
assert.strictEqual(parsed.summary.failed, 0, 'no postcondition failures expected');
|
||||
});
|
||||
|
||||
test('verify --format json-summary exit code 0 when all contracts pass', () => {
|
||||
const { status, stdout } = runCli([
|
||||
'verify', '--cwd', 'src/cli/__fixtures__/tiny-fastify',
|
||||
'--seed', '42', '--format', 'json-summary', '--quiet',
|
||||
]);
|
||||
|
||||
assert.strictEqual(status, 0, 'exit code must be 0 when all contracts pass');
|
||||
const parsed = JSON.parse(stdout);
|
||||
assert.strictEqual(parsed.summary.failed, 0);
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// qualify json-summary and round-trip
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
test('qualify --format json-summary emits parseable JSON', () => {
|
||||
const { status, stdout, stderr } = runCli([
|
||||
'qualify', '--cwd', 'src/cli/__fixtures__/plugin-contracts',
|
||||
'--seed', '42', '--format', 'json-summary',
|
||||
]);
|
||||
|
||||
const parsed = JSON.parse(stdout);
|
||||
assert.strictEqual(parsed.command, 'qualify');
|
||||
assert.strictEqual(typeof parsed.summary.total, 'number');
|
||||
assert.strictEqual(typeof parsed.summary.passed, 'number');
|
||||
assert.strictEqual(typeof parsed.summary.failed, 'number');
|
||||
assert.ok(typeof parsed.executionSummary === 'object', 'qualify artifact must include execution summary');
|
||||
});
|
||||
|
||||
test('verify + replay round-trip with plugin contracts', () => {
|
||||
const artifactDir = '/tmp/apophis-replay-test-' + Date.now();
|
||||
const { status: verifyStatus } = runCli([
|
||||
'verify', '--cwd', 'src/cli/__fixtures__/plugin-contracts',
|
||||
'--profile', 'quick', '--seed', '42', '--artifact-dir', artifactDir,
|
||||
'--format', 'human',
|
||||
]);
|
||||
|
||||
const files = fs.readdirSync(artifactDir);
|
||||
const artifactFile = files.find((f) => f.endsWith('.json'));
|
||||
assert.ok(artifactFile, 'artifact file should exist');
|
||||
const artifactPath = path.join(artifactDir, artifactFile);
|
||||
|
||||
const { status: replayStatus, stdout: replayOut } = runCli([
|
||||
'replay', '--artifact', artifactPath,
|
||||
]);
|
||||
|
||||
fs.rmSync(artifactDir, { recursive: true, force: true });
|
||||
|
||||
assert.strictEqual(verifyStatus, 0, 'verify should exit 0');
|
||||
// Replay exits 2 when artifact has no failures (nothing to reproduce).
|
||||
// This proves the round-trip: artifact written, artifact loaded successfully.
|
||||
assert.strictEqual(replayStatus, 2, 'replay exits 2 for no-failure artifact');
|
||||
assert.ok(replayOut.includes('no failures to replay'), 'replay reports no failures');
|
||||
});
|
||||
|
||||
test('replay preserves seed from artifact', () => {
|
||||
const artifactDir = '/tmp/apophis-seed-test-' + Date.now();
|
||||
runCli([
|
||||
'verify', '--cwd', 'src/cli/__fixtures__/plugin-contracts',
|
||||
'--profile', 'quick', '--seed', '99', '--artifact-dir', artifactDir,
|
||||
'--format', 'human',
|
||||
]);
|
||||
|
||||
const files = fs.readdirSync(artifactDir);
|
||||
const artifactFile = files.find((f) => f.endsWith('.json'));
|
||||
assert.ok(artifactFile, 'artifact file should exist');
|
||||
const artifactPath = path.join(artifactDir, artifactFile);
|
||||
|
||||
const artifactContent = fs.readFileSync(artifactPath, 'utf8');
|
||||
const artifact = JSON.parse(artifactContent);
|
||||
assert.strictEqual(artifact.seed, 99, 'artifact must preserve seed 99');
|
||||
|
||||
const { status: replayStatus, stdout: replayOut } = runCli([
|
||||
'replay', '--artifact', artifactPath,
|
||||
]);
|
||||
|
||||
fs.rmSync(artifactDir, { recursive: true, force: true });
|
||||
|
||||
assert.strictEqual(replayStatus, 2, 'replay exits 2 for no-failure artifact');
|
||||
assert.ok(replayOut.includes('no failures to replay'), 'replay loads artifact with seed 99');
|
||||
});
|
||||
|
||||
@@ -222,6 +222,38 @@ test('parse: F shorthand for false', () => {
|
||||
const result = parse('F')
|
||||
assert.deepStrictEqual(result.ast, { type: 'literal', value: false })
|
||||
})
|
||||
test('parse: F == true parses as comparison', () => {
|
||||
const result = parse('F == true')
|
||||
assert.deepStrictEqual(result.ast, {
|
||||
type: 'comparison', op: '==',
|
||||
left: { type: 'literal', value: false },
|
||||
right: { type: 'literal', value: true },
|
||||
})
|
||||
})
|
||||
test('parse: T == false parses as comparison', () => {
|
||||
const result = parse('T == false')
|
||||
assert.deepStrictEqual(result.ast, {
|
||||
type: 'comparison', op: '==',
|
||||
left: { type: 'literal', value: true },
|
||||
right: { type: 'literal', value: false },
|
||||
})
|
||||
})
|
||||
test('parse: F != true parses as comparison', () => {
|
||||
const result = parse('F != true')
|
||||
assert.deepStrictEqual(result.ast, {
|
||||
type: 'comparison', op: '!=',
|
||||
left: { type: 'literal', value: false },
|
||||
right: { type: 'literal', value: true },
|
||||
})
|
||||
})
|
||||
test('parse: F < T parses as comparison', () => {
|
||||
const result = parse('F < T')
|
||||
assert.deepStrictEqual(result.ast, {
|
||||
type: 'comparison', op: '<',
|
||||
left: { type: 'literal', value: false },
|
||||
right: { type: 'literal', value: true },
|
||||
})
|
||||
})
|
||||
test('parse: throws on empty formula', () => {
|
||||
assert.throws(() => parse(''), /Empty formula/)
|
||||
})
|
||||
|
||||
@@ -7,6 +7,7 @@ import assert from 'node:assert'
|
||||
import Fastify from 'fastify'
|
||||
import type { FastifyInstance } from 'fastify'
|
||||
import apophisPlugin from '../index.js'
|
||||
import { createFastify } from '../fastify-factory.js'
|
||||
import { runPetitTests } from '../quality/petit-runner.js'
|
||||
import { CleanupManager } from '../infrastructure/cleanup-manager.js'
|
||||
import { ScopeRegistry } from '../infrastructure/scope-registry.js'
|
||||
@@ -924,3 +925,198 @@ test('observe: violation event emitted but response still succeeds in observe mo
|
||||
await fastify.close()
|
||||
}
|
||||
})
|
||||
|
||||
test('observe: production mode still emits non-blocking sink events', async () => {
|
||||
const originalEnv = process.env.NODE_ENV
|
||||
process.env.NODE_ENV = 'production'
|
||||
const fastify = Fastify()
|
||||
try {
|
||||
await fastify.register(swagger, {})
|
||||
const emitted: ObserveEvent[] = []
|
||||
const sink: ObserveSink = { emit: (event) => { emitted.push(event) } }
|
||||
await fastify.register(apophisPlugin, {
|
||||
runtime: 'off',
|
||||
observe: { enabled: true, sinks: [sink], sampling: 1 },
|
||||
})
|
||||
fastify.get('/prod-observe', {
|
||||
schema: {
|
||||
'x-category': 'observer',
|
||||
'x-ensures': ['status:201'],
|
||||
} as Record<string, unknown>
|
||||
}, async () => ({ ok: true }))
|
||||
await fastify.ready()
|
||||
const response = await fastify.inject({ method: 'GET', url: '/prod-observe' })
|
||||
assert.strictEqual(response.statusCode, 200, 'production observe must not block responses')
|
||||
const violationEvents = emitted.filter(e => e.type === 'contract.violation')
|
||||
assert.ok(violationEvents.length > 0, 'production observe should emit contract violations')
|
||||
assert.strictEqual(violationEvents[0]!.formula, 'status:201')
|
||||
} finally {
|
||||
await fastify.close()
|
||||
if (originalEnv === undefined) delete process.env.NODE_ENV
|
||||
else process.env.NODE_ENV = originalEnv
|
||||
}
|
||||
})
|
||||
|
||||
test('createFastify: observe options activate sink-backed runtime observation', async () => {
|
||||
const emitted: ObserveEvent[] = []
|
||||
const sink: ObserveSink = { emit: (event) => { emitted.push(event) } }
|
||||
const fastify = await createFastify({
|
||||
apophis: {
|
||||
observe: { enabled: true, sinks: [sink], sampling: 1 },
|
||||
},
|
||||
})
|
||||
try {
|
||||
fastify.get('/factory-observe', {
|
||||
schema: {
|
||||
'x-category': 'observer',
|
||||
'x-ensures': ['status:200'],
|
||||
} as Record<string, unknown>
|
||||
}, async () => ({ ok: true }))
|
||||
await fastify.ready()
|
||||
const response = await fastify.inject({ method: 'GET', url: '/factory-observe' })
|
||||
assert.strictEqual(response.statusCode, 200)
|
||||
const passEvents = emitted.filter(e => e.type === 'contract.pass')
|
||||
assert.ok(passEvents.length > 0, 'createFastify observe options should wire runtime sink events')
|
||||
} finally {
|
||||
await fastify.close()
|
||||
}
|
||||
})
|
||||
|
||||
test('createFastify: runtime off alone keeps APOPHIS plugin inactive', async () => {
|
||||
const fastify = await createFastify({
|
||||
apophis: { runtime: 'off' },
|
||||
})
|
||||
try {
|
||||
assert.strictEqual(fastify.hasDecorator('apophis'), false)
|
||||
} finally {
|
||||
await fastify.close()
|
||||
}
|
||||
})
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Plugin composition: plugin contracts merged into route contracts
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
test('plugin composition: contracts merge into route and source is tracked on violation', async () => {
|
||||
const fastify = Fastify() as unknown as TestFastifyInstance
|
||||
try {
|
||||
await fastify.register(swagger, {})
|
||||
await fastify.register(apophisPlugin, {
|
||||
pluginContracts: {
|
||||
'auth-plugin': {
|
||||
appliesTo: '/api/**',
|
||||
hooks: {
|
||||
preHandler: { ensures: ['1 == 2'] },
|
||||
},
|
||||
meta: { name: 'auth-plugin' },
|
||||
},
|
||||
},
|
||||
})
|
||||
fastify.get('/api/resource', {
|
||||
schema: {
|
||||
'x-category': 'observer',
|
||||
response: { 200: { type: 'object' } },
|
||||
} as Record<string, unknown>,
|
||||
}, async () => ({ ok: true }))
|
||||
await fastify.ready()
|
||||
const suite = await fastify.apophis.contract({ runs: 5 })
|
||||
assert.ok(suite.summary.pluginContractsApplied! > 0, 'plugin contracts should be counted as applied')
|
||||
assert.ok(suite.summary.pluginContractsFailed! > 0, 'always-false ensures should cause at least one plugin contract failure')
|
||||
} finally {
|
||||
await fastify.close()
|
||||
}
|
||||
})
|
||||
|
||||
test('plugin composition: missing required extension skips the plugin silently', async () => {
|
||||
const fastify = Fastify() as unknown as TestFastifyInstance
|
||||
try {
|
||||
await fastify.register(swagger, {})
|
||||
await fastify.register(apophisPlugin, {
|
||||
pluginContracts: {
|
||||
'requires-missing-ext': {
|
||||
appliesTo: '/**',
|
||||
extensions: [{ name: 'nonexistent', required: true }],
|
||||
hooks: {
|
||||
preHandler: { ensures: ['1 == 2'] },
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
fastify.get('/plain', {
|
||||
schema: {
|
||||
'x-category': 'observer',
|
||||
'x-ensures': ['status:200'],
|
||||
response: { 200: { type: 'object' } },
|
||||
} as Record<string, unknown>,
|
||||
}, async () => ({ ok: true }))
|
||||
await fastify.ready()
|
||||
const suite = await fastify.apophis.contract({ runs: 5 })
|
||||
assert.strictEqual(suite.summary.failed, 0, 'route contract should pass without the skipped plugin formula')
|
||||
assert.strictEqual(
|
||||
suite.summary.pluginContractsApplied ?? 0,
|
||||
1,
|
||||
'only the route-level formula counts when plugin is skipped',
|
||||
)
|
||||
assert.strictEqual(
|
||||
suite.summary.pluginContractsFailed ?? 0,
|
||||
0,
|
||||
'no plugin failures when plugin is skipped',
|
||||
)
|
||||
} finally {
|
||||
await fastify.close()
|
||||
}
|
||||
})
|
||||
|
||||
test('plugin composition: formula from plugin with extension passes when condition met', async () => {
|
||||
const fastify = Fastify() as unknown as TestFastifyInstance
|
||||
try {
|
||||
await fastify.register(swagger, {})
|
||||
await fastify.register(apophisPlugin, {
|
||||
extensions: [
|
||||
{
|
||||
name: 'tenant-gate',
|
||||
headers: ['check_tenant'],
|
||||
predicates: {
|
||||
check_tenant: (ctx: { evalContext: { request: { headers: Record<string, string> } } }) => ({
|
||||
value: ctx.evalContext.request.headers['x-tenant'] === 'acme',
|
||||
success: true,
|
||||
}),
|
||||
},
|
||||
} as import('../extension/types.js').ApophisExtension,
|
||||
],
|
||||
pluginContracts: {
|
||||
'tenant-validator': {
|
||||
appliesTo: '/**',
|
||||
extensions: [{ name: 'tenant-gate', required: true }],
|
||||
hooks: {
|
||||
onRequest: { requires: ['check_tenant(this) == true'] },
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
fastify.get('/tenanted', {
|
||||
schema: {
|
||||
'x-category': 'observer',
|
||||
'x-ensures': ['status:200'],
|
||||
response: { 200: { type: 'object' } },
|
||||
} as Record<string, unknown>,
|
||||
}, async () => ({ ok: true }))
|
||||
await fastify.ready()
|
||||
const suite = await fastify.apophis.contract({
|
||||
runs: 5,
|
||||
variants: [{ name: 'acme', headers: { 'x-tenant': 'acme' } }],
|
||||
})
|
||||
assert.ok(suite.summary.pluginContractsApplied! > 0, 'tenant validator should be applied')
|
||||
assert.strictEqual(suite.summary.pluginContractsFailed ?? 0, 0, 'tenant check should pass for acme')
|
||||
assert.strictEqual(suite.summary.failed, 0, 'all tests should pass')
|
||||
} finally {
|
||||
await fastify.close()
|
||||
}
|
||||
})
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Plugin composition + scenarios (integration)
|
||||
// ---------------------------------------------------------------------------
|
||||
// NOTE: scenario + plugin contract integration test resides in
|
||||
// _smoke-scenario-plugin.ts for isolation from cross-test state.
|
||||
// The programmatic path is verified by running that smoke script directly.
|
||||
|
||||
@@ -45,6 +45,8 @@ export interface RouteContract {
|
||||
outbound?: readonly OutboundBinding[]
|
||||
/** Route-level variants for negotiated content-type or feature testing. Extracted from schema['x-variants']. */
|
||||
variants?: RouteVariant[]
|
||||
/** Formula → source mapping for plugin contract attribution. Populated at test time. */
|
||||
formulaSources?: Record<string, 'route' | `plugin:${string}`>
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
|
||||
Reference in New Issue
Block a user