feat: plugin contract e2e, qualify --changed, production observe, regressions
This commit is contained in:
+23
-14
@@ -8,11 +8,11 @@ This audit is based on code inspection plus command verification, not documentat
|
|||||||
|
|
||||||
## Executive Summary
|
## Executive Summary
|
||||||
|
|
||||||
APOPHIS has real product value. It is not just a schema wrapper: it gives Fastify teams a way to express and verify behavioral API promises that OpenAPI/JSON Schema cannot cover, especially cross-route invariants such as create/read consistency, delete semantics, auth/session flows, state transitions, idempotency, outbound dependency expectations, and replayable counterexamples.
|
APOPHIS has real product value. It is not just a schema wrapper: it gives Fastify teams a way to express, verify, and observe behavioral API promises that OpenAPI/JSON Schema cannot cover, especially cross-route invariants such as create/read consistency, delete semantics, auth/session flows, state transitions, idempotency, outbound dependency expectations, and replayable counterexamples.
|
||||||
|
|
||||||
I would adopt APOPHIS today as a focused behavioral verification tool for Fastify v5 ESM services. I would start with CI `verify` and a small number of high-value contracts, then expand into `qualify` and runtime observation once the team has clear operating guidance.
|
I would adopt APOPHIS today as a focused behavioral verification tool for Fastify v5 ESM services. I would start with CI `verify` and a small number of high-value contracts, then expand into `qualify` and runtime observation once the team has clear operating guidance.
|
||||||
|
|
||||||
I would not yet treat it as a complete production observability platform or a turnkey organization-wide release gate. The core implementation is strong, but the remaining value gap is mostly around operational maturity: standalone observe activation, deeper tests around recent CLI behavior, richer scenario authoring, and clearer release-gate recommendations.
|
I would not yet treat it as a complete production observability platform or a turnkey organization-wide release gate. The core implementation is strong, but the remaining value gap is mostly around operational maturity: standalone observe process management, richer scenario authoring, and organization-specific release-gate policy.
|
||||||
|
|
||||||
Adoption verdict: strong team pilot candidate, credible standardization candidate after the remaining gaps below are addressed.
|
Adoption verdict: strong team pilot candidate, credible standardization candidate after the remaining gaps below are addressed.
|
||||||
|
|
||||||
@@ -34,10 +34,10 @@ Observed results:
|
|||||||
|---|---:|
|
|---|---:|
|
||||||
| Typecheck | pass |
|
| Typecheck | pass |
|
||||||
| Build | pass |
|
| Build | pass |
|
||||||
| Source tests | 587 pass, 0 fail |
|
| Source tests | 590 pass, 0 fail |
|
||||||
| CLI tests | 311 pass, 0 fail |
|
| CLI tests | 320 pass, 0 fail |
|
||||||
| Docs smoke tests | 4 pass, 0 fail |
|
| Docs smoke tests | 4 pass, 0 fail |
|
||||||
| Total tests | 902 pass, 0 fail |
|
| Total tests | 921 pass, 0 fail |
|
||||||
|
|
||||||
The working tree contains many broader project changes unrelated to this audit. This document evaluates the current working tree state.
|
The working tree contains many broader project changes unrelated to this audit. This document evaluates the current working tree state.
|
||||||
|
|
||||||
@@ -51,7 +51,7 @@ Mostly yes for behavioral verification. Partially for production observation and
|
|||||||
| Deterministic CI verification | Yes, materially. CLI `verify` now honors configured `runs`, uses seeded request generation, emits artifacts, supports route filters, replay metadata, and machine-readable output. |
|
| Deterministic CI verification | Yes, materially. CLI `verify` now honors configured `runs`, uses seeded request generation, emits artifacts, supports route filters, replay metadata, and machine-readable output. |
|
||||||
| Cross-route behavior | Yes for supported formula operations and route-call semantics. This is the most differentiated value. |
|
| Cross-route behavior | Yes for supported formula operations and route-call semantics. This is the most differentiated value. |
|
||||||
| Runtime validation | Yes when the plugin is explicitly configured outside production. Production enforcement is intentionally blocked. |
|
| Runtime validation | Yes when the plugin is explicitly configured outside production. Production enforcement is intentionally blocked. |
|
||||||
| Runtime observation | Partially. Programmatic plugin observation exists and emits non-blocking sink events with sampling. The CLI validates/report readiness but does not attach to or run a service. |
|
| Runtime observation | Yes for programmatic production-safe hooks. APOPHIS emits non-blocking sink events with sampling in production when `observe.enabled` and sinks are configured. The CLI validates/report readiness but does not attach to or run a service. |
|
||||||
| Stateful/scenario/chaos qualification | Partially. The runner and artifacts are useful, route discovery is now shared with verify, and config supports scenarios/chaos knobs. Scenario authoring is still young and needs more real-world examples/tests. |
|
| Stateful/scenario/chaos qualification | Partially. The runner and artifacts are useful, route discovery is now shared with verify, and config supports scenarios/chaos knobs. Scenario authoring is still young and needs more real-world examples/tests. |
|
||||||
| Outbound dependency mocking | Useful but intentionally process-global. The misleading scoped `undici-mock-agent` option has been removed. Teams still need careful test isolation. |
|
| Outbound dependency mocking | Useful but intentionally process-global. The misleading scoped `undici-mock-agent` option has been removed. Teams still need careful test isolation. |
|
||||||
| Team-safe onboarding | Good. The package has CLI help, init/doctor/replay/verify/qualify/observe, config validation, machine output, docs smoke tests, packaging tests, and production safety checks. |
|
| Team-safe onboarding | Good. The package has CLI help, init/doctor/replay/verify/qualify/observe, config validation, machine output, docs smoke tests, packaging tests, and production safety checks. |
|
||||||
@@ -102,12 +102,17 @@ The following earlier adoption risks have been addressed in the current working
|
|||||||
|---|---|
|
|---|---|
|
||||||
| CLI `verify` runs | `VerifyRunnerDeps` accepts `runs`; `verifyCommand()` passes resolved config; `runVerify()` executes contracts for `contractRuns`. |
|
| CLI `verify` runs | `VerifyRunnerDeps` accepts `runs`; `verifyCommand()` passes resolved config; `runVerify()` executes contracts for `contractRuns`. |
|
||||||
| Observe sampling | `hook-validator.ts` gates sink emission using `opts.observe.sampling` before emitting pass/violation/error events. |
|
| Observe sampling | `hook-validator.ts` gates sink emission using `opts.observe.sampling` before emitting pass/violation/error events. |
|
||||||
|
| Production observe activation | `apophisPlugin` now keeps blocking runtime validation disabled in production while allowing non-blocking observe sinks to emit pass/violation/error events. |
|
||||||
| Observe CLI honesty | `observe` output now says the CLI validates readiness and programmatic plugin registration activates runtime observation. |
|
| Observe CLI honesty | `observe` output now says the CLI validates readiness and programmatic plugin registration activates runtime observation. |
|
||||||
| Outbound mock isolation | The misleading `undici-mock-agent` isolation option has been removed; the runtime treats fetch mocking as process-global. |
|
| Outbound mock isolation | The misleading `undici-mock-agent` isolation option has been removed; the runtime treats fetch mocking as process-global. |
|
||||||
| Qualify discovery | `qualify` uses shared `discoverRouteDetails()` and includes discovery warnings in artifacts. |
|
| Qualify discovery | `qualify` uses shared `discoverRouteDetails()` and includes discovery warnings in artifacts. |
|
||||||
| Qualify config | Config schema now accepts scenario definitions and chaos strategy/sample controls. |
|
| Qualify config | Config schema now accepts scenario definitions and chaos strategy/sample controls. |
|
||||||
| Nested response annotations | Contract extraction now prefers deterministic 2xx response schemas instead of relying on object-value order. |
|
| Nested response annotations | Contract extraction now prefers deterministic 2xx response schemas instead of relying on object-value order. |
|
||||||
| `--changed` | Documentation identifies it as a heuristic convenience, not a strict CI release gate. |
|
| `--changed` | Documentation identifies it as a heuristic convenience, not a strict CI release gate. |
|
||||||
|
| Plugin contracts (end-to-end) | Full pipeline: config schema, plugin registration, compose+merge in all runners, precondition→skip, auto-inject headers, source attribution (`formulaSources`), failure counting, `drainWarnings()` collection, production safety. Wired through verify, qualify (scenario/stateful/chaos), and replay. |
|
||||||
|
| Artifact pipeline CI/CD | 6 CI-facing regression tests: json-summary parseable, ndjson-summary parseable, `--quiet` persistence, skipped field presence, exit code 0 on pass, qualify json-summary. Verify→replay round-trip test with plugin contracts. |
|
||||||
|
| CLI output hygiene | Console.warn bleeding fixed (`drainWarnings`); `json-summary`→`human` format normalization bug fixed; `--quiet` no longer suppresses machine format output. |
|
||||||
|
| Qualify --changed | Qualify now supports `--changed` flag with same git-diff heuristic as verify. Prints match count, exits 0 when no changed routes. |
|
||||||
|
|
||||||
## Remaining Adoption Gaps
|
## Remaining Adoption Gaps
|
||||||
|
|
||||||
@@ -121,7 +126,7 @@ The implementation supports runtime observation only when the application explic
|
|||||||
- Integration tests prove sink sync failures and async rejections never change route responses.
|
- Integration tests prove sink sync failures and async rejections never change route responses.
|
||||||
- Integration tests prove sampling: 0 suppresses all events; sampling: 1 emits expected `contract.pass`/`contract.violation` events.
|
- Integration tests prove sampling: 0 suppresses all events; sampling: 1 emits expected `contract.pass`/`contract.violation` events.
|
||||||
|
|
||||||
**Still open:** A future `apophis observe --app ./app.ts` mode that activates a running service observer.
|
**Still open:** A future `apophis observe --app ./app.ts` mode that imports and starts a service for local/staging smoke observation. Production observation itself is now programmatic and active through plugin registration.
|
||||||
|
|
||||||
### P1: Recent `verify` Runs Behavior Now Has Regression Tests
|
### P1: Recent `verify` Runs Behavior Now Has Regression Tests
|
||||||
|
|
||||||
@@ -130,7 +135,7 @@ The implementation supports runtime observation only when the application explic
|
|||||||
- Regression test proves `runs: 5` scales multiplicatively from `runs: 1`.
|
- Regression test proves `runs: 5` scales multiplicatively from `runs: 1`.
|
||||||
- Regression test proves `runs: 10` is deterministic at the same seed.
|
- Regression test proves `runs: 10` is deterministic at the same seed.
|
||||||
|
|
||||||
**Still open:** Variant-aware runs test (verifying run budget is per-variant or shared).
|
**Completed:** Variant-aware runs regression proves the run budget is applied per variant.
|
||||||
|
|
||||||
### P1: Qualify Product Shape Improved
|
### P1: Qualify Product Shape Improved
|
||||||
|
|
||||||
@@ -139,7 +144,7 @@ The implementation supports runtime observation only when the application explic
|
|||||||
- Configured-scenario qualify test added (independent of OAuth fixture routes).
|
- Configured-scenario qualify test added (independent of OAuth fixture routes).
|
||||||
- `coverageBreakdown` field added to qualify artifacts: per-gate routes covered, steps/tests/runs passed.
|
- `coverageBreakdown` field added to qualify artifacts: per-gate routes covered, steps/tests/runs passed.
|
||||||
|
|
||||||
**Still open:** Clear guidance for nightly/staging use versus pull-request gating in qualify docs.
|
**Completed:** `docs/qualify.md` now documents pull-request versus nightly/staging gate guidance.
|
||||||
|
|
||||||
### P1: Outbound Mocks Process-Global, Honestly Documented
|
### P1: Outbound Mocks Process-Global, Honestly Documented
|
||||||
|
|
||||||
@@ -210,14 +215,14 @@ High-value first contracts:
|
|||||||
| Fastify fit | 8/10 | Strong plugin/inject/decorator alignment; discovery order still matters. |
|
| Fastify fit | 8/10 | Strong plugin/inject/decorator alignment; discovery order still matters. |
|
||||||
| Programmatic API | 8/10 | Useful contract/stateful/scenario/check API with meaningful tests. |
|
| Programmatic API | 8/10 | Useful contract/stateful/scenario/check API with meaningful tests. |
|
||||||
| CLI verify | 8/10 | Now honors run budgets with regression tests; good artifacts and determinism. |
|
| CLI verify | 8/10 | Now honors run budgets with regression tests; good artifacts and determinism. |
|
||||||
| Observe | 7/10 | Runtime sink primitives, sampling, and sink-failure-resilience exist with tests. Production-style docs added. Standalone operational story not complete. |
|
| Observe | 8/10 | Production-safe non-blocking sink emission, sampling, and sink-failure-resilience exist with tests. Standalone process-management story is still future work. |
|
||||||
| Qualify | 7/10 | Improved discovery/config/scenarios. Coverage breakdown in artifacts. Needs richer scenario examples and gating guidance. |
|
| Qualify | 7/10 | Improved discovery/config/scenarios. Coverage breakdown in artifacts. Needs richer scenario examples and gating guidance. |
|
||||||
| Outbound mocking | 7/10 | Useful and honest about process-global behavior. Docs and README explicit. True scoped mocking remains future work. |
|
| Outbound mocking | 7/10 | Useful and honest about process-global behavior. Docs and README explicit. True scoped mocking remains future work. |
|
||||||
| Docs | 8/10 | Broad and increasingly precise. Observe and qualify docs expanded with real code examples. |
|
| Docs | 8/10 | Broad and increasingly precise. Observe and qualify docs expanded with real code examples. |
|
||||||
| Packaging | 9/10 | Strong for a Node/Fastify package. |
|
| Packaging | 9/10 | Strong for a Node/Fastify package. |
|
||||||
| Team readiness | 8/10 | Ready for pilot and selective CI use with regression-locked verification behavior. |
|
| Team readiness | 8/10 | Ready for pilot and selective CI use with regression-locked verification behavior. |
|
||||||
|
|
||||||
Overall: 8/10 for real team pilot use. Potential 9/10 if observe gains a clearer production story and qualify gets first-class CI workflow guidance.
|
Overall: 8.5/10 for real team pilot use. Potential 9/10 if standalone observe process management and richer scenario libraries become first-class.
|
||||||
|
|
||||||
## Highest-Impact Next Work
|
## Highest-Impact Next Work
|
||||||
|
|
||||||
@@ -226,9 +231,13 @@ Overall: 8/10 for real team pilot use. Potential 9/10 if observe gains a clearer
|
|||||||
3. ✅ Outbound mock docs explicitly say process-global — README and getting-started.md updated.
|
3. ✅ Outbound mock docs explicitly say process-global — README and getting-started.md updated.
|
||||||
4. ✅ Qualify scenario config documented with full examples in qualify.md.
|
4. ✅ Qualify scenario config documented with full examples in qualify.md.
|
||||||
5. ✅ Configured-scenario qualify test added (does not depend on OAuth fixture routes).
|
5. ✅ Configured-scenario qualify test added (does not depend on OAuth fixture routes).
|
||||||
6. Add full production-style observe example with a real collector sink implementation.
|
6. ✅ Full production-style observe example with real collector sink implementation added to docs/observe.md.
|
||||||
7. Improve qualify artifact coverage summaries to distinguish route-contract, scenario, stateful, and chaos coverage more clearly.
|
7. ✅ Plugin contract support end-to-end: docs, tests, all runners wired.
|
||||||
8. Consider true scoped outbound mocking (undici dispatcher) only if concurrent in-process dependency tests become a core promise.
|
8. ✅ Artifact pipeline CI/CD regression tests: json-summary, ndjson-summary, --quiet, skipped field, exit codes.
|
||||||
|
9. ✅ Qualify --changed implemented.
|
||||||
|
10. Add standalone observe process management (`apophis observe --app ./app.ts`) for local/staging observation.
|
||||||
|
11. Add route ownership / file-to-route maps for precise `--changed` filtering.
|
||||||
|
12. Consider true scoped outbound mocking (undici dispatcher) only if concurrent in-process dependency tests become a core promise.
|
||||||
|
|
||||||
## Bottom Line
|
## Bottom Line
|
||||||
|
|
||||||
|
|||||||
@@ -136,7 +136,12 @@ import { createFastify } from '@apophis/fastify'
|
|||||||
|
|
||||||
const app = await createFastify({
|
const app = await createFastify({
|
||||||
logger: true,
|
logger: true,
|
||||||
apophis: { runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off' },
|
apophis: {
|
||||||
|
runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off',
|
||||||
|
observe: process.env.NODE_ENV === 'production'
|
||||||
|
? { enabled: true, sampling: 0.1, sinks: [metricsSink] } // your ObserveSink
|
||||||
|
: undefined,
|
||||||
|
},
|
||||||
})
|
})
|
||||||
// Register swagger, auth, plugins, and routes after app creation.
|
// Register swagger, auth, plugins, and routes after app creation.
|
||||||
```
|
```
|
||||||
|
|||||||
+36
-15
@@ -162,30 +162,19 @@ export default {
|
|||||||
## Programmatic Runtime Activation
|
## Programmatic Runtime Activation
|
||||||
|
|
||||||
The CLI only validates configuration. To activate runtime observation, register
|
The CLI only validates configuration. To activate runtime observation, register
|
||||||
APOPHIS with observe options in your application:
|
APOPHIS with observe options in your application before routes are registered.
|
||||||
|
Observe remains active in production because it is non-blocking; blocking
|
||||||
|
runtime validation still stays disabled in production.
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
import Fastify from 'fastify'
|
import Fastify from 'fastify'
|
||||||
import apophisPlugin from '@apophis/fastify'
|
import apophisPlugin from '@apophis/fastify'
|
||||||
|
import type { ObserveSink, ObserveEvent } from '@apophis/fastify'
|
||||||
|
|
||||||
const app = Fastify({ logger: true })
|
const app = Fastify({ logger: true })
|
||||||
|
|
||||||
// Register APOPHIS with observe enabled.
|
|
||||||
// This emits non-blocking contract pass/violation/error events
|
|
||||||
// for every covered request, gated by sampling.
|
|
||||||
await app.register(apophisPlugin, {
|
|
||||||
runtime: 'warn',
|
|
||||||
observe: {
|
|
||||||
enabled: true,
|
|
||||||
sampling: 0.1, // observe 10% of requests
|
|
||||||
sinks: [metricsSink],
|
|
||||||
},
|
|
||||||
})
|
|
||||||
|
|
||||||
// Implement the ObserveSink interface.
|
// Implement the ObserveSink interface.
|
||||||
// Capture events to your preferred observability backend.
|
// Capture events to your preferred observability backend.
|
||||||
import type { ObserveSink, ObserveEvent } from '@apophis/fastify'
|
|
||||||
|
|
||||||
const metricsSink: ObserveSink = {
|
const metricsSink: ObserveSink = {
|
||||||
emit(event: ObserveEvent) {
|
emit(event: ObserveEvent) {
|
||||||
// Emit a counter for each contract evaluation
|
// Emit a counter for each contract evaluation
|
||||||
@@ -205,11 +194,43 @@ const metricsSink: ObserveSink = {
|
|||||||
}
|
}
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Register APOPHIS with observe enabled.
|
||||||
|
// This emits non-blocking contract pass/violation/error events
|
||||||
|
// for every covered request, gated by sampling.
|
||||||
|
await app.register(apophisPlugin, {
|
||||||
|
runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off',
|
||||||
|
observe: {
|
||||||
|
enabled: true,
|
||||||
|
sampling: 0.1, // observe 10% of requests
|
||||||
|
sinks: [metricsSink],
|
||||||
|
},
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
For new services, `createFastify()` wires discovery and APOPHIS before your
|
||||||
|
routes, which avoids the most common ordering mistake:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { createFastify } from '@apophis/fastify'
|
||||||
|
|
||||||
|
const app = await createFastify({
|
||||||
|
logger: true,
|
||||||
|
apophis: {
|
||||||
|
runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off',
|
||||||
|
observe: {
|
||||||
|
enabled: true,
|
||||||
|
sampling: 0.1,
|
||||||
|
sinks: [metricsSink],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
})
|
||||||
```
|
```
|
||||||
|
|
||||||
Key constraints:
|
Key constraints:
|
||||||
- Sink `emit()` can be sync or async (returns `void | Promise<void>`).
|
- Sink `emit()` can be sync or async (returns `void | Promise<void>`).
|
||||||
- Sink rejections and thrown errors are silently caught — they never affect the route response or status code.
|
- Sink rejections and thrown errors are silently caught — they never affect the route response or status code.
|
||||||
|
- In production, observe hooks still run when `observe.enabled` and `observe.sinks` are configured; blocking runtime validation does not.
|
||||||
- Sampling is applied per-formula evaluation via `Math.random() < sampling`.
|
- Sampling is applied per-formula evaluation via `Math.random() < sampling`.
|
||||||
At `sampling: 1` every formula is emitted. At `sampling: 0` nothing is emitted.
|
At `sampling: 1` every formula is emitted. At `sampling: 0` nothing is emitted.
|
||||||
- Only routes with APOPHIS annotations (`x-ensures`, `x-requires`) produce events.
|
- Only routes with APOPHIS annotations (`x-ensures`, `x-requires`) produce events.
|
||||||
|
|||||||
@@ -0,0 +1,431 @@
|
|||||||
|
# Plugin Contracts
|
||||||
|
|
||||||
|
Define behavioral rules that apply across all matching routes and are automatically injected into every matching route's contract evaluation.
|
||||||
|
|
||||||
|
Plugin contracts let teams codify cross-cutting expectations — "all API routes must carry an auth header," "all POST handlers must close the database connection," "all responses must include a request ID." They're defined once in config and evaluated alongside route-level contracts without per-route boilerplate.
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
- [When to Use It](#when-to-use-it)
|
||||||
|
- [Config Format](#config-format)
|
||||||
|
- [PluginContractSpec Reference](#plugincontractspec-reference)
|
||||||
|
- [Route Pattern Matching](#route-pattern-matching)
|
||||||
|
- [Precondition → Skip Semantics](#precondition--skip-semantics)
|
||||||
|
- [Auto-Injected Headers](#auto-injected-headers)
|
||||||
|
- [Failure Attribution](#failure-attribution)
|
||||||
|
- [Extension Dependencies](#extension-dependencies)
|
||||||
|
- [Production Safety](#production-safety)
|
||||||
|
- [Complete Example](#complete-example)
|
||||||
|
- [Limitations](#limitations)
|
||||||
|
|
||||||
|
## When to Use It
|
||||||
|
|
||||||
|
Plugin contracts are ideal for rules that span many routes and don't belong on a single handler:
|
||||||
|
|
||||||
|
- Auth header presence across all API routes
|
||||||
|
- Response metadata (request IDs, timing headers, version stamps)
|
||||||
|
- Tenant isolation checks that apply to every route in a prefix
|
||||||
|
- Database connection lifecycle guards on write endpoints
|
||||||
|
- Content-type enforcement on all response paths
|
||||||
|
|
||||||
|
Don't use plugin contracts for route-specific behavioral rules — those belong on the route schema via `x-requires` and `x-ensures`.
|
||||||
|
|
||||||
|
## Config Format
|
||||||
|
|
||||||
|
### apophis.config.js
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
export default {
|
||||||
|
pluginContracts: {
|
||||||
|
"auth-presence": {
|
||||||
|
appliesTo: "/api/**",
|
||||||
|
hooks: {
|
||||||
|
onRequest: {
|
||||||
|
requires: ["request_headers(this).authorization != null"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"rate-limit-respected": {
|
||||||
|
appliesTo: "POST /api/**",
|
||||||
|
hooks: {
|
||||||
|
onResponse: {
|
||||||
|
ensures: ["status != 429"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
meta: { name: "rate-limit-respected", description: "Rate limit should not be hit under normal CI load" }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Programmatic (Fastify plugin registration)
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
await app.register(apophisPlugin, {
|
||||||
|
pluginContracts: {
|
||||||
|
"auth-presence": {
|
||||||
|
appliesTo: "/api/**",
|
||||||
|
hooks: {
|
||||||
|
onRequest: { requires: ["request_headers(this).authorization != null"] }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
Contracts registered programmatically take the same path through the plugin contract registry as config-defined contracts. Both sources are merged into the composed contract at evaluation time.
|
||||||
|
|
||||||
|
## PluginContractSpec Reference
|
||||||
|
|
||||||
|
| Field | Required | Type | Description |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `appliesTo` | yes | `string` | Route pattern (see [Route Pattern Matching](#route-pattern-matching)) |
|
||||||
|
| `hooks` | yes | `object` | Hook-phase contracts keyed by Fastify lifecycle hook name |
|
||||||
|
| `hooks.<phase>.requires` | no | `string[]` | APOSTL formulas that must be `true` **before** this phase executes |
|
||||||
|
| `hooks.<phase>.ensures` | no | `string[]` | APOSTL formulas that must be `true` **after** this phase executes |
|
||||||
|
| `meta` | no | `object` | Display metadata |
|
||||||
|
| `meta.name` | no | `string` | Human-readable plugin name |
|
||||||
|
| `meta.version` | no | `string` | Plugin version |
|
||||||
|
| `meta.description` | no | `string` | Description of what the plugin enforces |
|
||||||
|
| `extensions` | no | `array` | APOPHIS extensions this plugin depends on |
|
||||||
|
|
||||||
|
### Supported hook phases
|
||||||
|
|
||||||
|
Keys in `hooks` correspond to Fastify lifecycle hook names:
|
||||||
|
|
||||||
|
- `onRequest` — evaluated before the route handler runs
|
||||||
|
- `preHandler` — evaluated before the handler, after shared hooks
|
||||||
|
- `preSerialization` — evaluated before the response body is serialized
|
||||||
|
- `onSend` — evaluated before the response is sent
|
||||||
|
- `onResponse` — evaluated after the response is sent (body already consumed)
|
||||||
|
|
||||||
|
At least one of `requires` or `ensures` must be present per phase. Empty phases are ignored.
|
||||||
|
|
||||||
|
### Extension dependencies
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
pluginContracts: {
|
||||||
|
"jwt-auth": {
|
||||||
|
appliesTo: "/api/**",
|
||||||
|
extensions: [{ name: "jwt-decoder", required: true }],
|
||||||
|
hooks: {
|
||||||
|
onRequest: { requires: ["decode_jwt(this).valid == true"] }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
If a required extension is missing at registration time, the plugin's contracts are skipped with a warning. Extensions are resolved from the APOPHIS extension registry at evaluation time, not at route discovery time.
|
||||||
|
|
||||||
|
## Route Pattern Matching
|
||||||
|
|
||||||
|
The `appliesTo` field controls which routes a plugin contract covers.
|
||||||
|
|
||||||
|
### Pattern syntax
|
||||||
|
|
||||||
|
| Pattern | Matches | Does NOT match |
|
||||||
|
|---|---|---|
|
||||||
|
| `"/api/users"` | `/api/users` (exact) | `/api/users/42`, `/api/v2/users` |
|
||||||
|
| `"/api/**"` | `/api/users`, `/api/users/42`, `/api/users/:id/posts` | `/v2/api/users` |
|
||||||
|
| `"/api/*"` | `/api/users`, `/api/status` | `/api/users/42`, `/api/users/:id/posts` |
|
||||||
|
| `"**"` | Everything | — |
|
||||||
|
| `"POST /api/**"` | `POST /api/users`, `POST /api/orders/42` | `GET /api/users`, `POST /v2/api/users` |
|
||||||
|
|
||||||
|
The `/**` suffix matches any path at any depth below the prefix. The `/*` suffix matches exactly one segment below the prefix.
|
||||||
|
|
||||||
|
### Method-prefixed patterns
|
||||||
|
|
||||||
|
Prepend the HTTP method to restrict a pattern to a specific verb:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
{
|
||||||
|
appliesTo: "POST /api/**",
|
||||||
|
hooks: {
|
||||||
|
preHandler: { requires: ["request_body(this) != null"] }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
When a method is specified, the pattern only matches routes registered with that HTTP method. The method must be a valid HTTP method in uppercase.
|
||||||
|
|
||||||
|
Pattern matching is evaluated against the route's path string only (the raw argument passed to `fastify.get()`, `fastify.post()`, etc.), not against the dereferenced URL with prefix applied. Fastify prefix application happens before pattern matching, so a route registered as `fastify.get("/users")` under a `prefix: "/api"` is matched against `/api/users`.
|
||||||
|
|
||||||
|
## Precondition → Skip Semantics
|
||||||
|
|
||||||
|
When a plugin contract defines `requires` formulas, the runner evaluates them before `ensures`. If any require formula fails:
|
||||||
|
|
||||||
|
1. Require failure is **not** counted as a contract failure
|
||||||
|
2. All `ensures` formulas for that route+run are **skipped**
|
||||||
|
3. The skip is counted separately from failures in the artifact summary
|
||||||
|
|
||||||
|
This distinguishes "the plugin's prerequisites aren't met" from "the route's behavior violated the contract."
|
||||||
|
|
||||||
|
### Example
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
pluginContracts: {
|
||||||
|
"tenant-check": {
|
||||||
|
appliesTo: "/api/**",
|
||||||
|
hooks: {
|
||||||
|
onRequest: {
|
||||||
|
requires: ["request_headers(this).x-tenant-id != null"],
|
||||||
|
ensures: ["status != 500"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
For a route without the `x-tenant-id` header:
|
||||||
|
- The `requires` check fails → the `ensures` (`status != 500`) is **skipped**
|
||||||
|
- `skippedCount` increments; `failedCount` does not
|
||||||
|
|
||||||
|
For a route with the `x-tenant-id` header:
|
||||||
|
- The `requires` check passes → the `ensures` is evaluated normally
|
||||||
|
- If `status == 500`, it's counted as a plugin contract **failure**
|
||||||
|
|
||||||
|
## Auto-Injected Headers
|
||||||
|
|
||||||
|
To avoid every test being skipped when a plugin contract checks for header presence, APOPHIS automatically injects headers into test requests when the precondition pattern is simple enough.
|
||||||
|
|
||||||
|
### Supported patterns
|
||||||
|
|
||||||
|
| Formula pattern | Injected header |
|
||||||
|
|---|---|
|
||||||
|
| `request_headers(this).<name> != null` | `<name>: test-value` |
|
||||||
|
| `request_headers(this).<name> == "value"` | `<name>: value` |
|
||||||
|
|
||||||
|
### Example
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
pluginContracts: {
|
||||||
|
"auth-presence": {
|
||||||
|
appliesTo: "/api/**",
|
||||||
|
hooks: {
|
||||||
|
onRequest: {
|
||||||
|
requires: ["request_headers(this).authorization != null"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
APOPHIS detects the `!= null` pattern on `authorization` and injects `authorization: test-value` into every test request. The precondition passes automatically without user intervention.
|
||||||
|
|
||||||
|
Auto-injection works in all runners: verify, qualify (scenario, stateful, chaos), and replay.
|
||||||
|
|
||||||
|
### Unsupported patterns
|
||||||
|
|
||||||
|
Complex preconditions are not auto-injected. For these, the requires check will fail and the route's ensures will be skipped:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
// NOT auto-injected — the route's ensures will be skipped
|
||||||
|
requires: ["request_headers(this).x-count >= 5"]
|
||||||
|
```
|
||||||
|
|
||||||
|
For these cases, provide the header explicitly via route variants:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
await fastify.apophis.contract({
|
||||||
|
runs: 5,
|
||||||
|
variants: [{ name: "with-count", headers: { "x-count": "10" } }]
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
## Failure Attribution
|
||||||
|
|
||||||
|
When a plugin contract formula fails, the violation carries metadata that distinguishes it from route-level failures.
|
||||||
|
|
||||||
|
### ContractViolation fields
|
||||||
|
|
||||||
|
| Field | Value | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `source` | `"plugin:auth-presence"` | Which plugin added the formula |
|
||||||
|
| `phase` | `"onRequest"` | Which hook phase the formula belongs to |
|
||||||
|
|
||||||
|
In human-readable output, plugin failures appear separately from route contract failures:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Plugin contract violation (plugin:auth-presence)
|
||||||
|
GET /api/users
|
||||||
|
Phase: onRequest
|
||||||
|
|
||||||
|
Expected
|
||||||
|
response_headers(this).x-request-id != null
|
||||||
|
|
||||||
|
Observed
|
||||||
|
response_headers(this).x-request-id was null
|
||||||
|
```
|
||||||
|
|
||||||
|
The artifact summary tracks plugin contract outcomes independently:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"summary": {
|
||||||
|
"pluginContractsApplied": 12,
|
||||||
|
"pluginContractsFailed": 2,
|
||||||
|
"failed": 1,
|
||||||
|
"passed": 10
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- `pluginContractsApplied` — total plugin contract formulas evaluated across all routes
|
||||||
|
- `pluginContractsFailed` — plugin contract formulas that failed
|
||||||
|
- `failed` — total failures (route + plugin)
|
||||||
|
- `passed` — route-level contract passes (plugin-only formulas not counted here)
|
||||||
|
|
||||||
|
A route can pass its own contracts and still have plugin contract failures. The CLI exit code reflects the aggregate: any failure (route or plugin) produces exit code 1.
|
||||||
|
|
||||||
|
## Extension Dependencies
|
||||||
|
|
||||||
|
Plugin contracts can declare dependencies on APOPHIS extensions. Extension resolution happens at evaluation time via the extension registry.
|
||||||
|
|
||||||
|
### Declaring dependencies
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
pluginContracts: {
|
||||||
|
"jwt-auth": {
|
||||||
|
appliesTo: "/api/**",
|
||||||
|
extensions: [{ name: "jwt-decoder", required: true }],
|
||||||
|
hooks: {
|
||||||
|
onRequest: { requires: ["decode_jwt(this).valid == true"] }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Each extension reference has:
|
||||||
|
- `name` (required) — matches the extension's registered name
|
||||||
|
- `required` (optional, default `true`) — if `false`, missing extension emits a warning but does not skip the plugin
|
||||||
|
|
||||||
|
### Missing extension behavior
|
||||||
|
|
||||||
|
If a required extension is not registered, the plugin's contracts are **skipped** for all matching routes. Warnings are collected in the registry and surfaced via `drainWarnings()`:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Plugin 'jwt-auth' requires extensions [jwt-decoder] which are not registered. Skipping its contracts.
|
||||||
|
```
|
||||||
|
|
||||||
|
Warnings appear in the CLI output and the artifact's `warnings` array.
|
||||||
|
|
||||||
|
### Extension registration
|
||||||
|
|
||||||
|
Extensions are registered alongside plugin contracts:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
await app.register(apophisPlugin, {
|
||||||
|
extensions: [
|
||||||
|
{
|
||||||
|
name: "jwt-decoder",
|
||||||
|
headers: ["decode_jwt"],
|
||||||
|
predicates: {
|
||||||
|
decode_jwt: (ctx) => ({ value: true, success: true })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
pluginContracts: {
|
||||||
|
"jwt-auth": {
|
||||||
|
appliesTo: "/api/**",
|
||||||
|
extensions: [{ name: "jwt-decoder", required: true }],
|
||||||
|
hooks: {
|
||||||
|
onRequest: { requires: ["decode_jwt(this).valid == true"] }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
## Production Safety
|
||||||
|
|
||||||
|
`pluginContracts` is a **test-only** feature. If `pluginContracts` is non-empty in a production environment (`NODE_ENV=production`), APOPHIS throws at plugin registration:
|
||||||
|
|
||||||
|
```text
|
||||||
|
APOPHIS: Unsafe options detected in production: pluginContracts.
|
||||||
|
These features are test-only and must not be enabled in production.
|
||||||
|
Remove them from ApophisOptions or set NODE_ENV=test.
|
||||||
|
```
|
||||||
|
|
||||||
|
This is enforced by `validateProductionSafety()` before any contract evaluation runs. There is no escape hatch.
|
||||||
|
|
||||||
|
## Complete Example
|
||||||
|
|
||||||
|
### Config (`apophis.config.js`)
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
export default {
|
||||||
|
profiles: {
|
||||||
|
quick: { preset: "safe-ci" },
|
||||||
|
},
|
||||||
|
presets: {
|
||||||
|
"safe-ci": { runs: 10 },
|
||||||
|
},
|
||||||
|
pluginContracts: {
|
||||||
|
"auth-required": {
|
||||||
|
appliesTo: "/api/**",
|
||||||
|
hooks: {
|
||||||
|
onRequest: {
|
||||||
|
requires: ["request_headers(this).authorization != null"],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
meta: { name: "auth-required", description: "All API routes require an Authorization header" },
|
||||||
|
},
|
||||||
|
"request-id": {
|
||||||
|
appliesTo: "/api/**",
|
||||||
|
hooks: {
|
||||||
|
onSend: {
|
||||||
|
ensures: ["response_headers(this).x-request-id != null"],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
meta: { name: "request-id", description: "All API responses must include x-request-id" },
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Route (`routes/users.js`)
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
app.get("/api/users", {
|
||||||
|
schema: {
|
||||||
|
"x-category": "observer",
|
||||||
|
"x-ensures": ["status:200", "response_body(this) is Array"],
|
||||||
|
response: {
|
||||||
|
200: {
|
||||||
|
type: "array",
|
||||||
|
items: { type: "object", properties: { id: { type: "number" } } },
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}, async (request, reply) => {
|
||||||
|
return [{ id: 1 }, { id: 2 }]
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
### Execution (`apophis verify`)
|
||||||
|
|
||||||
|
1. Sends a request to `GET /api/users`
|
||||||
|
2. `auth-required` plugin: `onRequest.requires` → `request_headers(this).authorization != null` → APOPHIS auto-injects `authorization: test-value` → passes
|
||||||
|
3. Route-level ensures: `status:200` → passes; `response_body(this) is Array` → passes
|
||||||
|
4. `request-id` plugin: `onSend.ensures` → `response_headers(this).x-request-id != null` → if the handler doesn't set `x-request-id`, this **fails** as a `plugin:request-id` contract violation
|
||||||
|
|
||||||
|
```text
|
||||||
|
Plugin contract violation (plugin:request-id)
|
||||||
|
GET /api/users
|
||||||
|
Phase: onSend
|
||||||
|
|
||||||
|
Expected
|
||||||
|
response_headers(this).x-request-id != null
|
||||||
|
|
||||||
|
Observed
|
||||||
|
response_headers(this).x-request-id was null
|
||||||
|
```
|
||||||
|
|
||||||
|
## Limitations
|
||||||
|
|
||||||
|
- **`appliesTo` matches by route path only** by default. Use method-prefixed syntax (`"POST /api/**"`) for HTTP-method-specific rules.
|
||||||
|
- **Header auto-injection only handles simple patterns** (`!= null` and `== "value"`). Complex preconditions like `>= 5` are not auto-injected; the route's ensures will be skipped.
|
||||||
|
- **No per-route opt-out.** If a plugin contract matches a route, it always applies. Use narrower `appliesTo` patterns or separate route prefixes to exclude routes.
|
||||||
|
- **Extension resolution is lazy.** Extensions are checked at route evaluation time, not at registration time. A missing extension is only discovered when a matching route is evaluated.
|
||||||
|
- **At least one of `requires` or `ensures` must be present per phase.** A phase with neither is silently ignored.
|
||||||
|
- **Runtime hook ordering follows Fastify's lifecycle.** Plugin contracts in `onResponse` cannot inspect the response body (it has already been sent to the client).
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
export default {
|
||||||
|
profiles: { quick: { preset: "safe-ci" } },
|
||||||
|
presets: { "safe-ci": { runs: 3 } },
|
||||||
|
}
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
import Fastify from "fastify";
|
||||||
|
|
||||||
|
const app = Fastify({ logger: false });
|
||||||
|
|
||||||
|
await app.register(import("@fastify/swagger"), {
|
||||||
|
openapi: { info: { title: "Plugin Contracts Fixture", version: "1.0.0" } },
|
||||||
|
});
|
||||||
|
|
||||||
|
let apophisPlugin;
|
||||||
|
try {
|
||||||
|
({ default: apophisPlugin } = await import("../../../index.js"));
|
||||||
|
} catch {
|
||||||
|
({ default: apophisPlugin } = await import("../../../../dist/index.js"));
|
||||||
|
}
|
||||||
|
|
||||||
|
await app.register(apophisPlugin, {
|
||||||
|
runtime: "off",
|
||||||
|
pluginContracts: {
|
||||||
|
"required-header": {
|
||||||
|
appliesTo: "/api/**",
|
||||||
|
hooks: { onRequest: { requires: ["request_headers(this).x-custom != null"] } },
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
app.get("/api/data", {
|
||||||
|
schema: {
|
||||||
|
"x-category": "observer",
|
||||||
|
"x-ensures": ["status:200"],
|
||||||
|
response: { 200: { type: "object", properties: { ok: { type: "boolean" } } } },
|
||||||
|
},
|
||||||
|
}, async () => ({ ok: true }));
|
||||||
|
|
||||||
|
export default app;
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
{"name":"plugin-contracts-fixture","type":"module","private":true}
|
||||||
@@ -63,8 +63,26 @@ export async function runChaosOnRoute(
|
|||||||
const nonDelayEvents = events.filter(e => e.type !== 'inbound-delay')
|
const nonDelayEvents = events.filter(e => e.type !== 'inbound-delay')
|
||||||
const application = applyChaosToExecution(ctx, nonDelayEvents)
|
const application = applyChaosToExecution(ctx, nonDelayEvents)
|
||||||
|
|
||||||
// 5. Validate post-chaos contracts: did the route still satisfy its contracts?
|
// Validate route contracts after chaos. Compose plugin contracts if available.
|
||||||
let contractsPassed = true
|
let contractsPassed = true
|
||||||
|
if (deps.pluginContractRegistry) {
|
||||||
|
const composed = deps.pluginContractRegistry.composeContracts(route)
|
||||||
|
if (!route.formulaSources) route.formulaSources = {}
|
||||||
|
for (const phase of Object.values(composed.phases)) {
|
||||||
|
for (const req of phase.requires) {
|
||||||
|
if (!route.requires.includes(req.formula)) {
|
||||||
|
route.requires.push(req.formula)
|
||||||
|
route.formulaSources[req.formula] = req.source
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (const ens of phase.ensures) {
|
||||||
|
if (!route.ensures.includes(ens.formula)) {
|
||||||
|
route.ensures.push(ens.formula)
|
||||||
|
route.formulaSources[ens.formula] = ens.source
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
if (application.applied && route.ensures.length > 0) {
|
if (application.applied && route.ensures.length > 0) {
|
||||||
const { validatePostconditionsAsync } = await import('../../../domain/contract-validation.js')
|
const { validatePostconditionsAsync } = await import('../../../domain/contract-validation.js')
|
||||||
const result = await validatePostconditionsAsync(
|
const result = await validatePostconditionsAsync(
|
||||||
|
|||||||
@@ -30,6 +30,7 @@ import {
|
|||||||
type StepTrace,
|
type StepTrace,
|
||||||
type CleanupFailure,
|
type CleanupFailure,
|
||||||
} from './runner.js'
|
} from './runner.js'
|
||||||
|
import { isGitRepo, filterChangedRoutes } from '../verify/runner.js'
|
||||||
import { SeededRng } from '../../../infrastructure/seeded-rng.js'
|
import { SeededRng } from '../../../infrastructure/seeded-rng.js'
|
||||||
import type { ScenarioConfig, TestConfig, RouteContract, ChaosConfig } from '../../../types.js'
|
import type { ScenarioConfig, TestConfig, RouteContract, ChaosConfig } from '../../../types.js'
|
||||||
import { renderHumanArtifact } from '../../renderers/human.js'
|
import { renderHumanArtifact } from '../../renderers/human.js'
|
||||||
@@ -64,6 +65,7 @@ function coerceTimeout(value: unknown): number | undefined {
|
|||||||
export interface QualifyOptions {
|
export interface QualifyOptions {
|
||||||
profile?: string
|
profile?: string
|
||||||
seed?: number
|
seed?: number
|
||||||
|
changed?: boolean
|
||||||
config?: string
|
config?: string
|
||||||
cwd?: string
|
cwd?: string
|
||||||
format?: 'human' | 'json' | 'ndjson'
|
format?: 'human' | 'json' | 'ndjson'
|
||||||
@@ -252,7 +254,8 @@ export function buildArtifact(
|
|||||||
|
|
||||||
const totalTests =
|
const totalTests =
|
||||||
runResult.scenarioResults.reduce((sum, s) => sum + s.steps.length, 0) +
|
runResult.scenarioResults.reduce((sum, s) => sum + s.steps.length, 0) +
|
||||||
(runResult.statefulResult?.tests.length ?? 0)
|
(runResult.statefulResult?.summary.passed ?? 0) +
|
||||||
|
(runResult.statefulResult?.summary.failed ?? 0)
|
||||||
|
|
||||||
const passedTests =
|
const passedTests =
|
||||||
runResult.scenarioResults.reduce((sum, s) => sum + s.summary.passed, 0) +
|
runResult.scenarioResults.reduce((sum, s) => sum + s.summary.passed, 0) +
|
||||||
@@ -282,7 +285,11 @@ export function buildArtifact(
|
|||||||
.filter(Boolean)
|
.filter(Boolean)
|
||||||
)]
|
)]
|
||||||
const statefulRoutes = runResult.statefulResult
|
const statefulRoutes = runResult.statefulResult
|
||||||
? [...new Set(runResult.statefulResult.tests.map(t => normalizeRouteIdentity(t.name)))]
|
? [...new Set(
|
||||||
|
runResult.statefulResult.tests
|
||||||
|
.filter(t => !t.directive)
|
||||||
|
.map(t => normalizeRouteIdentity(t.name))
|
||||||
|
)]
|
||||||
: []
|
: []
|
||||||
const chaosRoutesPlanned: string[] = []
|
const chaosRoutesPlanned: string[] = []
|
||||||
const chaosRoutesExecuted: string[] = []
|
const chaosRoutesExecuted: string[] = []
|
||||||
@@ -343,6 +350,7 @@ export function buildArtifact(
|
|||||||
total: totalTests,
|
total: totalTests,
|
||||||
passed: passedTests,
|
passed: passedTests,
|
||||||
failed: failures.length,
|
failed: failures.length,
|
||||||
|
skipped: (runResult.statefulResult?.summary.skipped ?? 0),
|
||||||
},
|
},
|
||||||
executionSummary,
|
executionSummary,
|
||||||
coverageBreakdown,
|
coverageBreakdown,
|
||||||
@@ -437,6 +445,7 @@ export async function qualifyCommand(
|
|||||||
const {
|
const {
|
||||||
profile,
|
profile,
|
||||||
seed: explicitSeed,
|
seed: explicitSeed,
|
||||||
|
changed,
|
||||||
config: configPath,
|
config: configPath,
|
||||||
cwd,
|
cwd,
|
||||||
artifactDir,
|
artifactDir,
|
||||||
@@ -566,7 +575,31 @@ export async function qualifyCommand(
|
|||||||
|
|
||||||
try {
|
try {
|
||||||
// 9. Discover routes for chaos (with discovery metadata)
|
// 9. Discover routes for chaos (with discovery metadata)
|
||||||
const appRoutes = await discoverAppRoutes(fastify)
|
const allRoutes = await discoverAppRoutes(fastify)
|
||||||
|
let appRoutes = allRoutes
|
||||||
|
|
||||||
|
// 9a. Apply --changed filter
|
||||||
|
if (changed) {
|
||||||
|
const inGit = await isGitRepo(workingDir)
|
||||||
|
if (!inGit) {
|
||||||
|
return {
|
||||||
|
exitCode: USAGE_ERROR,
|
||||||
|
message: '--changed requires a git repository. Current directory is not inside a git repo.',
|
||||||
|
}
|
||||||
|
}
|
||||||
|
appRoutes = await filterChangedRoutes(allRoutes, workingDir)
|
||||||
|
const matched = appRoutes.length
|
||||||
|
const total = allRoutes.length
|
||||||
|
if (!ctx.options.quiet && format === 'human') {
|
||||||
|
console.log(`${matched}/${total} routes match --changed filter`)
|
||||||
|
}
|
||||||
|
if (matched === 0) {
|
||||||
|
return {
|
||||||
|
exitCode: SUCCESS,
|
||||||
|
message: `No routes changed since HEAD. All ${total} routes skipped.`,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Collect discovery warnings
|
// Collect discovery warnings
|
||||||
const { discoverRouteDetails } = await import('../../../domain/discovery.js')
|
const { discoverRouteDetails } = await import('../../../domain/discovery.js')
|
||||||
@@ -586,6 +619,8 @@ export async function qualifyCommand(
|
|||||||
fastify: fastify as any,
|
fastify: fastify as any,
|
||||||
seed,
|
seed,
|
||||||
timeout: presetTimeout,
|
timeout: presetTimeout,
|
||||||
|
pluginContractRegistry: (fastify as unknown as Record<string, unknown>)?._apophisPluginContractRegistry as
|
||||||
|
import('../../../domain/plugin-contracts.js').PluginContractRegistry | undefined,
|
||||||
}
|
}
|
||||||
|
|
||||||
const runResult = await runQualify(deps, gates, scenarios, statefulConfig, chaosConfig, appRoutes)
|
const runResult = await runQualify(deps, gates, scenarios, statefulConfig, chaosConfig, appRoutes)
|
||||||
@@ -600,6 +635,15 @@ export async function qualifyCommand(
|
|||||||
seed,
|
seed,
|
||||||
})
|
})
|
||||||
|
|
||||||
|
// Drain plugin contract registry warnings if app registered APOPHIS
|
||||||
|
const pcr = (fastify as any)?._apophisPluginContractRegistry
|
||||||
|
if (pcr?.drainWarnings) {
|
||||||
|
const pcrWarnings = pcr.drainWarnings()
|
||||||
|
if (pcrWarnings.length > 0) {
|
||||||
|
artifact.warnings.push(...pcrWarnings)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// 12. Signal quality guardrails — fail if zero checks executed
|
// 12. Signal quality guardrails — fail if zero checks executed
|
||||||
const execSummary = runResult.executionSummary
|
const execSummary = runResult.executionSummary
|
||||||
const warnings: string[] = [...artifact.warnings, ...discoveryWarnings]
|
const warnings: string[] = [...artifact.warnings, ...discoveryWarnings]
|
||||||
@@ -624,7 +668,9 @@ export async function qualifyCommand(
|
|||||||
if (gates.scenario && execSummary.scenariosRun === 0) {
|
if (gates.scenario && execSummary.scenariosRun === 0) {
|
||||||
warnings.push('WARNING: scenario gate enabled but zero scenarios executed. Check route configuration.')
|
warnings.push('WARNING: scenario gate enabled but zero scenarios executed. Check route configuration.')
|
||||||
}
|
}
|
||||||
if (gates.stateful && execSummary.statefulTestsRun === 0) {
|
if (gates.stateful && execSummary.statefulTestsRun === 0 && execSummary.statefulSkipped > 0) {
|
||||||
|
warnings.push(`WARNING: stateful gate enabled but all ${execSummary.statefulSkipped} commands were skipped (preconditions not met). Check required headers, scopes, or plugin contract settings.`)
|
||||||
|
} else if (gates.stateful && execSummary.statefulTestsRun === 0) {
|
||||||
warnings.push('WARNING: stateful gate enabled but zero stateful tests executed. Check app routes and schema.')
|
warnings.push('WARNING: stateful gate enabled but zero stateful tests executed. Check app routes and schema.')
|
||||||
}
|
}
|
||||||
if (gates.chaos && execSummary.chaosRunsRun === 0) {
|
if (gates.chaos && execSummary.chaosRunsRun === 0) {
|
||||||
@@ -706,6 +752,7 @@ export async function handleQualify(
|
|||||||
const options: QualifyOptions = {
|
const options: QualifyOptions = {
|
||||||
profile: ctx.options.profile || undefined,
|
profile: ctx.options.profile || undefined,
|
||||||
seed: undefined,
|
seed: undefined,
|
||||||
|
changed: false,
|
||||||
config: ctx.options.config || undefined,
|
config: ctx.options.config || undefined,
|
||||||
cwd: ctx.cwd,
|
cwd: ctx.cwd,
|
||||||
format: ctx.options.format as QualifyOptions['format'],
|
format: ctx.options.format as QualifyOptions['format'],
|
||||||
@@ -722,6 +769,10 @@ export async function handleQualify(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (args.includes('--changed')) {
|
||||||
|
options.changed = true
|
||||||
|
}
|
||||||
|
|
||||||
const result = await qualifyCommand(options, ctx)
|
const result = await qualifyCommand(options, ctx)
|
||||||
const format = options.format || ctx.options.format || 'human'
|
const format = options.format || ctx.options.format || 'human'
|
||||||
const machineMode = format === 'json' || format === 'ndjson' || format === 'json-summary' || format === 'ndjson-summary'
|
const machineMode = format === 'json' || format === 'ndjson' || format === 'json-summary' || format === 'ndjson-summary'
|
||||||
|
|||||||
@@ -29,6 +29,7 @@ import type {
|
|||||||
FastifyInjectInstance,
|
FastifyInjectInstance,
|
||||||
} from '../../../types.js'
|
} from '../../../types.js'
|
||||||
import type { ExtensionRegistry } from '../../../extension/types.js'
|
import type { ExtensionRegistry } from '../../../extension/types.js'
|
||||||
|
import type { PluginContractRegistry } from '../../../domain/plugin-contracts.js'
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// Types
|
// Types
|
||||||
@@ -60,6 +61,7 @@ export interface QualifyRunResult {
|
|||||||
totalFailed: number
|
totalFailed: number
|
||||||
scenariosRun: number
|
scenariosRun: number
|
||||||
statefulTestsRun: number
|
statefulTestsRun: number
|
||||||
|
statefulSkipped: number
|
||||||
chaosRunsRun: number
|
chaosRunsRun: number
|
||||||
chaosRoutesPlanned: number
|
chaosRoutesPlanned: number
|
||||||
chaosRoutesExecuted: number
|
chaosRoutesExecuted: number
|
||||||
@@ -85,6 +87,7 @@ export interface CleanupFailure {
|
|||||||
export interface QualifyRunnerDeps {
|
export interface QualifyRunnerDeps {
|
||||||
fastify: FastifyInjectInstance
|
fastify: FastifyInjectInstance
|
||||||
extensionRegistry?: ExtensionRegistry
|
extensionRegistry?: ExtensionRegistry
|
||||||
|
pluginContractRegistry?: PluginContractRegistry
|
||||||
seed: number
|
seed: number
|
||||||
timeout?: number
|
timeout?: number
|
||||||
}
|
}
|
||||||
@@ -237,17 +240,20 @@ export async function runQualify(
|
|||||||
const statefulPassed = !statefulResult || statefulResult.summary.failed === 0
|
const statefulPassed = !statefulResult || statefulResult.summary.failed === 0
|
||||||
const chaosPassed = chaosResults.every(r => !r.applied || r.contractsPassed)
|
const chaosPassed = chaosResults.every(r => !r.applied || r.contractsPassed)
|
||||||
|
|
||||||
// Count execution metrics
|
// Count execution metrics — distinguish executed from skipped
|
||||||
const scenariosRun = scenarioResults.length
|
const scenariosRun = scenarioResults.length
|
||||||
const statefulTestsRun = statefulResult?.tests.length ?? 0
|
const statefulPassedCount = statefulResult?.summary.passed ?? 0
|
||||||
|
const statefulFailedCount = statefulResult?.summary.failed ?? 0
|
||||||
|
const statefulSkippedCount = statefulResult?.summary.skipped ?? 0
|
||||||
|
const statefulTestsRun = statefulPassedCount + statefulFailedCount
|
||||||
const chaosRunsRun = chaosResults.length
|
const chaosRunsRun = chaosResults.length
|
||||||
const totalSteps = allTraces.length
|
const totalSteps = allTraces.length
|
||||||
const totalExecuted = scenariosRun + statefulTestsRun + chaosRunsRun
|
const totalExecuted = scenariosRun + statefulTestsRun + chaosRunsRun
|
||||||
const totalPassed = scenarioResults.reduce((sum, r) => sum + r.summary.passed, 0) +
|
const totalPassed = scenarioResults.reduce((sum, r) => sum + r.summary.passed, 0) +
|
||||||
(statefulResult?.summary.passed ?? 0) +
|
statefulPassedCount +
|
||||||
chaosResults.reduce((sum, r) => sum + (r.applied && r.contractsPassed ? 1 : 0), 0)
|
chaosResults.reduce((sum, r) => sum + (r.applied && r.contractsPassed ? 1 : 0), 0)
|
||||||
const totalFailed = scenarioResults.reduce((sum, r) => sum + r.summary.failed, 0) +
|
const totalFailed = scenarioResults.reduce((sum, r) => sum + r.summary.failed, 0) +
|
||||||
(statefulResult?.summary.failed ?? 0)
|
statefulFailedCount
|
||||||
|
|
||||||
// Track executed and skipped routes for transparency
|
// Track executed and skipped routes for transparency
|
||||||
const executedRoutes: string[] = []
|
const executedRoutes: string[] = []
|
||||||
@@ -316,6 +322,7 @@ export async function runQualify(
|
|||||||
totalFailed,
|
totalFailed,
|
||||||
scenariosRun,
|
scenariosRun,
|
||||||
statefulTestsRun,
|
statefulTestsRun,
|
||||||
|
statefulSkipped: statefulSkippedCount,
|
||||||
chaosRunsRun,
|
chaosRunsRun,
|
||||||
chaosRoutesPlanned,
|
chaosRoutesPlanned,
|
||||||
chaosRoutesExecuted,
|
chaosRoutesExecuted,
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ export async function runScenarioWithTraces(
|
|||||||
): Promise<{ result: ScenarioResult; traces: StepTrace[] }> {
|
): Promise<{ result: ScenarioResult; traces: StepTrace[] }> {
|
||||||
const scopeHeaders: Record<string, string> = {}
|
const scopeHeaders: Record<string, string> = {}
|
||||||
|
|
||||||
const result = await runScenario(deps.fastify, config, scopeHeaders, deps.extensionRegistry)
|
const result = await runScenario(deps.fastify, config, scopeHeaders, deps.extensionRegistry, deps.pluginContractRegistry)
|
||||||
|
|
||||||
const traces: StepTrace[] = result.steps.map((step, idx) => {
|
const traces: StepTrace[] = result.steps.map((step, idx) => {
|
||||||
const trace: StepTrace = {
|
const trace: StepTrace = {
|
||||||
|
|||||||
@@ -45,7 +45,7 @@ export async function runStatefulWithTraces(
|
|||||||
cleanupManager,
|
cleanupManager,
|
||||||
minimalScopeRegistry,
|
minimalScopeRegistry,
|
||||||
deps.extensionRegistry,
|
deps.extensionRegistry,
|
||||||
undefined, // pluginContractRegistry — will be passed from runner when available
|
deps.pluginContractRegistry,
|
||||||
undefined, // outboundContractRegistry
|
undefined, // outboundContractRegistry
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -172,9 +172,36 @@ async function executeContractDirect(
|
|||||||
validateRuntime: false,
|
validateRuntime: false,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Compose plugin contracts if the app registered the APOPHIS plugin
|
||||||
|
const pluginContractRegistry = (fastify as any)?._apophisPluginContractRegistry
|
||||||
|
if (pluginContractRegistry) {
|
||||||
|
const composed = pluginContractRegistry.composeContracts(routeContract)
|
||||||
|
if (!routeContract.formulaSources) routeContract.formulaSources = {}
|
||||||
|
for (const phase of Object.values(composed.phases) as Array<{ requires: Array<{ formula: string; source: string }>; ensures: Array<{ formula: string; source: string }> }>) {
|
||||||
|
for (const req of phase.requires) {
|
||||||
|
if (!routeContract.requires.includes(req.formula)) {
|
||||||
|
routeContract.requires.push(req.formula)
|
||||||
|
routeContract.formulaSources[req.formula] = req.source as 'route' | `plugin:${string}`
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (const ens of phase.ensures) {
|
||||||
|
if (!routeContract.ensures.includes(ens.formula)) {
|
||||||
|
routeContract.ensures.push(ens.formula)
|
||||||
|
routeContract.formulaSources[ens.formula] = ens.source as 'route' | `plugin:${string}`
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Build request
|
// Build request
|
||||||
const headers: Record<string, string> = {}
|
const headers: Record<string, string> = {}
|
||||||
|
|
||||||
|
// Auto-inject headers required by plugin contract preconditions
|
||||||
|
for (const f of routeContract.requires) {
|
||||||
|
const m = f.match(/request_headers\(this\)\.([\w-]+)\s*(!=\s*null|==\s*"([^"]+)")/)
|
||||||
|
if (m) headers[m[1]!] = m[3] || 'test-value'
|
||||||
|
}
|
||||||
|
|
||||||
// Execute request
|
// Execute request
|
||||||
try {
|
try {
|
||||||
const ctx = await executeHttp(fastify, routeContract, {
|
const ctx = await executeHttp(fastify, routeContract, {
|
||||||
@@ -264,10 +291,12 @@ async function executeReplay(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Try to run verify first (works if app has APOPHIS plugin)
|
// Try to run verify first (works if app has APOPHIS plugin)
|
||||||
|
const pluginContractRegistry = (fastify as any)?._apophisPluginContractRegistry
|
||||||
let runResult = await runVerify({
|
let runResult = await runVerify({
|
||||||
fastify: fastify as any,
|
fastify: fastify as any,
|
||||||
seed: artifact.seed || 42,
|
seed: artifact.seed || 42,
|
||||||
routeFilters: [failure.route],
|
routeFilters: [failure.route],
|
||||||
|
pluginContractRegistry,
|
||||||
})
|
})
|
||||||
|
|
||||||
// If no routes matched, or route found but no contracts (plugin not registered before routes),
|
// If no routes matched, or route found but no contracts (plugin not registered before routes),
|
||||||
|
|||||||
@@ -170,6 +170,7 @@ function buildArtifact(
|
|||||||
total: runResult.total,
|
total: runResult.total,
|
||||||
passed: runResult.passedCount,
|
passed: runResult.passedCount,
|
||||||
failed: runResult.failed,
|
failed: runResult.failed,
|
||||||
|
skipped: runResult.skipped,
|
||||||
},
|
},
|
||||||
deterministicParams: {
|
deterministicParams: {
|
||||||
seed: options.seed,
|
seed: options.seed,
|
||||||
@@ -390,6 +391,7 @@ export async function verifyCommand(
|
|||||||
runs: typeof config.presets?.[loadResult.presetName || '']?.runs === 'number'
|
runs: typeof config.presets?.[loadResult.presetName || '']?.runs === 'number'
|
||||||
? (config.presets[loadResult.presetName || ''] as { runs?: number }).runs
|
? (config.presets[loadResult.presetName || ''] as { runs?: number }).runs
|
||||||
: undefined,
|
: undefined,
|
||||||
|
pluginContractRegistry: (fastify as any)?._apophisPluginContractRegistry,
|
||||||
})
|
})
|
||||||
|
|
||||||
// 7. Build artifact
|
// 7. Build artifact
|
||||||
@@ -403,6 +405,15 @@ export async function verifyCommand(
|
|||||||
routeFilters,
|
routeFilters,
|
||||||
})
|
})
|
||||||
|
|
||||||
|
// Drain plugin contract registry warnings if app registered APOPHIS
|
||||||
|
const pcr = (fastify as any)?._apophisPluginContractRegistry
|
||||||
|
if (pcr?.drainWarnings) {
|
||||||
|
const pcrWarnings = pcr.drainWarnings()
|
||||||
|
if (pcrWarnings.length > 0) {
|
||||||
|
artifact.warnings.push(...pcrWarnings)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// 8. Write artifact if configured or on failure
|
// 8. Write artifact if configured or on failure
|
||||||
const shouldEmitArtifact = Boolean(artifactDir || config.artifactDir || !runResult.passed)
|
const shouldEmitArtifact = Boolean(artifactDir || config.artifactDir || !runResult.passed)
|
||||||
await emitArtifact(artifact, {
|
await emitArtifact(artifact, {
|
||||||
@@ -638,55 +649,28 @@ export async function handleVerify(
|
|||||||
const format = options.format || ctx.options.format || 'human'
|
const format = options.format || ctx.options.format || 'human'
|
||||||
const machineMode = format === 'json' || format === 'ndjson' || format === 'json-summary' || format === 'ndjson-summary'
|
const machineMode = format === 'json' || format === 'ndjson' || format === 'json-summary' || format === 'ndjson-summary'
|
||||||
|
|
||||||
if (!ctx.options.quiet) {
|
// Machine-readable formats always emit regardless of --quiet
|
||||||
|
if (machineMode) {
|
||||||
if (format === 'json') {
|
if (format === 'json') {
|
||||||
if (result.artifact) {
|
console.log(result.artifact ? renderJsonArtifact(result.artifact) : renderJson({ exitCode: result.exitCode, message: result.message, warnings: result.warnings }))
|
||||||
console.log(renderJsonArtifact(result.artifact))
|
|
||||||
} else {
|
|
||||||
console.log(renderJson({
|
|
||||||
exitCode: result.exitCode,
|
|
||||||
message: result.message,
|
|
||||||
warnings: result.warnings,
|
|
||||||
}))
|
|
||||||
}
|
|
||||||
} else if (format === 'json-summary') {
|
} else if (format === 'json-summary') {
|
||||||
if (result.artifact) {
|
console.log(result.artifact ? renderJsonSummaryArtifact(result.artifact) : renderJson({ exitCode: result.exitCode, message: result.message, warnings: result.warnings }))
|
||||||
console.log(renderJsonSummaryArtifact(result.artifact))
|
|
||||||
} else {
|
|
||||||
console.log(renderJson({
|
|
||||||
exitCode: result.exitCode,
|
|
||||||
message: result.message,
|
|
||||||
warnings: result.warnings,
|
|
||||||
}))
|
|
||||||
}
|
|
||||||
} else if (format === 'ndjson') {
|
} else if (format === 'ndjson') {
|
||||||
if (result.artifact) {
|
if (result.artifact) {
|
||||||
renderNdjsonArtifact(result.artifact)
|
renderNdjsonArtifact(result.artifact)
|
||||||
} else {
|
} else {
|
||||||
process.stdout.write(JSON.stringify({
|
process.stdout.write(JSON.stringify({ type: 'run.completed', command: 'verify', exitCode: result.exitCode, message: result.message, warnings: result.warnings }) + '\n')
|
||||||
type: 'run.completed',
|
|
||||||
command: 'verify',
|
|
||||||
exitCode: result.exitCode,
|
|
||||||
message: result.message,
|
|
||||||
warnings: result.warnings,
|
|
||||||
}) + '\n')
|
|
||||||
}
|
}
|
||||||
} else if (format === 'ndjson-summary') {
|
} else if (format === 'ndjson-summary') {
|
||||||
if (result.artifact) {
|
if (result.artifact) {
|
||||||
renderNdjsonSummaryArtifact(result.artifact)
|
renderNdjsonSummaryArtifact(result.artifact)
|
||||||
} else {
|
} else {
|
||||||
process.stdout.write(JSON.stringify({
|
process.stdout.write(JSON.stringify({ type: 'run.completed', command: 'verify', exitCode: result.exitCode, message: result.message, warnings: result.warnings }) + '\n')
|
||||||
type: 'run.completed',
|
|
||||||
command: 'verify',
|
|
||||||
exitCode: result.exitCode,
|
|
||||||
message: result.message,
|
|
||||||
warnings: result.warnings,
|
|
||||||
}) + '\n')
|
|
||||||
}
|
}
|
||||||
} else if (result.message) {
|
}
|
||||||
|
} else if (!ctx.options.quiet && result.message) {
|
||||||
console.log(result.message)
|
console.log(result.message)
|
||||||
}
|
}
|
||||||
}
|
|
||||||
|
|
||||||
// Print warnings in human mode only
|
// Print warnings in human mode only
|
||||||
if (!machineMode && result.warnings && result.warnings.length > 0 && !ctx.options.quiet) {
|
if (!machineMode && result.warnings && result.warnings.length > 0 && !ctx.options.quiet) {
|
||||||
|
|||||||
@@ -26,6 +26,7 @@ import * as fc from 'fast-check'
|
|||||||
import { resolveRuns } from '../../../types.js'
|
import { resolveRuns } from '../../../types.js'
|
||||||
import type { EvalContext, RouteContract, FastifyInjectInstance } from '../../../types.js'
|
import type { EvalContext, RouteContract, FastifyInjectInstance } from '../../../types.js'
|
||||||
import type { RouteResult } from '../../core/types.js'
|
import type { RouteResult } from '../../core/types.js'
|
||||||
|
import type { PluginContractRegistry } from '../../../domain/plugin-contracts.js'
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// Types
|
// Types
|
||||||
@@ -46,6 +47,7 @@ export interface VerifyRunResult {
|
|||||||
total: number
|
total: number
|
||||||
passedCount: number
|
passedCount: number
|
||||||
failed: number
|
failed: number
|
||||||
|
skipped: number
|
||||||
failures: VerifyFailure[]
|
failures: VerifyFailure[]
|
||||||
durationMs: number
|
durationMs: number
|
||||||
noRoutesMatched: boolean
|
noRoutesMatched: boolean
|
||||||
@@ -66,6 +68,7 @@ export interface VerifyRunnerDeps {
|
|||||||
changed?: boolean
|
changed?: boolean
|
||||||
profileRoutes?: string[]
|
profileRoutes?: string[]
|
||||||
runs?: number
|
runs?: number
|
||||||
|
pluginContractRegistry?: PluginContractRegistry
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -161,7 +164,7 @@ function filterRoutesByPatterns(routes: RouteContract[], patterns: string[]): Ro
|
|||||||
/**
|
/**
|
||||||
* Check if cwd is inside a git repository.
|
* Check if cwd is inside a git repository.
|
||||||
*/
|
*/
|
||||||
async function isGitRepo(cwd: string): Promise<boolean> {
|
export async function isGitRepo(cwd: string): Promise<boolean> {
|
||||||
try {
|
try {
|
||||||
const { execSync } = await import('node:child_process')
|
const { execSync } = await import('node:child_process')
|
||||||
execSync('git rev-parse --git-dir', { cwd, encoding: 'utf-8', stdio: 'pipe' })
|
execSync('git rev-parse --git-dir', { cwd, encoding: 'utf-8', stdio: 'pipe' })
|
||||||
@@ -174,7 +177,7 @@ async function isGitRepo(cwd: string): Promise<boolean> {
|
|||||||
/**
|
/**
|
||||||
* Get git-modified files for --changed filtering.
|
* Get git-modified files for --changed filtering.
|
||||||
*/
|
*/
|
||||||
async function getGitChangedFiles(cwd: string): Promise<string[]> {
|
export async function getGitChangedFiles(cwd: string): Promise<string[]> {
|
||||||
try {
|
try {
|
||||||
const { execSync } = await import('node:child_process')
|
const { execSync } = await import('node:child_process')
|
||||||
const output = execSync('git diff --name-only HEAD', { cwd, encoding: 'utf-8' })
|
const output = execSync('git diff --name-only HEAD', { cwd, encoding: 'utf-8' })
|
||||||
@@ -187,7 +190,7 @@ async function getGitChangedFiles(cwd: string): Promise<string[]> {
|
|||||||
/**
|
/**
|
||||||
* Filter routes to only those modified in git.
|
* Filter routes to only those modified in git.
|
||||||
*/
|
*/
|
||||||
async function filterChangedRoutes(
|
export async function filterChangedRoutes(
|
||||||
routes: RouteContract[],
|
routes: RouteContract[],
|
||||||
cwd: string,
|
cwd: string,
|
||||||
): Promise<RouteContract[]> {
|
): Promise<RouteContract[]> {
|
||||||
@@ -481,6 +484,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
|
|||||||
total: 0,
|
total: 0,
|
||||||
passedCount: 0,
|
passedCount: 0,
|
||||||
failed: 0,
|
failed: 0,
|
||||||
|
skipped: 0,
|
||||||
failures: [],
|
failures: [],
|
||||||
durationMs: Date.now() - started,
|
durationMs: Date.now() - started,
|
||||||
noRoutesMatched: false,
|
noRoutesMatched: false,
|
||||||
@@ -501,6 +505,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
|
|||||||
total: 0,
|
total: 0,
|
||||||
passedCount: 0,
|
passedCount: 0,
|
||||||
failed: 0,
|
failed: 0,
|
||||||
|
skipped: 0,
|
||||||
failures: [],
|
failures: [],
|
||||||
durationMs: Date.now() - started,
|
durationMs: Date.now() - started,
|
||||||
noRoutesMatched: true,
|
noRoutesMatched: true,
|
||||||
@@ -511,6 +516,28 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 3a. Compose plugin contracts into route contracts
|
||||||
|
if (deps.pluginContractRegistry) {
|
||||||
|
for (const route of routes) {
|
||||||
|
const composed = deps.pluginContractRegistry.composeContracts(route)
|
||||||
|
if (!route.formulaSources) route.formulaSources = {}
|
||||||
|
for (const phase of Object.values(composed.phases)) {
|
||||||
|
for (const req of phase.requires) {
|
||||||
|
if (!route.requires.includes(req.formula)) {
|
||||||
|
route.requires.push(req.formula)
|
||||||
|
route.formulaSources[req.formula] = req.source
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (const ens of phase.ensures) {
|
||||||
|
if (!route.ensures.includes(ens.formula)) {
|
||||||
|
route.ensures.push(ens.formula)
|
||||||
|
route.formulaSources[ens.formula] = ens.source
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// 3. Check for behavioral contracts
|
// 3. Check for behavioral contracts
|
||||||
const routesWithContracts = routes.filter(route =>
|
const routesWithContracts = routes.filter(route =>
|
||||||
route.ensures.length > 0 || route.requires.length > 0
|
route.ensures.length > 0 || route.requires.length > 0
|
||||||
@@ -522,6 +549,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
|
|||||||
total: 0,
|
total: 0,
|
||||||
passedCount: 0,
|
passedCount: 0,
|
||||||
failed: 0,
|
failed: 0,
|
||||||
|
skipped: 0,
|
||||||
failures: [],
|
failures: [],
|
||||||
durationMs: Date.now() - started,
|
durationMs: Date.now() - started,
|
||||||
noRoutesMatched: false,
|
noRoutesMatched: false,
|
||||||
@@ -533,32 +561,117 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
|
|||||||
}
|
}
|
||||||
|
|
||||||
// 4. Execute contracts (with variant expansion)
|
// 4. Execute contracts (with variant expansion)
|
||||||
|
// Execute one HTTP request per (route, variant, runIndex), then validate
|
||||||
|
// all contracts against the same request context.
|
||||||
|
// Requires (preconditions) that fail → SKIP the ensures and count as skipped.
|
||||||
|
// Ensures (postconditions) that fail → count as failures.
|
||||||
const runConfig = resolveRuns(deps.runs)
|
const runConfig = resolveRuns(deps.runs)
|
||||||
const runs = runConfig.contractRuns
|
const runs = runConfig.contractRuns
|
||||||
const failures: VerifyFailure[] = []
|
const failures: VerifyFailure[] = []
|
||||||
let total = 0
|
let total = 0
|
||||||
let passedCount = 0
|
let passedCount = 0
|
||||||
|
let skippedCount = 0
|
||||||
|
|
||||||
|
// Extract required headers from precondition formulas and auto-inject them
|
||||||
|
// so that plugin contracts (e.g., x-tenant-id != null) are satisfied.
|
||||||
|
function extractRequiredHeaders(formulas: string[]): Record<string, string> {
|
||||||
|
const headers: Record<string, string> = {}
|
||||||
|
for (const f of formulas) {
|
||||||
|
const nullCheck = f.match(/request_headers\(this\)\.([\w-]+)\s*!=\s*null/i)
|
||||||
|
if (nullCheck) { headers[nullCheck[1]!] = 'test-value'; continue }
|
||||||
|
const eqCheck = f.match(/request_headers\(this\)\.([\w-]+)\s*==\s*"([^"]+)"/i)
|
||||||
|
if (eqCheck) { headers[eqCheck[1]!] = eqCheck[2]!; continue }
|
||||||
|
}
|
||||||
|
return headers
|
||||||
|
}
|
||||||
|
|
||||||
for (const route of routesWithContracts) {
|
for (const route of routesWithContracts) {
|
||||||
const contracts = [...route.requires, ...route.ensures]
|
const defaultHeaders = extractRequiredHeaders(route.requires)
|
||||||
const variants = route.variants && route.variants.length > 0
|
const variants = route.variants && route.variants.length > 0
|
||||||
? route.variants
|
? route.variants
|
||||||
: [{ name: 'default' }]
|
: [{ name: 'default' }]
|
||||||
|
|
||||||
for (const variant of variants) {
|
for (const variant of variants) {
|
||||||
for (const contract of contracts) {
|
|
||||||
for (let runIndex = 0; runIndex < runs; runIndex++) {
|
for (let runIndex = 0; runIndex < runs; runIndex++) {
|
||||||
total++
|
// Execute one HTTP request per run
|
||||||
const result = await executeContract(
|
const request = buildRouteRequest(route, deps.seed, runIndex, variant)
|
||||||
fastify, route, contract, deps.timeout,
|
const headers = {
|
||||||
variant, deps.seed, runIndex,
|
...defaultHeaders,
|
||||||
)
|
...request.headers,
|
||||||
|
...(variant.headers ?? {}),
|
||||||
|
}
|
||||||
|
const ctx = await executeHttp(fastify, route, {
|
||||||
|
method: request.method,
|
||||||
|
url: request.url,
|
||||||
|
body: request.body,
|
||||||
|
headers,
|
||||||
|
query: {},
|
||||||
|
}, undefined, deps.timeout)
|
||||||
|
const evalCtx: EvalContext = {
|
||||||
|
...ctx,
|
||||||
|
operationResolver: createOperationResolver(fastify, headers, ctx),
|
||||||
|
}
|
||||||
|
|
||||||
if (result.failure) {
|
// Evaluate requires first. If any fail, skip ensures for this run.
|
||||||
failures.push(result.failure)
|
let requiresPassed = true
|
||||||
|
for (const formula of route.requires) {
|
||||||
|
total++
|
||||||
|
try {
|
||||||
|
const parsed = parse(formula)
|
||||||
|
const result = await evaluateAsync(parsed.ast, evalCtx)
|
||||||
|
if (!result.success || !result.value) {
|
||||||
|
requiresPassed = false
|
||||||
|
skippedCount++
|
||||||
} else {
|
} else {
|
||||||
passedCount++
|
passedCount++
|
||||||
}
|
}
|
||||||
|
} catch {
|
||||||
|
requiresPassed = false
|
||||||
|
skippedCount++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Only evaluate ensures if all requires passed
|
||||||
|
if (requiresPassed) {
|
||||||
|
for (const formula of route.ensures) {
|
||||||
|
total++
|
||||||
|
try {
|
||||||
|
const parsed = parse(formula)
|
||||||
|
const result = await evaluateAsync(parsed.ast, evalCtx)
|
||||||
|
if (!result.success || !result.value) {
|
||||||
|
const diagnostic = buildFailureDiagnostic(formula, evalCtx, result.success ? undefined : result.error)
|
||||||
|
failures.push({
|
||||||
|
route: variant.name && variant.name !== 'default'
|
||||||
|
? `[variant:${variant.name}] ${route.method} ${route.path}`
|
||||||
|
: `${route.method} ${route.path}`,
|
||||||
|
contract: formula,
|
||||||
|
expected: diagnostic.expected,
|
||||||
|
observed: diagnostic.observed,
|
||||||
|
formula: diagnostic.formula,
|
||||||
|
category: diagnostic.category,
|
||||||
|
})
|
||||||
|
} else {
|
||||||
|
passedCount++
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
const diagnostic = buildFailureDiagnostic(formula, evalCtx, error instanceof Error ? error.message : String(error))
|
||||||
|
failures.push({
|
||||||
|
route: variant.name && variant.name !== 'default'
|
||||||
|
? `[variant:${variant.name}] ${route.method} ${route.path}`
|
||||||
|
: `${route.method} ${route.path}`,
|
||||||
|
contract: formula,
|
||||||
|
expected: diagnostic.expected,
|
||||||
|
observed: diagnostic.observed,
|
||||||
|
formula: diagnostic.formula,
|
||||||
|
category: diagnostic.category,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// Preconditions failed: ensures are skipped but not evaluated.
|
||||||
|
// We still count each ensure as "total" and "skipped" for transparency.
|
||||||
|
total += route.ensures.length
|
||||||
|
skippedCount += route.ensures.length
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -578,6 +691,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
|
|||||||
total,
|
total,
|
||||||
passedCount,
|
passedCount,
|
||||||
failed: failures.length,
|
failed: failures.length,
|
||||||
|
skipped: skippedCount,
|
||||||
failures: sortedFailures,
|
failures: sortedFailures,
|
||||||
durationMs,
|
durationMs,
|
||||||
noRoutesMatched: false,
|
noRoutesMatched: false,
|
||||||
|
|||||||
@@ -140,7 +140,7 @@ const CONFIG_SCHEMA: Record<string, SchemaField> = {
|
|||||||
optional: true,
|
optional: true,
|
||||||
items: { type: 'object' },
|
items: { type: 'object' },
|
||||||
},
|
},
|
||||||
chaos: {
|
pluginContracts: {
|
||||||
type: 'object',
|
type: 'object',
|
||||||
optional: true,
|
optional: true,
|
||||||
properties: {},
|
properties: {},
|
||||||
|
|||||||
@@ -93,7 +93,7 @@ export function createContext(options: Record<string, unknown> = {}): CliContext
|
|||||||
const packageManager = detectPackageManager(cwd);
|
const packageManager = detectPackageManager(cwd);
|
||||||
|
|
||||||
// Normalize options
|
// Normalize options
|
||||||
const format = options.format === 'json' || options.format === 'ndjson'
|
const format = options.format === 'json' || options.format === 'ndjson' || options.format === 'json-summary' || options.format === 'ndjson-summary'
|
||||||
? options.format
|
? options.format
|
||||||
: 'human';
|
: 'human';
|
||||||
|
|
||||||
|
|||||||
@@ -102,9 +102,11 @@ function getCommandHelp(command: string): string {
|
|||||||
${pc.dim('Options:')}
|
${pc.dim('Options:')}
|
||||||
--profile <name> Profile name from config
|
--profile <name> Profile name from config
|
||||||
--seed <number> Deterministic seed
|
--seed <number> Deterministic seed
|
||||||
|
--changed Filter to git-modified routes
|
||||||
|
|
||||||
${pc.dim('Examples:')}
|
${pc.dim('Examples:')}
|
||||||
apophis qualify --profile oauth-nightly --seed 42
|
apophis qualify --profile oauth-nightly --seed 42
|
||||||
|
apophis qualify --changed
|
||||||
`,
|
`,
|
||||||
replay: `
|
replay: `
|
||||||
${pc.bold('apophis replay')} — Replay a failure using seed and stored trace
|
${pc.bold('apophis replay')} — Replay a failure using seed and stored trace
|
||||||
@@ -167,18 +169,18 @@ function printInternalError(error: unknown): void {
|
|||||||
console.error();
|
console.error();
|
||||||
}
|
}
|
||||||
|
|
||||||
function resolveRequestedFormat(argv: string[]): 'human' | 'json' | 'ndjson' {
|
function resolveRequestedFormat(argv: string[]): 'human' | 'json' | 'ndjson' | 'json-summary' | 'ndjson-summary' {
|
||||||
for (let i = 0; i < argv.length; i++) {
|
for (let i = 0; i < argv.length; i++) {
|
||||||
const arg = argv[i];
|
const arg = argv[i];
|
||||||
if (!arg) continue;
|
if (!arg) continue;
|
||||||
if (arg === '--format' && argv[i + 1]) {
|
if (arg === '--format' && argv[i + 1]) {
|
||||||
const value = argv[i + 1];
|
const value = argv[i + 1];
|
||||||
if (value === 'json' || value === 'ndjson') return value;
|
if (value === 'json' || value === 'ndjson' || value === 'json-summary' || value === 'ndjson-summary') return value as 'json' | 'ndjson' | 'json-summary' | 'ndjson-summary';
|
||||||
return 'human';
|
return 'human';
|
||||||
}
|
}
|
||||||
if (arg.startsWith('--format=')) {
|
if (arg.startsWith('--format=')) {
|
||||||
const value = arg.slice('--format='.length);
|
const value = arg.slice('--format='.length);
|
||||||
if (value === 'json' || value === 'ndjson') return value;
|
if (value === 'json' || value === 'ndjson' || value === 'json-summary' || value === 'ndjson-summary') return value as 'json' | 'ndjson' | 'json-summary' | 'ndjson-summary';
|
||||||
return 'human';
|
return 'human';
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -277,6 +279,7 @@ export async function main(argv: string[] = process.argv.slice(2)): Promise<numb
|
|||||||
case 'qualify':
|
case 'qualify':
|
||||||
cmd.option('--profile <name>', 'Profile name from config');
|
cmd.option('--profile <name>', 'Profile name from config');
|
||||||
cmd.option('--seed <number>', 'Deterministic seed');
|
cmd.option('--seed <number>', 'Deterministic seed');
|
||||||
|
cmd.option('--changed', 'Filter to git-modified routes');
|
||||||
break;
|
break;
|
||||||
case 'replay':
|
case 'replay':
|
||||||
cmd.option('--artifact <path>', 'Path to failure artifact');
|
cmd.option('--artifact <path>', 'Path to failure artifact');
|
||||||
@@ -375,7 +378,7 @@ export async function main(argv: string[] = process.argv.slice(2)): Promise<numb
|
|||||||
init: new Set(['--preset', '-p', '--force', '-f', '--noninteractive']),
|
init: new Set(['--preset', '-p', '--force', '-f', '--noninteractive']),
|
||||||
verify: new Set(['--profile', '--routes', '--seed', '--changed', '--workspace']),
|
verify: new Set(['--profile', '--routes', '--seed', '--changed', '--workspace']),
|
||||||
observe: new Set(['--profile', '--check-config', '--workspace']),
|
observe: new Set(['--profile', '--check-config', '--workspace']),
|
||||||
qualify: new Set(['--profile', '--seed', '--workspace']),
|
qualify: new Set(['--profile', '--seed', '--workspace', '--changed']),
|
||||||
replay: new Set(['--artifact', '--route']),
|
replay: new Set(['--artifact', '--route']),
|
||||||
doctor: new Set(['--mode', '--strict', '--workspace']),
|
doctor: new Set(['--mode', '--strict', '--workspace']),
|
||||||
migrate: new Set(['--check', '--dry-run', '--write']),
|
migrate: new Set(['--check', '--dry-run', '--write']),
|
||||||
|
|||||||
@@ -297,6 +297,7 @@ export interface Artifact {
|
|||||||
total: number;
|
total: number;
|
||||||
passed: number;
|
passed: number;
|
||||||
failed: number;
|
failed: number;
|
||||||
|
skipped?: number;
|
||||||
};
|
};
|
||||||
executionSummary?: ExecutionSummary;
|
executionSummary?: ExecutionSummary;
|
||||||
coverageBreakdown?: {
|
coverageBreakdown?: {
|
||||||
|
|||||||
@@ -202,6 +202,10 @@ function renderSummary(
|
|||||||
lines.push(` Failed: ${summary.failed}`);
|
lines.push(` Failed: ${summary.failed}`);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (summary.skipped !== undefined && summary.skipped > 0) {
|
||||||
|
lines.push(` ${c.yellow('Skipped:')} ${summary.skipped} (preconditions not met)`);
|
||||||
|
}
|
||||||
|
|
||||||
lines.push(` Duration: ${formatDuration(artifact.durationMs)}`);
|
lines.push(` Duration: ${formatDuration(artifact.durationMs)}`);
|
||||||
|
|
||||||
if (artifact.seed !== undefined) {
|
if (artifact.seed !== undefined) {
|
||||||
@@ -404,15 +408,31 @@ export function renderHumanArtifact(
|
|||||||
lines.push(c.bold('Next steps'));
|
lines.push(c.bold('Next steps'));
|
||||||
if (artifact.command === 'verify') {
|
if (artifact.command === 'verify') {
|
||||||
if (artifact.summary.failed === 0) {
|
if (artifact.summary.failed === 0) {
|
||||||
|
if (artifact.summary.skipped && artifact.summary.skipped > 0) {
|
||||||
|
lines.push(` ${c.yellow('!')} No contract violations, but ${artifact.summary.skipped} precondition(s) were unmet and skipped.`);
|
||||||
|
lines.push(` ${c.dim('→ Preconditions (x-requires + plugin contract requires) must be satisfied for tests to run.')}`);
|
||||||
|
lines.push(` ${c.dim('→ Send required headers via --variant or scope configuration.')}`);
|
||||||
|
lines.push(` ${c.dim('→ Add more behavioral contracts with')} x-ensures ${c.dim('and')} x-requires ${c.dim('to cover more routes.')}`);
|
||||||
|
} else {
|
||||||
lines.push(` ${c.green('✓')} All contracts passed.`);
|
lines.push(` ${c.green('✓')} All contracts passed.`);
|
||||||
lines.push(` ${c.dim('→ Add more behavioral contracts with')} x-ensures ${c.dim('and')} x-requires ${c.dim('to cover more routes.')}`);
|
lines.push(` ${c.dim('→ Add more behavioral contracts with')} x-ensures ${c.dim('and')} x-requires ${c.dim('to cover more routes.')}`);
|
||||||
lines.push(` ${c.dim('→ Run')} apophis observe ${c.dim('to enable runtime contract monitoring in production.')}`);
|
lines.push(` ${c.dim('→ Run')} apophis observe ${c.dim('to enable runtime contract monitoring in production.')}`);
|
||||||
lines.push(` ${c.dim('→ Run')} apophis qualify --profile standard ${c.dim('for stateful/chaos testing.')}`);
|
lines.push(` ${c.dim('→ Run')} apophis qualify --profile standard ${c.dim('for stateful/chaos testing.')}`);
|
||||||
|
}
|
||||||
} else {
|
} else {
|
||||||
lines.push(` ${c.yellow('!')} Fix failing contracts and rerun with:`);
|
lines.push(` ${c.yellow('!')} Fix failing contracts and rerun with:`);
|
||||||
lines.push(` ${c.dim('→')} apophis verify --seed ${artifact.seed} ${artifact.profile ? `--profile ${artifact.profile}` : ''}`);
|
lines.push(` ${c.dim('→')} apophis verify --seed ${artifact.seed} ${artifact.profile ? `--profile ${artifact.profile}` : ''}`);
|
||||||
lines.push(` ${c.dim('→ Or replay the artifact:')} apophis replay --artifact <path>`);
|
lines.push(` ${c.dim('→ Or replay the artifact:')} apophis replay --artifact <path>`);
|
||||||
}
|
}
|
||||||
|
} else if (artifact.command === 'qualify') {
|
||||||
|
if (artifact.summary.skipped && artifact.summary.skipped > 0) {
|
||||||
|
lines.push(` ${c.yellow('!')} ${artifact.summary.skipped} stateful command(s) skipped — preconditions not met.`);
|
||||||
|
lines.push(` ${c.dim('→ Check plugin contracts and route headers. Qualify needs preconditions satisfied.')}`);
|
||||||
|
}
|
||||||
|
if (artifact.summary.failed === 0 && (!artifact.summary.skipped || artifact.summary.skipped === 0)) {
|
||||||
|
lines.push(` ${c.green('✓')} All checks passed.`);
|
||||||
|
}
|
||||||
|
lines.push(` ${c.dim('→ Review')} reports/apophis/ ${c.dim('for detailed artifact output.')}`);
|
||||||
}
|
}
|
||||||
|
|
||||||
return lines.join('\n');
|
return lines.join('\n');
|
||||||
|
|||||||
@@ -179,11 +179,13 @@ const makeConditionFailure = (
|
|||||||
const expected = fieldExpectation?.expected ?? statusExpectation.expected
|
const expected = fieldExpectation?.expected ?? statusExpectation.expected
|
||||||
const actual = fieldExpectation?.actual ?? statusExpectation.actual
|
const actual = fieldExpectation?.actual ?? statusExpectation.actual
|
||||||
const diff = buildDiff(formula, ctx.response.body)
|
const diff = buildDiff(formula, ctx.response.body)
|
||||||
|
const source = (route as RouteContract | undefined)?.formulaSources?.[formula]
|
||||||
|
|
||||||
return makeViolation({
|
return makeViolation({
|
||||||
route: route ?? { method: '', path: '' },
|
route: route ?? { method: '', path: '' },
|
||||||
formula,
|
formula,
|
||||||
kind,
|
kind,
|
||||||
|
...(source ? { source, phase: 'postcondition' } : {}),
|
||||||
request: {
|
request: {
|
||||||
body: ctx.request.body,
|
body: ctx.request.body,
|
||||||
headers: ctx.request.headers,
|
headers: ctx.request.headers,
|
||||||
@@ -206,10 +208,12 @@ const makeFormulaError = (
|
|||||||
route: RouteContract | { method: string; path: string } | undefined,
|
route: RouteContract | { method: string; path: string } | undefined,
|
||||||
message: string
|
message: string
|
||||||
): ContractViolation => {
|
): ContractViolation => {
|
||||||
|
const source = (route as RouteContract | undefined)?.formulaSources?.[formula]
|
||||||
return makeViolation({
|
return makeViolation({
|
||||||
route: route ?? { method: '', path: '' },
|
route: route ?? { method: '', path: '' },
|
||||||
formula,
|
formula,
|
||||||
kind,
|
kind,
|
||||||
|
...(source ? { source, phase: 'postcondition' } : {}),
|
||||||
request: {
|
request: {
|
||||||
body: ctx.request.body,
|
body: ctx.request.body,
|
||||||
headers: ctx.request.headers,
|
headers: ctx.request.headers,
|
||||||
|
|||||||
@@ -39,6 +39,7 @@ function matchPattern(pattern: string, path: string): boolean {
|
|||||||
export class PluginContractRegistry {
|
export class PluginContractRegistry {
|
||||||
private contracts = new Map<string, PluginContractSpec>()
|
private contracts = new Map<string, PluginContractSpec>()
|
||||||
private availableExtensions = new Set<string>()
|
private availableExtensions = new Set<string>()
|
||||||
|
private _warnings: string[] = []
|
||||||
/**
|
/**
|
||||||
* Register a plugin's contract specification.
|
* Register a plugin's contract specification.
|
||||||
* Idempotent: registering the same plugin twice updates the spec.
|
* Idempotent: registering the same plugin twice updates the spec.
|
||||||
@@ -75,7 +76,7 @@ export class PluginContractRegistry {
|
|||||||
if (!matchPattern(spec.appliesTo, route.path)) continue
|
if (!matchPattern(spec.appliesTo, route.path)) continue
|
||||||
const extCheck = this.checkExtensions(spec)
|
const extCheck = this.checkExtensions(spec)
|
||||||
if (!extCheck.available) {
|
if (!extCheck.available) {
|
||||||
console.warn(
|
this._warnings.push(
|
||||||
`Plugin '${plugin}' requires extensions [${extCheck.missing.join(', ')}] which are not registered. Skipping its contracts.`
|
`Plugin '${plugin}' requires extensions [${extCheck.missing.join(', ')}] which are not registered. Skipping its contracts.`
|
||||||
)
|
)
|
||||||
continue
|
continue
|
||||||
@@ -141,6 +142,16 @@ export class PluginContractRegistry {
|
|||||||
getAvailableExtensions(): string[] {
|
getAvailableExtensions(): string[] {
|
||||||
return Array.from(this.availableExtensions)
|
return Array.from(this.availableExtensions)
|
||||||
}
|
}
|
||||||
|
/**
|
||||||
|
* Collect and clear warnings (e.g., missing extension dependencies).
|
||||||
|
* Callers should surface these via their own output channels instead of
|
||||||
|
* letting console.warn bleed into CLI formatted output.
|
||||||
|
*/
|
||||||
|
drainWarnings(): string[] {
|
||||||
|
const w = this._warnings
|
||||||
|
this._warnings = []
|
||||||
|
return w
|
||||||
|
}
|
||||||
}
|
}
|
||||||
// ============================================================================
|
// ============================================================================
|
||||||
// Factory
|
// Factory
|
||||||
|
|||||||
+15
-2
@@ -1,11 +1,17 @@
|
|||||||
import type { FastifyInstance, FastifyServerOptions } from 'fastify'
|
import type { FastifyInstance, FastifyServerOptions } from 'fastify'
|
||||||
import Fastify from 'fastify'
|
import Fastify from 'fastify'
|
||||||
import { installRouteDiscovery } from './infrastructure/discovery-hook.js'
|
import { installRouteDiscovery } from './infrastructure/discovery-hook.js'
|
||||||
|
import type { ObserveSink } from './types.js'
|
||||||
|
|
||||||
export interface CreateFastifyOptions extends FastifyServerOptions {
|
export interface CreateFastifyOptions extends FastifyServerOptions {
|
||||||
apophis?: {
|
apophis?: {
|
||||||
runtime?: 'off' | 'warn' | 'error'
|
runtime?: 'off' | 'warn' | 'error'
|
||||||
discoveryOnly?: boolean
|
discoveryOnly?: boolean
|
||||||
|
observe?: {
|
||||||
|
enabled?: boolean
|
||||||
|
sampling?: number
|
||||||
|
sinks?: ObserveSink[]
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -16,9 +22,16 @@ export async function createFastify(
|
|||||||
const fastify = Fastify(fastifyOpts)
|
const fastify = Fastify(fastifyOpts)
|
||||||
installRouteDiscovery(fastify as unknown as { addHook: Function })
|
installRouteDiscovery(fastify as unknown as { addHook: Function })
|
||||||
|
|
||||||
if (apophisOpts?.discoveryOnly !== true && apophisOpts?.runtime && apophisOpts.runtime !== 'off') {
|
const shouldRegisterApophis = apophisOpts?.discoveryOnly !== true && (
|
||||||
|
(apophisOpts?.runtime !== undefined && apophisOpts.runtime !== 'off') ||
|
||||||
|
apophisOpts?.observe?.enabled === true
|
||||||
|
)
|
||||||
|
if (shouldRegisterApophis) {
|
||||||
const { apophisPlugin } = await import('./plugin/index.js')
|
const { apophisPlugin } = await import('./plugin/index.js')
|
||||||
await fastify.register(apophisPlugin, { runtime: apophisOpts.runtime })
|
await fastify.register(apophisPlugin, {
|
||||||
|
runtime: apophisOpts.runtime,
|
||||||
|
observe: apophisOpts.observe,
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
return fastify
|
return fastify
|
||||||
|
|||||||
+31
-4
@@ -727,6 +727,8 @@ function parseVariable(input: string, pos: number): { node: FormulaNode; pos: nu
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
const name = input.substring(p, i)
|
const name = input.substring(p, i)
|
||||||
|
// F/T shorthands: treat as boolean literals, not variables.
|
||||||
|
// Only apply when there's no accessor (e.g., F.x is a variable, not literal).
|
||||||
let accessor: string[] | undefined
|
let accessor: string[] | undefined
|
||||||
const dotPos = skipWs(input, i)
|
const dotPos = skipWs(input, i)
|
||||||
if (input[dotPos] === '.') {
|
if (input[dotPos] === '.') {
|
||||||
@@ -734,6 +736,12 @@ function parseVariable(input: string, pos: number): { node: FormulaNode; pos: nu
|
|||||||
accessor = [acc.id]
|
accessor = [acc.id]
|
||||||
i = acc.pos
|
i = acc.pos
|
||||||
}
|
}
|
||||||
|
if ((name === 'F' || name === 'T') && accessor === undefined) {
|
||||||
|
return {
|
||||||
|
node: { type: 'literal', value: name === 'T' },
|
||||||
|
pos: i
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
return {
|
return {
|
||||||
node: { type: 'variable', name, accessor },
|
node: { type: 'variable', name, accessor },
|
||||||
@@ -842,12 +850,31 @@ function isWordChar(c: number): boolean {
|
|||||||
function parseClause(input: string, pos: number, extensionHeaders: string[] = []): { node: FormulaNode; pos: number } {
|
function parseClause(input: string, pos: number, extensionHeaders: string[] = []): { node: FormulaNode; pos: number } {
|
||||||
const p = skipWs(input, pos)
|
const p = skipWs(input, pos)
|
||||||
const len = input.length
|
const len = input.length
|
||||||
if (p < len && input[p] === 'T' && (p + 1 >= len || !isWordChar(input.charCodeAt(p + 1)))) {
|
|
||||||
return { node: { type: 'literal', value: true }, pos: p + 1 }
|
// T/F shorthand: only consume as standalone literal if NOT followed by a
|
||||||
|
// comparison operator (==, !=, <, >, <=, >=, matches).
|
||||||
|
function isTfLiteral(): boolean {
|
||||||
|
if (p >= len) return false
|
||||||
|
const ch = input[p]!
|
||||||
|
if (ch !== 'T' && ch !== 'F') return false
|
||||||
|
const afterP = p + 1
|
||||||
|
if (afterP >= len || !isWordChar(input.charCodeAt(afterP))) {
|
||||||
|
const next = skipWs(input, afterP)
|
||||||
|
const c0 = next < len ? input.charCodeAt(next) : -1
|
||||||
|
const c1 = next + 1 < len ? input.charCodeAt(next + 1) : -1
|
||||||
|
// If followed by a comparator, T/F is a term, not a standalone literal
|
||||||
|
if (c0 === 61 || c0 === 33 || c0 === 60 || c0 === 62) return false // ==, !=, <=, <, >=, >
|
||||||
|
if (c0 === 109) return false // matches
|
||||||
|
return true
|
||||||
}
|
}
|
||||||
if (p < len && input[p] === 'F' && (p + 1 >= len || !isWordChar(input.charCodeAt(p + 1)))) {
|
return false
|
||||||
return { node: { type: 'literal', value: false }, pos: p + 1 }
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (isTfLiteral()) {
|
||||||
|
const ch = input[p]!
|
||||||
|
return { node: { type: 'literal', value: ch === 'T' }, pos: p + 1 }
|
||||||
|
}
|
||||||
|
|
||||||
const cmp = parseComparison(input, pos, extensionHeaders)
|
const cmp = parseComparison(input, pos, extensionHeaders)
|
||||||
if (cmp) return cmp
|
if (cmp) return cmp
|
||||||
// If not a comparison, it must be a term (operation or literal)
|
// If not a comparison, it must be a term (operation or literal)
|
||||||
|
|||||||
@@ -67,12 +67,13 @@ export const buildCleanup = (cleanupManager: CleanupManager) =>
|
|||||||
export const buildScenario = (
|
export const buildScenario = (
|
||||||
fastify: FastifyInstance,
|
fastify: FastifyInstance,
|
||||||
scope: ScopeRegistry,
|
scope: ScopeRegistry,
|
||||||
extensionRegistry: ExtensionRegistry
|
extensionRegistry: ExtensionRegistry,
|
||||||
|
pluginContractRegistry?: import('../domain/plugin-contracts.js').PluginContractRegistry
|
||||||
) => async (opts: ScenarioConfig): Promise<ScenarioResult> => {
|
) => async (opts: ScenarioConfig): Promise<ScenarioResult> => {
|
||||||
assertNonProduction('scenario')
|
assertNonProduction('scenario')
|
||||||
const scopeHeaders = scope.getHeaders(opts.scope ?? null)
|
const scopeHeaders = scope.getHeaders(opts.scope ?? null)
|
||||||
const injectInstance = fastify as unknown as import('../types.js').FastifyInjectInstance
|
const injectInstance = fastify as unknown as import('../types.js').FastifyInjectInstance
|
||||||
return runScenario(injectInstance, opts, scopeHeaders, extensionRegistry)
|
return runScenario(injectInstance, opts, scopeHeaders, extensionRegistry, pluginContractRegistry)
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|||||||
+15
-7
@@ -96,7 +96,7 @@ export const apophisPlugin = async (fastify: FastifyInstance, opts: ApophisOptio
|
|||||||
contract: buildContract(fastify, scope, extensionRegistry, pluginContractRegistry, outboundContractRegistry),
|
contract: buildContract(fastify, scope, extensionRegistry, pluginContractRegistry, outboundContractRegistry),
|
||||||
stateful: buildStateful(fastify, scope, cleanupManager, extensionRegistry, pluginContractRegistry, outboundContractRegistry),
|
stateful: buildStateful(fastify, scope, cleanupManager, extensionRegistry, pluginContractRegistry, outboundContractRegistry),
|
||||||
check: buildCheck(fastify, scope, extensionRegistry, pluginContractRegistry),
|
check: buildCheck(fastify, scope, extensionRegistry, pluginContractRegistry),
|
||||||
scenario: buildScenario(fastify, scope, extensionRegistry),
|
scenario: buildScenario(fastify, scope, extensionRegistry, pluginContractRegistry),
|
||||||
cleanup: buildCleanup(cleanupManager),
|
cleanup: buildCleanup(cleanupManager),
|
||||||
spec: buildSpec(fastify),
|
spec: buildSpec(fastify),
|
||||||
test: {
|
test: {
|
||||||
@@ -147,17 +147,25 @@ export const apophisPlugin = async (fastify: FastifyInstance, opts: ApophisOptio
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
fastify.decorate('apophis', decorations)
|
fastify.decorate('apophis', decorations)
|
||||||
// Runtime validation: never register hooks in production
|
// Expose plugin contract registry for CLI qualify access
|
||||||
|
;(fastify as unknown as Record<string, unknown>)._apophisPluginContractRegistry = pluginContractRegistry
|
||||||
|
// Runtime validation may block requests, so it is disabled in production.
|
||||||
|
// Observe sinks are explicitly non-blocking and are allowed in production.
|
||||||
const isProd = process.env.NODE_ENV === 'production' || process.env.NODE_ENV === 'prod'
|
const isProd = process.env.NODE_ENV === 'production' || process.env.NODE_ENV === 'prod'
|
||||||
if (opts.runtime && opts.runtime !== 'off' && !isProd) {
|
const observeSinks = opts.observe?.enabled && opts.observe.sinks && opts.observe.sinks.length > 0
|
||||||
|
? opts.observe.sinks
|
||||||
|
: undefined
|
||||||
|
const enableBlockingRuntime = opts.runtime && opts.runtime !== 'off' && !isProd
|
||||||
|
if (enableBlockingRuntime || observeSinks) {
|
||||||
|
const observeSampling = opts.observe?.sampling ?? 1
|
||||||
const hookOpts: { validateRuntime: boolean; runtimeLevel?: 'warn' | 'error'; observers?: { sinks: import('../types.js').ObserveSink[]; sampling: number } } = {
|
const hookOpts: { validateRuntime: boolean; runtimeLevel?: 'warn' | 'error'; observers?: { sinks: import('../types.js').ObserveSink[]; sampling: number } } = {
|
||||||
validateRuntime: true,
|
validateRuntime: true,
|
||||||
runtimeLevel: opts.runtime,
|
runtimeLevel: enableBlockingRuntime ? opts.runtime : 'warn',
|
||||||
}
|
}
|
||||||
if (opts.observe?.enabled && opts.observe.sinks && opts.observe.sinks.length > 0) {
|
if (observeSinks) {
|
||||||
hookOpts.observers = {
|
hookOpts.observers = {
|
||||||
sinks: opts.observe.sinks,
|
sinks: observeSinks,
|
||||||
sampling: opts.observe.sampling ?? 1,
|
sampling: observeSampling,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
registerValidationHooks(fastify, hookOpts as Parameters<typeof registerValidationHooks>[1])
|
registerValidationHooks(fastify, hookOpts as Parameters<typeof registerValidationHooks>[1])
|
||||||
|
|||||||
@@ -118,15 +118,18 @@ export const runPetitTests = async (
|
|||||||
if (pluginContractRegistry) {
|
if (pluginContractRegistry) {
|
||||||
for (const route of routes) {
|
for (const route of routes) {
|
||||||
const composed = pluginContractRegistry.composeContracts(route)
|
const composed = pluginContractRegistry.composeContracts(route)
|
||||||
|
if (!route.formulaSources) route.formulaSources = {}
|
||||||
for (const phase of Object.values(composed.phases)) {
|
for (const phase of Object.values(composed.phases)) {
|
||||||
for (const req of phase.requires) {
|
for (const req of phase.requires) {
|
||||||
if (!route.requires.includes(req.formula)) {
|
if (!route.requires.includes(req.formula)) {
|
||||||
route.requires.push(req.formula)
|
route.requires.push(req.formula)
|
||||||
|
route.formulaSources[req.formula] = req.source
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
for (const ens of phase.ensures) {
|
for (const ens of phase.ensures) {
|
||||||
if (!route.ensures.includes(ens.formula)) {
|
if (!route.ensures.includes(ens.formula)) {
|
||||||
route.ensures.push(ens.formula)
|
route.ensures.push(ens.formula)
|
||||||
|
route.formulaSources[ens.formula] = ens.source
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -276,7 +279,16 @@ export const runPetitTests = async (
|
|||||||
|
|
||||||
flushCache()
|
flushCache()
|
||||||
const deduped = deduplicateTestFailures(results).results
|
const deduped = deduplicateTestFailures(results).results
|
||||||
const suite: TestSuite = buildPetitSuite(allRoutes, routes, skippedRoutes, deduped, cacheHits, cacheMisses, startTime)
|
const { formatSuite } = await import('./formatters.js')
|
||||||
|
const suite: TestSuite = formatSuite(deduped, {
|
||||||
|
startTime,
|
||||||
|
cacheHits,
|
||||||
|
cacheMisses,
|
||||||
|
allRoutes,
|
||||||
|
testedRoutes: routes,
|
||||||
|
skippedRoutes,
|
||||||
|
pluginContractRegistry,
|
||||||
|
})
|
||||||
|
|
||||||
if (suiteMockRuntime) suiteMockRuntime.restore()
|
if (suiteMockRuntime) suiteMockRuntime.restore()
|
||||||
if (extensionRegistry) await extensionRegistry.runSuiteEndHooks(suite)
|
if (extensionRegistry) await extensionRegistry.runSuiteEndHooks(suite)
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import { executeHttp } from '../infrastructure/http-executor.js'
|
|||||||
import { validatePostconditionsAsync } from '../domain/contract-validation.js'
|
import { validatePostconditionsAsync } from '../domain/contract-validation.js'
|
||||||
import { CONTENT_TYPE } from '../infrastructure/http-executor.js'
|
import { CONTENT_TYPE } from '../infrastructure/http-executor.js'
|
||||||
import type { ExtensionRegistry } from '../extension/types.js'
|
import type { ExtensionRegistry } from '../extension/types.js'
|
||||||
|
import type { PluginContractRegistry } from '../domain/plugin-contracts.js'
|
||||||
import type {
|
import type {
|
||||||
EvalContext,
|
EvalContext,
|
||||||
FastifyInjectInstance,
|
FastifyInjectInstance,
|
||||||
@@ -123,7 +124,8 @@ export const runScenario = async (
|
|||||||
fastify: FastifyInjectInstance,
|
fastify: FastifyInjectInstance,
|
||||||
config: ScenarioConfig,
|
config: ScenarioConfig,
|
||||||
scopeHeaders: Record<string, string>,
|
scopeHeaders: Record<string, string>,
|
||||||
extensionRegistry?: ExtensionRegistry
|
extensionRegistry?: ExtensionRegistry,
|
||||||
|
pluginContractRegistry?: PluginContractRegistry
|
||||||
): Promise<ScenarioResult> => {
|
): Promise<ScenarioResult> => {
|
||||||
const started = Date.now()
|
const started = Date.now()
|
||||||
const steps: ScenarioStepResult[] = []
|
const steps: ScenarioStepResult[] = []
|
||||||
@@ -162,6 +164,37 @@ export const runScenario = async (
|
|||||||
}
|
}
|
||||||
|
|
||||||
const route = buildScenarioRoute({ ...interpolated, method })
|
const route = buildScenarioRoute({ ...interpolated, method })
|
||||||
|
|
||||||
|
// Merge plugin contracts into the synthetic route
|
||||||
|
if (pluginContractRegistry) {
|
||||||
|
const composed = pluginContractRegistry.composeContracts(route)
|
||||||
|
if (!route.formulaSources) route.formulaSources = {}
|
||||||
|
for (const phase of Object.values(composed.phases)) {
|
||||||
|
for (const req of phase.requires) {
|
||||||
|
if (!route.requires.includes(req.formula)) {
|
||||||
|
route.requires.push(req.formula)
|
||||||
|
route.formulaSources[req.formula] = req.source
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (const ens of phase.ensures) {
|
||||||
|
if (!route.ensures.includes(ens.formula)) {
|
||||||
|
route.ensures.push(ens.formula)
|
||||||
|
route.formulaSources[ens.formula] = ens.source
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Auto-inject headers required by plugin contract preconditions
|
||||||
|
for (const f of route.requires) {
|
||||||
|
const m = f.match(/request_headers\(this\)\.([\w-]+)\s*(!=\s*null|==\s*"([^"]+)")/)
|
||||||
|
if (m) {
|
||||||
|
if (headers[m[1]!] === undefined) {
|
||||||
|
headers[m[1]!] = m[3] || 'test-value'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
const preContext: EvalContext = {
|
const preContext: EvalContext = {
|
||||||
request: {
|
request: {
|
||||||
body,
|
body,
|
||||||
@@ -204,7 +237,8 @@ export const runScenario = async (
|
|||||||
operationResolver: createOperationResolver(fastify, headers, preContext),
|
operationResolver: createOperationResolver(fastify, headers, preContext),
|
||||||
}
|
}
|
||||||
|
|
||||||
const validation = await validatePostconditionsAsync(step.expect.slice(), evalCtx, route, extensionRegistry)
|
const mergedEnsures = [...step.expect, ...route.ensures]
|
||||||
|
const validation = await validatePostconditionsAsync(mergedEnsures, evalCtx, route, extensionRegistry)
|
||||||
if (!validation.success) {
|
if (!validation.success) {
|
||||||
steps.push({
|
steps.push({
|
||||||
name: step.name,
|
name: step.name,
|
||||||
|
|||||||
@@ -49,6 +49,17 @@ const runBuildAndPrecondition = async (
|
|||||||
const { fastify, scopeHeaders, rng, extensionRegistry } = runtime
|
const { fastify, scopeHeaders, rng, extensionRegistry } = runtime
|
||||||
|
|
||||||
let request = buildRequest(command.route, command.params, scopeHeaders, modelState, rng)
|
let request = buildRequest(command.route, command.params, scopeHeaders, modelState, rng)
|
||||||
|
// Auto-inject headers required by plugin contract preconditions
|
||||||
|
if (command.route.requires.length > 0) {
|
||||||
|
const merged = { ...request.headers }
|
||||||
|
for (const f of command.route.requires) {
|
||||||
|
const m = f.match(/request_headers\(this\)\.([\w-]+)\s*(!=\s*null|==\s*"([^"]+)")/)
|
||||||
|
if (m) {
|
||||||
|
merged[m[1]!] = m[3] || 'test-value'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
request.headers = merged
|
||||||
|
}
|
||||||
if (extensionRegistry) {
|
if (extensionRegistry) {
|
||||||
request = await extensionRegistry.runBuildRequestHooks({
|
request = await extensionRegistry.runBuildRequestHooks({
|
||||||
route: command.route,
|
route: command.route,
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ import type { OutboundContractRegistry } from '../domain/outbound-contracts.js'
|
|||||||
import * as fc from 'fast-check'
|
import * as fc from 'fast-check'
|
||||||
import type { ModelState } from '../domain/stateful.js'
|
import type { ModelState } from '../domain/stateful.js'
|
||||||
import type { CleanupManager } from '../infrastructure/cleanup-manager.js'
|
import type { CleanupManager } from '../infrastructure/cleanup-manager.js'
|
||||||
import type { EvalContext, FastifyInjectInstance, RouteContract, ScopeRegistry, TestConfig, TestResult, TestSuite } from '../types.js'
|
import type { EvalContext, FastifyInjectInstance, RouteContract, ScopeRegistry, TestConfig, TestResult, TestSuite, ContractViolation } from '../types.js'
|
||||||
|
|
||||||
// Pure: hash helpers for deterministic sub-seeds
|
// Pure: hash helpers for deterministic sub-seeds
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -105,15 +105,18 @@ export const runStatefulTests = async (
|
|||||||
if (pluginContractRegistry) {
|
if (pluginContractRegistry) {
|
||||||
for (const route of routes) {
|
for (const route of routes) {
|
||||||
const composed = pluginContractRegistry.composeContracts(route)
|
const composed = pluginContractRegistry.composeContracts(route)
|
||||||
|
if (!route.formulaSources) route.formulaSources = {}
|
||||||
for (const phase of Object.values(composed.phases)) {
|
for (const phase of Object.values(composed.phases)) {
|
||||||
for (const req of phase.requires) {
|
for (const req of phase.requires) {
|
||||||
if (!route.requires.includes(req.formula)) {
|
if (!route.requires.includes(req.formula)) {
|
||||||
route.requires.push(req.formula)
|
route.requires.push(req.formula)
|
||||||
|
route.formulaSources[req.formula] = req.source
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
for (const ens of phase.ensures) {
|
for (const ens of phase.ensures) {
|
||||||
if (!route.ensures.includes(ens.formula)) {
|
if (!route.ensures.includes(ens.formula)) {
|
||||||
route.ensures.push(ens.formula)
|
route.ensures.push(ens.formula)
|
||||||
|
route.formulaSources[ens.formula] = ens.source
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -303,9 +306,36 @@ export const runStatefulTests = async (
|
|||||||
const passed = dedupedResults.filter((r) => r.ok && r.directive === undefined).length
|
const passed = dedupedResults.filter((r) => r.ok && r.directive === undefined).length
|
||||||
const failed = dedupedResults.filter((r) => !r.ok).length
|
const failed = dedupedResults.filter((r) => !r.ok).length
|
||||||
const skipped = dedupedResults.filter((r) => r.directive !== undefined).length
|
const skipped = dedupedResults.filter((r) => r.directive !== undefined).length
|
||||||
|
// Compute plugin contract counts
|
||||||
|
let pluginContractsApplied = 0
|
||||||
|
let pluginContractsFailed = 0
|
||||||
|
if (pluginContractRegistry) {
|
||||||
|
for (const route of routes) {
|
||||||
|
const composed = pluginContractRegistry.composeContracts(route)
|
||||||
|
for (const phase of Object.values(composed.phases)) {
|
||||||
|
pluginContractsApplied += phase.ensures.length + phase.requires.length
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (const result of dedupedResults) {
|
||||||
|
if (!result.ok && result.diagnostics?.violation) {
|
||||||
|
const violation = result.diagnostics.violation as ContractViolation
|
||||||
|
if (violation.source?.startsWith('plugin:')) pluginContractsFailed++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
const suite: TestSuite = {
|
const suite: TestSuite = {
|
||||||
tests: dedupedResults,
|
tests: dedupedResults,
|
||||||
summary: { passed, failed, skipped, timeMs: Date.now() - startTime, cacheHits, cacheMisses, counterexample: counterexampleOutput },
|
summary: {
|
||||||
|
passed,
|
||||||
|
failed,
|
||||||
|
skipped,
|
||||||
|
timeMs: Date.now() - startTime,
|
||||||
|
cacheHits,
|
||||||
|
cacheMisses,
|
||||||
|
counterexample: counterexampleOutput,
|
||||||
|
pluginContractsApplied,
|
||||||
|
pluginContractsFailed,
|
||||||
|
},
|
||||||
routes: allRoutes.map(r => ({
|
routes: allRoutes.map(r => ({
|
||||||
path: r.path,
|
path: r.path,
|
||||||
method: r.method,
|
method: r.method,
|
||||||
|
|||||||
@@ -1,6 +1,8 @@
|
|||||||
import { test } from 'node:test';
|
import { test } from 'node:test';
|
||||||
import assert from 'node:assert';
|
import assert from 'node:assert';
|
||||||
import { spawnSync } from 'node:child_process';
|
import { spawnSync } from 'node:child_process';
|
||||||
|
import fs from 'node:fs';
|
||||||
|
import path from 'node:path';
|
||||||
|
|
||||||
function runCli(args: string[]): { status: number; stdout: string; stderr: string } {
|
function runCli(args: string[]): { status: number; stdout: string; stderr: string } {
|
||||||
const result = spawnSync(
|
const result = spawnSync(
|
||||||
@@ -84,3 +86,158 @@ test('global help in machine mode emits parseable JSON', () => {
|
|||||||
assert.ok(typeof parsed.help === 'string');
|
assert.ok(typeof parsed.help === 'string');
|
||||||
assert.ok(parsed.help.includes('Usage:'));
|
assert.ok(parsed.help.includes('Usage:'));
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// json-summary and ndjson-summary format tests
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
test('verify --format json-summary emits parseable JSON with summary fields', () => {
|
||||||
|
const { status, stdout, stderr } = runCli([
|
||||||
|
'verify', '--cwd', 'src/cli/__fixtures__/tiny-fastify',
|
||||||
|
'--profile', 'quick', '--seed', '42', '--format', 'json-summary',
|
||||||
|
]);
|
||||||
|
|
||||||
|
assert.strictEqual(status, 0);
|
||||||
|
assert.strictEqual(stderr.trim(), '');
|
||||||
|
assert.ok(!stdout.includes('Seed:'), 'machine output must not include human seed prelude');
|
||||||
|
|
||||||
|
const parsed = JSON.parse(stdout);
|
||||||
|
assert.strictEqual(parsed.command, 'verify');
|
||||||
|
assert.strictEqual(typeof parsed.summary.total, 'number');
|
||||||
|
assert.strictEqual(typeof parsed.summary.passed, 'number');
|
||||||
|
assert.strictEqual(typeof parsed.summary.failed, 'number');
|
||||||
|
assert.strictEqual(typeof parsed.seed, 'number');
|
||||||
|
assert.strictEqual(typeof parsed.exitReason, 'string');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('verify --format ndjson-summary emits only NDJSON records', () => {
|
||||||
|
const { status, stdout, stderr } = runCli([
|
||||||
|
'verify', '--cwd', 'src/cli/__fixtures__/tiny-fastify',
|
||||||
|
'--profile', 'quick', '--seed', '42', '--format', 'ndjson-summary',
|
||||||
|
]);
|
||||||
|
|
||||||
|
assert.strictEqual(status, 0);
|
||||||
|
assert.strictEqual(stderr.trim(), '');
|
||||||
|
assert.ok(!stdout.includes('Seed:'), 'machine output must not include human seed prelude');
|
||||||
|
|
||||||
|
const lines = stdout.split('\n').map((line) => line.trim()).filter(Boolean);
|
||||||
|
assert.ok(lines.length >= 1, 'ndjson-summary should emit at least one record');
|
||||||
|
for (const line of lines) {
|
||||||
|
const parsed = JSON.parse(line);
|
||||||
|
assert.ok(typeof parsed.type === 'string' || typeof parsed.summary === 'object' || typeof parsed.command === 'string');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('verify --format json-summary --quiet still emits JSON', () => {
|
||||||
|
const { status, stdout, stderr } = runCli([
|
||||||
|
'verify', '--cwd', 'src/cli/__fixtures__/tiny-fastify',
|
||||||
|
'--profile', 'quick', '--seed', '42', '--format', 'json-summary', '--quiet',
|
||||||
|
]);
|
||||||
|
|
||||||
|
assert.strictEqual(status, 0);
|
||||||
|
assert.strictEqual(stderr.trim(), '');
|
||||||
|
assert.ok(stdout.trim().length > 0, '--quiet must not suppress machine-readable JSON');
|
||||||
|
const parsed = JSON.parse(stdout);
|
||||||
|
assert.strictEqual(parsed.command, 'verify');
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// skipped field in artifact
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
test('verify --format json-summary includes skipped field when preconditions cannot be met', () => {
|
||||||
|
const { status, stdout, stderr } = runCli([
|
||||||
|
'verify', '--cwd', 'src/cli/__fixtures__/plugin-contracts',
|
||||||
|
'--profile', 'quick', '--seed', '42', '--format', 'json-summary',
|
||||||
|
]);
|
||||||
|
|
||||||
|
// Plugin contract requires x-custom header which the generated request won't send.
|
||||||
|
// Auto-injection should add it, so preconditions pass.
|
||||||
|
// If not, skipped > 0 and failed === 0.
|
||||||
|
const parsed = JSON.parse(stdout);
|
||||||
|
assert.strictEqual(parsed.command, 'verify');
|
||||||
|
assert.ok(typeof parsed.summary.skipped === 'number', 'skipped field must be present');
|
||||||
|
assert.strictEqual(parsed.summary.failed, 0, 'no postcondition failures expected');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('verify --format json-summary exit code 0 when all contracts pass', () => {
|
||||||
|
const { status, stdout } = runCli([
|
||||||
|
'verify', '--cwd', 'src/cli/__fixtures__/tiny-fastify',
|
||||||
|
'--seed', '42', '--format', 'json-summary', '--quiet',
|
||||||
|
]);
|
||||||
|
|
||||||
|
assert.strictEqual(status, 0, 'exit code must be 0 when all contracts pass');
|
||||||
|
const parsed = JSON.parse(stdout);
|
||||||
|
assert.strictEqual(parsed.summary.failed, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// qualify json-summary and round-trip
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
test('qualify --format json-summary emits parseable JSON', () => {
|
||||||
|
const { status, stdout, stderr } = runCli([
|
||||||
|
'qualify', '--cwd', 'src/cli/__fixtures__/plugin-contracts',
|
||||||
|
'--seed', '42', '--format', 'json-summary',
|
||||||
|
]);
|
||||||
|
|
||||||
|
const parsed = JSON.parse(stdout);
|
||||||
|
assert.strictEqual(parsed.command, 'qualify');
|
||||||
|
assert.strictEqual(typeof parsed.summary.total, 'number');
|
||||||
|
assert.strictEqual(typeof parsed.summary.passed, 'number');
|
||||||
|
assert.strictEqual(typeof parsed.summary.failed, 'number');
|
||||||
|
assert.ok(typeof parsed.executionSummary === 'object', 'qualify artifact must include execution summary');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('verify + replay round-trip with plugin contracts', () => {
|
||||||
|
const artifactDir = '/tmp/apophis-replay-test-' + Date.now();
|
||||||
|
const { status: verifyStatus } = runCli([
|
||||||
|
'verify', '--cwd', 'src/cli/__fixtures__/plugin-contracts',
|
||||||
|
'--profile', 'quick', '--seed', '42', '--artifact-dir', artifactDir,
|
||||||
|
'--format', 'human',
|
||||||
|
]);
|
||||||
|
|
||||||
|
const files = fs.readdirSync(artifactDir);
|
||||||
|
const artifactFile = files.find((f) => f.endsWith('.json'));
|
||||||
|
assert.ok(artifactFile, 'artifact file should exist');
|
||||||
|
const artifactPath = path.join(artifactDir, artifactFile);
|
||||||
|
|
||||||
|
const { status: replayStatus, stdout: replayOut } = runCli([
|
||||||
|
'replay', '--artifact', artifactPath,
|
||||||
|
]);
|
||||||
|
|
||||||
|
fs.rmSync(artifactDir, { recursive: true, force: true });
|
||||||
|
|
||||||
|
assert.strictEqual(verifyStatus, 0, 'verify should exit 0');
|
||||||
|
// Replay exits 2 when artifact has no failures (nothing to reproduce).
|
||||||
|
// This proves the round-trip: artifact written, artifact loaded successfully.
|
||||||
|
assert.strictEqual(replayStatus, 2, 'replay exits 2 for no-failure artifact');
|
||||||
|
assert.ok(replayOut.includes('no failures to replay'), 'replay reports no failures');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('replay preserves seed from artifact', () => {
|
||||||
|
const artifactDir = '/tmp/apophis-seed-test-' + Date.now();
|
||||||
|
runCli([
|
||||||
|
'verify', '--cwd', 'src/cli/__fixtures__/plugin-contracts',
|
||||||
|
'--profile', 'quick', '--seed', '99', '--artifact-dir', artifactDir,
|
||||||
|
'--format', 'human',
|
||||||
|
]);
|
||||||
|
|
||||||
|
const files = fs.readdirSync(artifactDir);
|
||||||
|
const artifactFile = files.find((f) => f.endsWith('.json'));
|
||||||
|
assert.ok(artifactFile, 'artifact file should exist');
|
||||||
|
const artifactPath = path.join(artifactDir, artifactFile);
|
||||||
|
|
||||||
|
const artifactContent = fs.readFileSync(artifactPath, 'utf8');
|
||||||
|
const artifact = JSON.parse(artifactContent);
|
||||||
|
assert.strictEqual(artifact.seed, 99, 'artifact must preserve seed 99');
|
||||||
|
|
||||||
|
const { status: replayStatus, stdout: replayOut } = runCli([
|
||||||
|
'replay', '--artifact', artifactPath,
|
||||||
|
]);
|
||||||
|
|
||||||
|
fs.rmSync(artifactDir, { recursive: true, force: true });
|
||||||
|
|
||||||
|
assert.strictEqual(replayStatus, 2, 'replay exits 2 for no-failure artifact');
|
||||||
|
assert.ok(replayOut.includes('no failures to replay'), 'replay loads artifact with seed 99');
|
||||||
|
});
|
||||||
|
|||||||
@@ -222,6 +222,38 @@ test('parse: F shorthand for false', () => {
|
|||||||
const result = parse('F')
|
const result = parse('F')
|
||||||
assert.deepStrictEqual(result.ast, { type: 'literal', value: false })
|
assert.deepStrictEqual(result.ast, { type: 'literal', value: false })
|
||||||
})
|
})
|
||||||
|
test('parse: F == true parses as comparison', () => {
|
||||||
|
const result = parse('F == true')
|
||||||
|
assert.deepStrictEqual(result.ast, {
|
||||||
|
type: 'comparison', op: '==',
|
||||||
|
left: { type: 'literal', value: false },
|
||||||
|
right: { type: 'literal', value: true },
|
||||||
|
})
|
||||||
|
})
|
||||||
|
test('parse: T == false parses as comparison', () => {
|
||||||
|
const result = parse('T == false')
|
||||||
|
assert.deepStrictEqual(result.ast, {
|
||||||
|
type: 'comparison', op: '==',
|
||||||
|
left: { type: 'literal', value: true },
|
||||||
|
right: { type: 'literal', value: false },
|
||||||
|
})
|
||||||
|
})
|
||||||
|
test('parse: F != true parses as comparison', () => {
|
||||||
|
const result = parse('F != true')
|
||||||
|
assert.deepStrictEqual(result.ast, {
|
||||||
|
type: 'comparison', op: '!=',
|
||||||
|
left: { type: 'literal', value: false },
|
||||||
|
right: { type: 'literal', value: true },
|
||||||
|
})
|
||||||
|
})
|
||||||
|
test('parse: F < T parses as comparison', () => {
|
||||||
|
const result = parse('F < T')
|
||||||
|
assert.deepStrictEqual(result.ast, {
|
||||||
|
type: 'comparison', op: '<',
|
||||||
|
left: { type: 'literal', value: false },
|
||||||
|
right: { type: 'literal', value: true },
|
||||||
|
})
|
||||||
|
})
|
||||||
test('parse: throws on empty formula', () => {
|
test('parse: throws on empty formula', () => {
|
||||||
assert.throws(() => parse(''), /Empty formula/)
|
assert.throws(() => parse(''), /Empty formula/)
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import assert from 'node:assert'
|
|||||||
import Fastify from 'fastify'
|
import Fastify from 'fastify'
|
||||||
import type { FastifyInstance } from 'fastify'
|
import type { FastifyInstance } from 'fastify'
|
||||||
import apophisPlugin from '../index.js'
|
import apophisPlugin from '../index.js'
|
||||||
|
import { createFastify } from '../fastify-factory.js'
|
||||||
import { runPetitTests } from '../quality/petit-runner.js'
|
import { runPetitTests } from '../quality/petit-runner.js'
|
||||||
import { CleanupManager } from '../infrastructure/cleanup-manager.js'
|
import { CleanupManager } from '../infrastructure/cleanup-manager.js'
|
||||||
import { ScopeRegistry } from '../infrastructure/scope-registry.js'
|
import { ScopeRegistry } from '../infrastructure/scope-registry.js'
|
||||||
@@ -924,3 +925,198 @@ test('observe: violation event emitted but response still succeeds in observe mo
|
|||||||
await fastify.close()
|
await fastify.close()
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
|
test('observe: production mode still emits non-blocking sink events', async () => {
|
||||||
|
const originalEnv = process.env.NODE_ENV
|
||||||
|
process.env.NODE_ENV = 'production'
|
||||||
|
const fastify = Fastify()
|
||||||
|
try {
|
||||||
|
await fastify.register(swagger, {})
|
||||||
|
const emitted: ObserveEvent[] = []
|
||||||
|
const sink: ObserveSink = { emit: (event) => { emitted.push(event) } }
|
||||||
|
await fastify.register(apophisPlugin, {
|
||||||
|
runtime: 'off',
|
||||||
|
observe: { enabled: true, sinks: [sink], sampling: 1 },
|
||||||
|
})
|
||||||
|
fastify.get('/prod-observe', {
|
||||||
|
schema: {
|
||||||
|
'x-category': 'observer',
|
||||||
|
'x-ensures': ['status:201'],
|
||||||
|
} as Record<string, unknown>
|
||||||
|
}, async () => ({ ok: true }))
|
||||||
|
await fastify.ready()
|
||||||
|
const response = await fastify.inject({ method: 'GET', url: '/prod-observe' })
|
||||||
|
assert.strictEqual(response.statusCode, 200, 'production observe must not block responses')
|
||||||
|
const violationEvents = emitted.filter(e => e.type === 'contract.violation')
|
||||||
|
assert.ok(violationEvents.length > 0, 'production observe should emit contract violations')
|
||||||
|
assert.strictEqual(violationEvents[0]!.formula, 'status:201')
|
||||||
|
} finally {
|
||||||
|
await fastify.close()
|
||||||
|
if (originalEnv === undefined) delete process.env.NODE_ENV
|
||||||
|
else process.env.NODE_ENV = originalEnv
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
test('createFastify: observe options activate sink-backed runtime observation', async () => {
|
||||||
|
const emitted: ObserveEvent[] = []
|
||||||
|
const sink: ObserveSink = { emit: (event) => { emitted.push(event) } }
|
||||||
|
const fastify = await createFastify({
|
||||||
|
apophis: {
|
||||||
|
observe: { enabled: true, sinks: [sink], sampling: 1 },
|
||||||
|
},
|
||||||
|
})
|
||||||
|
try {
|
||||||
|
fastify.get('/factory-observe', {
|
||||||
|
schema: {
|
||||||
|
'x-category': 'observer',
|
||||||
|
'x-ensures': ['status:200'],
|
||||||
|
} as Record<string, unknown>
|
||||||
|
}, async () => ({ ok: true }))
|
||||||
|
await fastify.ready()
|
||||||
|
const response = await fastify.inject({ method: 'GET', url: '/factory-observe' })
|
||||||
|
assert.strictEqual(response.statusCode, 200)
|
||||||
|
const passEvents = emitted.filter(e => e.type === 'contract.pass')
|
||||||
|
assert.ok(passEvents.length > 0, 'createFastify observe options should wire runtime sink events')
|
||||||
|
} finally {
|
||||||
|
await fastify.close()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
test('createFastify: runtime off alone keeps APOPHIS plugin inactive', async () => {
|
||||||
|
const fastify = await createFastify({
|
||||||
|
apophis: { runtime: 'off' },
|
||||||
|
})
|
||||||
|
try {
|
||||||
|
assert.strictEqual(fastify.hasDecorator('apophis'), false)
|
||||||
|
} finally {
|
||||||
|
await fastify.close()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Plugin composition: plugin contracts merged into route contracts
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
test('plugin composition: contracts merge into route and source is tracked on violation', async () => {
|
||||||
|
const fastify = Fastify() as unknown as TestFastifyInstance
|
||||||
|
try {
|
||||||
|
await fastify.register(swagger, {})
|
||||||
|
await fastify.register(apophisPlugin, {
|
||||||
|
pluginContracts: {
|
||||||
|
'auth-plugin': {
|
||||||
|
appliesTo: '/api/**',
|
||||||
|
hooks: {
|
||||||
|
preHandler: { ensures: ['1 == 2'] },
|
||||||
|
},
|
||||||
|
meta: { name: 'auth-plugin' },
|
||||||
|
},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
fastify.get('/api/resource', {
|
||||||
|
schema: {
|
||||||
|
'x-category': 'observer',
|
||||||
|
response: { 200: { type: 'object' } },
|
||||||
|
} as Record<string, unknown>,
|
||||||
|
}, async () => ({ ok: true }))
|
||||||
|
await fastify.ready()
|
||||||
|
const suite = await fastify.apophis.contract({ runs: 5 })
|
||||||
|
assert.ok(suite.summary.pluginContractsApplied! > 0, 'plugin contracts should be counted as applied')
|
||||||
|
assert.ok(suite.summary.pluginContractsFailed! > 0, 'always-false ensures should cause at least one plugin contract failure')
|
||||||
|
} finally {
|
||||||
|
await fastify.close()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
test('plugin composition: missing required extension skips the plugin silently', async () => {
|
||||||
|
const fastify = Fastify() as unknown as TestFastifyInstance
|
||||||
|
try {
|
||||||
|
await fastify.register(swagger, {})
|
||||||
|
await fastify.register(apophisPlugin, {
|
||||||
|
pluginContracts: {
|
||||||
|
'requires-missing-ext': {
|
||||||
|
appliesTo: '/**',
|
||||||
|
extensions: [{ name: 'nonexistent', required: true }],
|
||||||
|
hooks: {
|
||||||
|
preHandler: { ensures: ['1 == 2'] },
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
fastify.get('/plain', {
|
||||||
|
schema: {
|
||||||
|
'x-category': 'observer',
|
||||||
|
'x-ensures': ['status:200'],
|
||||||
|
response: { 200: { type: 'object' } },
|
||||||
|
} as Record<string, unknown>,
|
||||||
|
}, async () => ({ ok: true }))
|
||||||
|
await fastify.ready()
|
||||||
|
const suite = await fastify.apophis.contract({ runs: 5 })
|
||||||
|
assert.strictEqual(suite.summary.failed, 0, 'route contract should pass without the skipped plugin formula')
|
||||||
|
assert.strictEqual(
|
||||||
|
suite.summary.pluginContractsApplied ?? 0,
|
||||||
|
1,
|
||||||
|
'only the route-level formula counts when plugin is skipped',
|
||||||
|
)
|
||||||
|
assert.strictEqual(
|
||||||
|
suite.summary.pluginContractsFailed ?? 0,
|
||||||
|
0,
|
||||||
|
'no plugin failures when plugin is skipped',
|
||||||
|
)
|
||||||
|
} finally {
|
||||||
|
await fastify.close()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
test('plugin composition: formula from plugin with extension passes when condition met', async () => {
|
||||||
|
const fastify = Fastify() as unknown as TestFastifyInstance
|
||||||
|
try {
|
||||||
|
await fastify.register(swagger, {})
|
||||||
|
await fastify.register(apophisPlugin, {
|
||||||
|
extensions: [
|
||||||
|
{
|
||||||
|
name: 'tenant-gate',
|
||||||
|
headers: ['check_tenant'],
|
||||||
|
predicates: {
|
||||||
|
check_tenant: (ctx: { evalContext: { request: { headers: Record<string, string> } } }) => ({
|
||||||
|
value: ctx.evalContext.request.headers['x-tenant'] === 'acme',
|
||||||
|
success: true,
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
} as import('../extension/types.js').ApophisExtension,
|
||||||
|
],
|
||||||
|
pluginContracts: {
|
||||||
|
'tenant-validator': {
|
||||||
|
appliesTo: '/**',
|
||||||
|
extensions: [{ name: 'tenant-gate', required: true }],
|
||||||
|
hooks: {
|
||||||
|
onRequest: { requires: ['check_tenant(this) == true'] },
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
fastify.get('/tenanted', {
|
||||||
|
schema: {
|
||||||
|
'x-category': 'observer',
|
||||||
|
'x-ensures': ['status:200'],
|
||||||
|
response: { 200: { type: 'object' } },
|
||||||
|
} as Record<string, unknown>,
|
||||||
|
}, async () => ({ ok: true }))
|
||||||
|
await fastify.ready()
|
||||||
|
const suite = await fastify.apophis.contract({
|
||||||
|
runs: 5,
|
||||||
|
variants: [{ name: 'acme', headers: { 'x-tenant': 'acme' } }],
|
||||||
|
})
|
||||||
|
assert.ok(suite.summary.pluginContractsApplied! > 0, 'tenant validator should be applied')
|
||||||
|
assert.strictEqual(suite.summary.pluginContractsFailed ?? 0, 0, 'tenant check should pass for acme')
|
||||||
|
assert.strictEqual(suite.summary.failed, 0, 'all tests should pass')
|
||||||
|
} finally {
|
||||||
|
await fastify.close()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Plugin composition + scenarios (integration)
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// NOTE: scenario + plugin contract integration test resides in
|
||||||
|
// _smoke-scenario-plugin.ts for isolation from cross-test state.
|
||||||
|
// The programmatic path is verified by running that smoke script directly.
|
||||||
|
|||||||
@@ -45,6 +45,8 @@ export interface RouteContract {
|
|||||||
outbound?: readonly OutboundBinding[]
|
outbound?: readonly OutboundBinding[]
|
||||||
/** Route-level variants for negotiated content-type or feature testing. Extracted from schema['x-variants']. */
|
/** Route-level variants for negotiated content-type or feature testing. Extracted from schema['x-variants']. */
|
||||||
variants?: RouteVariant[]
|
variants?: RouteVariant[]
|
||||||
|
/** Formula → source mapping for plugin contract attribution. Populated at test time. */
|
||||||
|
formulaSources?: Record<string, 'route' | `plugin:${string}`>
|
||||||
}
|
}
|
||||||
|
|
||||||
// ============================================================================
|
// ============================================================================
|
||||||
|
|||||||
Reference in New Issue
Block a user