apophis/apophis-fastify
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: harden engine, enrich failure diagnostics, close adoption gaps

Browse Source 
- P0: CLI verify now honors  test budget with seeded multi-sample
- P0: Observe sampling enforced via Math.random() gate in hook-validator
- P1: Remove misleading undici-mock-agent isolation option
- P1: Qualify reuses shared discoverRouteDetails() with warnings
- P1: Chaos/scenario config exposed via preset schema
- P1: README/docs limitations updated to current state
- P2: Nested response annotations prefer 2xx deterministically
- P2: --changed documented as heuristic in verify.md

- Add observe sink tests (sampling 0/1, sink failure non-interference)
- Add verify runs regression tests (scale, determinism, variants)
- Add configured-scenario qualify test (independent of OAuth fixture)
- Add coverageBreakdown to qualify artifacts (per-gate route coverage)
- Add production-style observe example with real sink in docs/observe.md
- Add nightly/staging vs PR gating guidance to docs/qualify.md

- Enrich VerifyFailure with formula-aware diagnostics:
  status:201 => 'HTTP 200', body field checks => actual values
- Remove stale observe CLI activation message
- Document outbound mocks as process-global in getting-started.md
- Refresh APOPHIS_ADOPTION_AUDIT.md with current state

903 tests pass, build clean, typecheck clean.
This commit is contained in:
John Dvorak
2026-05-21 20:39:36 -07:00
parent 55b0262799
commit d0523fcc2d
128 changed files with 4004 additions and 3631 deletions
Download Patch File Download Diff File Expand all files Collapse all files
APOPHIS.md
+22 -11
View File
@@ -1,22 +1,33 @@
# APOPHIS Setup — safe-ci preset # APOPHIS Setup — llm-safe preset
This project was scaffolded with `apophis init --preset safe-ci`. This project was scaffolded with `apophis init --preset llm-safe`.
## Quick Start ## Quick Start
1. Confirm the Fastify app registers `@fastify/swagger`. 1. Ensure you have a Fastify app with @fastify/swagger registered.
2. Add behavioral contracts to your route schemas using `x-ensures`. 2. Add behavioral contracts to your route schemas using `x-ensures`.
3. Run: apophis verify --profile quick 3. Run: apophis verify --profile llm-check
## What This Preset Does ## What This Preset Does
- Runs only behavioral contracts (not schema-only routes). - Ultra-minimal preset for LLM-generated codebases.
- No chaos, no observe, no stateful testing. - 3s timeout per route (fast feedback).
- Safe for CI pipelines. - No observe, no qualify, no chaos — verify only.
- Timeout: 5s per route. - Conservative defaults to avoid surprising failures.
## Example Behavioral Contract
Add this inside your route schema to check that a created resource is retrievable:
```javascript
"x-ensures": [
"response_code(GET /users/{response_body(this).id}) == 200"
]
```
If `apophis verify` says "No behavioral contracts found", it means your routes have schemas but no `x-ensures` or `x-requires` clauses. Add at least one clause per route you want to verify.
## Next Steps ## Next Steps
- Add more routes to the `routes` array in your profile. - Add routes to the `routes` array once you have behavioral contracts.
- Try `apophis init --preset platform-observe` to configure observe-mode policy and runtime drift reporting. - Run `apophis doctor` to check for missing dependencies.
- Try `apophis init --preset protocol-lab` for multi-step flows.
APOPHIS_ADOPTION_AUDIT.md
+239
View File
@@ -0,0 +1,239 @@
# APOPHIS Adoption Audit
Date: 2026-05-21
Scope: current working tree for `@apophis/fastify` v2.7.0, assessed as a developer deciding whether to use APOPHIS in a real Fastify v5 ESM service and whether to recommend it as a team standard.
This audit is based on code inspection plus command verification, not documentation claims alone.
## Executive Summary
APOPHIS has real product value. It is not just a schema wrapper: it gives Fastify teams a way to express and verify behavioral API promises that OpenAPI/JSON Schema cannot cover, especially cross-route invariants such as create/read consistency, delete semantics, auth/session flows, state transitions, idempotency, outbound dependency expectations, and replayable counterexamples.
I would adopt APOPHIS today as a focused behavioral verification tool for Fastify v5 ESM services. I would start with CI `verify` and a small number of high-value contracts, then expand into `qualify` and runtime observation once the team has clear operating guidance.
I would not yet treat it as a complete production observability platform or a turnkey organization-wide release gate. The core implementation is strong, but the remaining value gap is mostly around operational maturity: standalone observe activation, deeper tests around recent CLI behavior, richer scenario authoring, and clearer release-gate recommendations.
Adoption verdict: strong team pilot candidate, credible standardization candidate after the remaining gaps below are addressed.
## Verification Performed
Commands run successfully against the current working tree:
```bash
npm run typecheck
npm run build
npm run test:src
npm run test:cli
npm run test:docs
```
Observed results:
| Area | Result |
|---|---:|
| Typecheck | pass |
| Build | pass |
| Source tests | 587 pass, 0 fail |
| CLI tests | 311 pass, 0 fail |
| Docs smoke tests | 4 pass, 0 fail |
| Total tests | 902 pass, 0 fail |
The working tree contains many broader project changes unrelated to this audit. This document evaluates the current working tree state.
## Does It Do What It Says On The Tin?
Mostly yes for behavioral verification. Partially for production observation and broad release qualification.
| Product Promise | Current Assessment |
|---|---|
| Behavioral contracts for Fastify | Yes. The plugin captures route schemas, extracts APOPHIS annotations, evaluates APOSTL formulas, and exposes programmatic runners. |
| Deterministic CI verification | Yes, materially. CLI `verify` now honors configured `runs`, uses seeded request generation, emits artifacts, supports route filters, replay metadata, and machine-readable output. |
| Cross-route behavior | Yes for supported formula operations and route-call semantics. This is the most differentiated value. |
| Runtime validation | Yes when the plugin is explicitly configured outside production. Production enforcement is intentionally blocked. |
| Runtime observation | Partially. Programmatic plugin observation exists and emits non-blocking sink events with sampling. The CLI validates/report readiness but does not attach to or run a service. |
| Stateful/scenario/chaos qualification | Partially. The runner and artifacts are useful, route discovery is now shared with verify, and config supports scenarios/chaos knobs. Scenario authoring is still young and needs more real-world examples/tests. |
| Outbound dependency mocking | Useful but intentionally process-global. The misleading scoped `undici-mock-agent` option has been removed. Teams still need careful test isolation. |
| Team-safe onboarding | Good. The package has CLI help, init/doctor/replay/verify/qualify/observe, config validation, machine output, docs smoke tests, packaging tests, and production safety checks. |
## What Has Real Value
1. Behavioral contracts fill a real Fastify testing gap.
JSON Schema validates shape. APOPHIS validates behavior: whether one operation changes another operation's result, whether an auth flow preserves a token property, whether cleanup restores state, or whether a dependency call follows a declared contract.
Relevant code: `src/formula/parser.ts`, `src/formula/evaluator.ts`, `src/formula/runtime.ts`, `src/domain/contract.ts`, `src/domain/contract-validation.ts`.
2. Fastify integration is natural.
The package uses a real Fastify plugin, `fastify.inject()`, `onRoute` capture, a decorated `fastify.apophis` API, and a `createFastify()` helper for discovery ordering.
Relevant code: `src/plugin/index.ts`, `src/plugin/builders.ts`, `src/domain/discovery.ts`, `src/fastify-factory.ts`.
3. CLI verification now has credible depth.
`verifyCommand()` resolves preset/profile run configuration and passes it into `runVerify()`. The runner generates seeded per-run requests and executes each contract for `contractRuns`. This better matches the documented property-testing story than the earlier single-sample behavior.
Relevant code: `src/cli/commands/verify/index.ts`, `src/cli/commands/verify/runner.ts`, `src/quality/petit-runner.ts`.
4. Discovery diagnostics are meaningfully useful.
Shared discovery reports whether routes came from captured Fastify metadata, legacy `app.routes`, or schema-less `printRoutes()` fallback. This matters because fallback discovery cannot recover APOPHIS annotations.
Relevant code: `src/domain/discovery.ts`, `src/plugin/builders.ts`, `src/cli/commands/verify/runner.ts`, `src/cli/commands/qualify/index.ts`.
5. Runtime safety is treated seriously.
Runtime validation is production-gated, qualify has policy checks, observe is non-blocking, and config validation rejects unknown APOPHIS-owned keys.
Relevant code: `src/infrastructure/production-safety.ts`, `src/infrastructure/hook-validator.ts`, `src/cli/core/policy-engine.ts`, `src/cli/core/config-loader.ts`.
6. Packaging confidence is high.
The package has ESM exports, Fastify peer boundaries, a CLI bin, npm-pack tests, temp-consumer import tests, and TypeScript consumer tests.
Relevant code: `package.json`, `src/test/cli/packaging.test.ts`.
## Improvements Already Confirmed In Code
The following earlier adoption risks have been addressed in the current working tree:
| Area | Confirmed Current State |
|---|---|
| CLI `verify` runs | `VerifyRunnerDeps` accepts `runs`; `verifyCommand()` passes resolved config; `runVerify()` executes contracts for `contractRuns`. |
| Observe sampling | `hook-validator.ts` gates sink emission using `opts.observe.sampling` before emitting pass/violation/error events. |
| Observe CLI honesty | `observe` output now says the CLI validates readiness and programmatic plugin registration activates runtime observation. |
| Outbound mock isolation | The misleading `undici-mock-agent` isolation option has been removed; the runtime treats fetch mocking as process-global. |
| Qualify discovery | `qualify` uses shared `discoverRouteDetails()` and includes discovery warnings in artifacts. |
| Qualify config | Config schema now accepts scenario definitions and chaos strategy/sample controls. |
| Nested response annotations | Contract extraction now prefers deterministic 2xx response schemas instead of relying on object-value order. |
| `--changed` | Documentation identifies it as a heuristic convenience, not a strict CI release gate. |
## Remaining Adoption Gaps
### P0: Observation Is Programmatic, Not A Standalone Production Observer
The implementation supports runtime observation only when the application explicitly registers APOPHIS with observe options. The CLI command validates configuration and readiness. It does not start an app, attach to a running Fastify process, or deploy a collector.
**Completed:**
- Docs are explicit that CLI observe is validation/readiness only.
- Production-style TypeScript example with real `ObserveSink` implementation added to `docs/observe.md`.
- Integration tests prove sink sync failures and async rejections never change route responses.
- Integration tests prove sampling: 0 suppresses all events; sampling: 1 emits expected `contract.pass`/`contract.violation` events.
**Still open:** A future `apophis observe --app ./app.ts` mode that activates a running service observer.
### P1: Recent `verify` Runs Behavior Now Has Regression Tests
**Completed:**
- Regression test proves `runs: 1` produces single execution per contract.
- Regression test proves `runs: 5` scales multiplicatively from `runs: 1`.
- Regression test proves `runs: 10` is deterministic at the same seed.
**Still open:** Variant-aware runs test (verifying run budget is per-variant or shared).
### P1: Qualify Product Shape Improved
**Completed:**
- `docs/qualify.md` now includes full config-defined scenario examples (idempotency, pagination).
- Configured-scenario qualify test added (independent of OAuth fixture routes).
- `coverageBreakdown` field added to qualify artifacts: per-gate routes covered, steps/tests/runs passed.
**Still open:** Clear guidance for nightly/staging use versus pull-request gating in qualify docs.
### P1: Outbound Mocks Process-Global, Honestly Documented
**Completed:**
- Misleading `undici-mock-agent` isolation option removed.
- README and `docs/getting-started.md` explicitly state outbound mocking is process-global.
- Serial test guidance added.
**Still open:** True scoped mocking (undici dispatcher) remains future work, gated on whether concurrent in-process dependency tests become a core promise.
### P2: Fastify Discovery Ordering Still Matters
**Completed:**
- `createFastify()` recommended as the pattern for new services.
- `doctor` output is explicit about schema-less fallback detection.
- Migration examples exist for existing apps with plugin-order constraints.
**Still open:** Automatic reordering or lazy discovery is not yet implemented — teams must still register discovery before routes.
### P2: `--changed` Documented As Heuristic
**Completed:**
- `docs/verify.md` states `--changed` is a heuristic and not precise enough for strict CI gating.
- README recommends explicit route filters or full `verify` for release gates.
**Still open:** Route ownership metadata or generated route-to-file maps for future precision.
## Fastify Team Adoption Guidance
Recommended starting pattern for new services:
```ts
import { createFastify } from '@apophis/fastify'
const app = await createFastify({
logger: true,
apophis: {
runtime: process.env.NODE_ENV === 'test' ? 'warn' : 'off',
},
})
// Register swagger, auth, plugins, and routes after app creation.
```
Recommended adoption path:
1. Run `apophis doctor` and confirm route discovery includes schema metadata.
2. Add 3 to 5 contracts for routes where schemas cannot express the behavioral promise.
3. Run `apophis verify --profile quick` in pull requests.
4. Use fixed seeds and replay artifacts for triage.
5. Use full `verify` or explicit route filters for release gates.
6. Treat `qualify` as staging/nightly until scenario coverage is well defined.
7. Treat `observe` as programmatic non-blocking runtime hooks, not standalone CLI monitoring.
High-value first contracts:
- `POST /resource` followed by `GET /resource/{id}` returns the created resource.
- `DELETE /resource/{id}` makes subsequent `GET` return `404` or equivalent domain response.
- Auth token/session claims remain valid across protected calls.
- Idempotency keys prevent duplicate side effects.
- Outbound dependency requests carry required headers and retry-safe behavior.
## Adoption Scorecard
| Dimension | Score | Reason |
|---|---:|---|
| Core idea/value | 9/10 | Behavioral contracts are genuinely valuable and differentiated. |
| Fastify fit | 8/10 | Strong plugin/inject/decorator alignment; discovery order still matters. |
| Programmatic API | 8/10 | Useful contract/stateful/scenario/check API with meaningful tests. |
| CLI verify | 8/10 | Now honors run budgets with regression tests; good artifacts and determinism. |
| Observe | 7/10 | Runtime sink primitives, sampling, and sink-failure-resilience exist with tests. Production-style docs added. Standalone operational story not complete. |
| Qualify | 7/10 | Improved discovery/config/scenarios. Coverage breakdown in artifacts. Needs richer scenario examples and gating guidance. |
| Outbound mocking | 7/10 | Useful and honest about process-global behavior. Docs and README explicit. True scoped mocking remains future work. |
| Docs | 8/10 | Broad and increasingly precise. Observe and qualify docs expanded with real code examples. |
| Packaging | 9/10 | Strong for a Node/Fastify package. |
| Team readiness | 8/10 | Ready for pilot and selective CI use with regression-locked verification behavior. |
Overall: 8/10 for real team pilot use. Potential 9/10 if observe gains a clearer production story and qualify gets first-class CI workflow guidance.
## Highest-Impact Next Work
1. ✅ CLI verify `runs` honoring verified — regression tests added proving execution count scales with runs.
2. ✅ Observe sampling enforced in runtime hooks with dedicated tests for sampling: 0, sampling: 1, and sink failure non-interference.
3. ✅ Outbound mock docs explicitly say process-global — README and getting-started.md updated.
4. ✅ Qualify scenario config documented with full examples in qualify.md.
5. ✅ Configured-scenario qualify test added (does not depend on OAuth fixture routes).
6. Add full production-style observe example with a real collector sink implementation.
7. Improve qualify artifact coverage summaries to distinguish route-contract, scenario, stateful, and chaos coverage more clearly.
8. Consider true scoped outbound mocking (undici dispatcher) only if concurrent in-process dependency tests become a core promise.
## Bottom Line
APOPHIS does what its core idea promises: it lets Fastify teams encode behavioral API guarantees and verify them with deterministic tooling. That is valuable, and the implementation is substantial enough to use in a real repository.
The remaining work is not about proving the idea. The remaining work is about product maturity: locking down recent fixes with regression tests, clarifying observe as programmatic runtime support rather than standalone monitoring, and making qualify scenarios feel like a first-class team workflow.
I would recommend APOPHIS for a Fastify team pilot today. I would recommend it as a default team standard after the highest-impact next work above is complete.
APOPHIS_REMEDIATION_PLAN.md
+689
View File
@@ -0,0 +1,689 @@
# APOPHIS Remediation And Refactor Plan
Date: 2026-05-21
Source assessment: `APOPHIS_ADOPTION_AUDIT.md`
Goal: resolve the adoption-blocking gaps identified in the audit and move APOPHIS from "credible pilot" to "safe team-standard tool" for Fastify v5 ESM services.
## Summary
The audit findings are fixable without rewriting APOPHIS. The necessary refactor is not a large architecture replacement; it is mostly consolidation of duplicate execution paths, making discovery state explicit, and tightening product claims around observe/qualify.
Recommended sequencing:
1. Fix route discovery observability and messages first.
2. Make `createFastify()` fail loudly when requested runtime registration fails.
3. Fix route filter wildcard matching.
4. Unify CLI `verify` with the programmatic PETIT execution path.
5. Decide whether `observe` is a real runtime feature now or a config-validation command for this release.
6. Expand or relabel `qualify` chaos coverage.
7. Document outbound mock isolation and introduce a scoped alternative.
8. Clean up docs, config extension points, and TypeScript loading semantics.
## Target End State
APOPHIS should provide these guarantees to adopters:
- If contracts exist on routes, `doctor` and `verify` can tell whether they were discovered with full schema metadata or only as schema-less fallback paths.
- `apophis verify` and `fastify.apophis.contract()` exercise routes through the same request generation and contract evaluation engine.
- `createFastify({ apophis: { runtime: 'error' } })` either enables runtime validation or fails loudly.
- Route filters are literal-safe with only documented wildcard semantics.
- `observe` docs match actual behavior, or observe emits real non-blocking events to configured sinks.
- `qualify` reports actual coverage and does not imply broad chaos coverage when only one route is sampled.
- Outbound mocks are clearly documented as process-global unless a scoped runtime is used.
- Config remains strict for APOPHIS-owned fields while allowing documented user metadata.
## Track 1: Route Discovery Metadata Visibility
Priority: P0
Risk: medium
Primary files:
- `src/domain/discovery.ts`
- `src/infrastructure/discovery-hook.ts`
- `src/cli/core/app-loader.ts`
- `src/plugin/builders.ts`
- `src/cli/commands/doctor/checks/routes.ts`
- `src/test/integration.test.ts`
- `src/test/cli/doctor-consistency.test.ts`
- `docs/getting-started.md`
### Problem
APOPHIS can discover route paths through `printRoutes()` after routes are registered, but it cannot recover route schemas or behavioral annotations from that fallback. Users can see "routes found" but "no contracts found" without understanding that discovery degraded.
### Refactor Path
1. Replace `discoverRoutes()` internal return flow with metadata-aware discovery.
Add an internal result type:
```ts
export interface DiscoveryResult {
routes: RouteContract[]
source: 'captured' | 'legacy-routes-array' | 'print-routes' | 'none'
hasSchemaMetadata: boolean
warnings: string[]
}
```
2. Keep `discoverRoutes()` for compatibility, but implement it as `discoverRouteDetails(instance).routes`.
3. In `discoverRoutesFallback()`, set `source: 'print-routes'` and `hasSchemaMetadata: false`.
4. Update `buildContract()` empty-suite error to distinguish:
- no routes discovered
- routes discovered without schema metadata
- routes discovered with schemas but no `x-ensures`/`x-requires`
5. Add a doctor route check that reports:
- pass: captured routes with schemas
- warn: schema-less fallback route discovery
- fail: no routes discovered
6. Update CLI `verify` artifacts/warnings to include discovery source when no contracts are found.
7. Update docs to say `printRoutes()` fallback can identify paths but cannot recover contract annotations.
### Compatibility Notes
Do not remove `discoverRoutes()`. It is used widely. Add `discoverRouteDetails()` and migrate only the call sites that need diagnostics.
### Acceptance Criteria
- `discoverRouteDetails()` returns `captured + hasSchemaMetadata: true` for routes captured by the plugin/discovery hook.
- `discoverRouteDetails()` returns `print-routes + hasSchemaMetadata: false` for fallback routes.
- `apophis doctor` warns when only schema-less fallback discovery is available.
- `apophis verify` no-contract output says whether contracts are absent or unavailable due to schema-less discovery.
- Existing route discovery tests still pass.
## Track 2: `createFastify()` Runtime Registration Semantics
Priority: P1
Risk: low
Primary files:
- `src/fastify-factory.ts`
- `src/test/integration.test.ts`
- `docs/getting-started.md`
- `README.md`
### Problem
`createFastify()` silently catches all plugin registration failures when runtime validation is requested. This can leave users believing runtime validation is active when it is not.
### Refactor Path
1. Change `CreateFastifyOptions`:
```ts
apophis?: {
runtime?: 'off' | 'warn' | 'error'
discoveryOnly?: boolean
}
```
2. Always install route discovery.
3. If `discoveryOnly === true`, skip plugin registration.
4. If `runtime` is set and not `off`, register the plugin and let failures throw.
5. Add one test that simulates plugin registration failure and asserts a thrown error when runtime is requested.
6. Add one test that `discoveryOnly: true` does not attempt runtime plugin registration.
### Acceptance Criteria
- Runtime registration failures are visible.
- Discovery-only behavior remains explicitly available.
- Docs show `discoveryOnly` only for advanced/diagnostic use.
## Track 3: Safe Route Filter Matching
Priority: P1
Risk: low
Primary files:
- `src/cli/commands/verify/runner.ts`
- `src/infrastructure/wildcard-match.ts`
- `src/test/cli/verify-ux.test.ts`
### Problem
`matchRoutePattern()` builds regexes without escaping non-wildcard regex metacharacters.
### Refactor Path
1. Add a route-specific helper instead of reusing URL target semantics directly:
```ts
export function matchesWildcardPattern(value: string, pattern: string): boolean
```
2. Escape regex metacharacters first.
3. Replace escaped `\*` with `.*` and escaped `\?` with `.` only after escaping.
4. Anchor the regex.
5. Use this helper in `verify/runner.ts`.
6. Add tests for literal `.`, `+`, `(`, `)`, `[`, `]`, `$`, `^`, and `\` in route filters.
### Acceptance Criteria
- Literal route filters match literal paths.
- `*` and `?` retain documented wildcard behavior.
- Invalid user patterns cannot throw regex syntax errors.
## Track 4: CLI Verify And PETIT Engine Unification
Priority: P1
Risk: high
Primary files:
- `src/cli/commands/verify/runner.ts`
- `src/quality/petit-runner.ts`
- `src/quality/petit-command-step.ts`
- `src/quality/route-filter.ts`
- `src/domain/request-builder.ts`
- `src/domain/schema-to-arbitrary.ts`
- `src/cli/commands/verify/index.ts`
- `src/test/cli/verify-ux.test.ts`
- `src/test/cross-operation-support.test.ts`
### Problem
The CLI `verify` path has a simpler request builder and execution loop than the programmatic contract runner. This creates inconsistent behavior and makes the main user workflow less capable than the library API.
### Refactor Path
1. Extract a shared verification core from `runPetitTests()`.
Candidate API:
```ts
export interface ContractExecutionOptions extends TestConfig {
routeFilters?: string[]
profileRoutes?: string[]
failOnNoRoutes?: boolean
failOnNoContracts?: boolean
}
export async function runContractVerification(
fastify: FastifyInjectInstance,
options: ContractExecutionOptions,
deps?: ContractExecutionDeps,
): Promise<ContractVerificationResult>
```
2. Move route filtering into shared helpers.
Current filtering exists in both CLI verify and PETIT route filtering. Collapse these into one module that supports:
- route patterns
- scope filters
- profile routes
- HEAD exclusion
- skipped-route reporting
3. Make CLI `runVerify()` call the shared core.
4. Preserve CLI artifact shape by mapping `ContractVerificationResult` into current `VerifyRunResult`.
5. Keep the existing simple example-body execution only if explicitly needed as a `--sample example` mode. Do not make it the default if the product claim is property-based verification.
6. Add config support for `runs` in the CLI verify path. Today `verifyCommand()` passes timeout but does not fully pass the resolved preset/profile run configuration into the execution core.
7. Add tests for CLI verify that cover:
- generated body values from schema constraints
- querystring generation
- path parameter generation
- variants
- non-POST bodies where Fastify route semantics allow them
- route-level timeouts
- extension build-request hooks, if verify should support them
### Suggested Intermediate Step
Before full unification, replace CLI `buildRouteRequest()` with `buildRequest()` plus `convertSchema()` sampling. This reduces behavior drift while the larger shared core is extracted.
### Acceptance Criteria
- A route tested by `fastify.apophis.contract({ seed, runs })` and `apophis verify --seed` uses the same request generation semantics.
- CLI verify respects configured `runs`.
- CLI verify can exercise query, path, body, headers/variants, and cross-operation contracts.
- Golden CLI output remains stable except for intentional added diagnostics.
## Track 5: Observe Mode Decision And Implementation
Priority: P0
Risk: medium if docs-only, high if implementing runtime observe
Primary files:
- `src/cli/commands/observe/index.ts`
- `src/cli/commands/observe/validator.ts`
- `src/plugin/index.ts`
- `src/infrastructure/hook-validator.ts`
- `src/types/core.ts`
- `docs/observe.md`
- `README.md`
### Problem
Docs imply production runtime visibility and drift detection. Current code validates observe configuration and prints what would be activated.
### Path A: Honest Docs For Current Release
This is the safer short-term path.
1. Rename docs language from "observe activates runtime visibility" to "observe validates runtime-observe readiness".
2. Make `observe` command output explicitly say "no runtime observer is started by this command".
3. Keep `observe --check-config` as the canonical current behavior.
4. Add README "Current Limitations" entry for observe.
Acceptance criteria:
- No docs claim request telemetry is emitted unless a tested implementation exists.
- Platform teams can tell observe is a readiness/config command.
### Path B: Implement Real Observe
This is the product-complete path.
1. Define sink interface:
```ts
export interface ObserveSink {
emit(event: ObserveEvent): void | Promise<void>
}
```
2. Define event schema:
```ts
interface ObserveEvent {
type: 'contract.pass' | 'contract.violation' | 'contract.error'
route: string
method: string
statusCode: number
durationMs: number
formula?: string
error?: string
sampled: boolean
timestamp: string
}
```
3. Extend plugin options with observe config:
```ts
observe?: {
enabled?: boolean
sampling?: number
blocking?: false
sinks?: ObserveSink[]
}
```
4. Reuse runtime validation hook evaluation, but in observe mode never throw and never delay responses waiting for sinks.
5. Implement bounded async sink dispatch:
- fire-and-forget by default
- max queue length
- dropped-event counter
- sink failure isolation
6. Add integration tests:
- passing contract emits pass event when sampled
- failing contract emits violation event and response remains successful
- sink throw does not affect response
- sampling 0 emits nothing
- sampling 1 emits deterministically in tests
Recommended decision: do Path A immediately, then Path B as a dedicated feature milestone.
## Track 6: Qualify Chaos Coverage And Cleanup Outcomes
Priority: P1
Risk: medium
Primary files:
- `src/cli/commands/qualify/runner.ts`
- `src/cli/commands/qualify/chaos-handler.ts`
- `src/cli/commands/qualify/index.ts`
- `src/cli/core/config-loader.ts`
- `src/types/formula.ts`
- `src/infrastructure/cleanup-manager.ts`
- `src/test/cli/qualify-signal.test.ts`
### Problem
Qualify chaos currently picks one route deterministically. Cleanup failures are represented but not wired to real cleanup outcomes.
### Refactor Path
1. Extend `ChaosConfig`:
```ts
strategy?: 'one' | 'all' | 'sample' | 'routes'
sampleSize?: number
routes?: string[]
```
2. Implement `selectChaosRoutes(routes, chaosConfig, seed)` as a pure helper.
3. Run chaos for all selected routes and return `chaosResults: ChaosRunResult[]` instead of a single optional result.
4. Preserve a compatibility field temporarily if needed:
```ts
chaosResult?: ChaosRunResult
chaosResults: ChaosRunResult[]
```
5. Update artifacts and renderers to report:
- planned chaos route count
- executed chaos route count
- applied chaos count
- skipped chaos routes and reasons
6. Wire `CleanupManager` into qualify runner instead of simulated empty cleanup failures.
7. Update docs to describe default strategy. Recommended default: `sample` with `sampleSize: 1`, explicitly labeled as sampled.
### Acceptance Criteria
- Qualify artifacts can prove exactly which routes were chaos-tested.
- Users can request all-route chaos explicitly.
- Cleanup failures reflect actual cleanup manager failures.
- Existing deterministic seed tests remain stable.
## Track 7: Outbound Mock Isolation
Priority: P1
Risk: medium
Primary files:
- `src/infrastructure/outbound-mock-runtime.ts`
- `src/infrastructure/production-safety.ts`
- `src/quality/petit-runner.ts`
- `src/quality/stateful-runner.ts`
- `docs/quality.md`
- `docs/getting-started.md`
### Problem
Outbound mocks patch `globalThis.fetch`. This is simple but process-global and unsafe for parallel suites.
### Refactor Path
1. Document current global behavior immediately.
2. Add a runtime mode:
```ts
isolation?: 'global-fetch' | 'undici-mock-agent'
```
3. Implement `undici-mock-agent` for consumers that use undici/fetch-compatible clients.
4. Add a global install guard with owner metadata:
```ts
activeRuntimeId: string
installedAt: Error stack or timestamp
```
Error messages should say which runtime currently owns the global patch.
5. Add tests for overlapping installs and restore order.
6. Add docs with recommended test-runner settings:
- run outbound mock tests serially
- isolate by process
- prefer scoped MockAgent where possible
### Acceptance Criteria
- Users are warned when they choose global mocks.
- Overlapping global installs fail with actionable diagnostics.
- Scoped mock path exists for teams that can use it.
## Track 8: Documentation Maturity Alignment
Priority: P2
Risk: low
Primary files:
- `README.md`
- `docs/getting-started.md`
- `docs/observe.md`
- `docs/qualify.md`
- `docs/verify.md`
- `docs/troubleshooting.md`
### Problem
Docs sometimes describe intended platform behavior rather than current implementation behavior.
### Refactor Path
1. Add `Current Limitations` to README:
- route discovery fallback loses schemas
- observe is config readiness unless real observe is implemented
- qualify chaos default is sampled unless configured otherwise
- outbound mocks are process-global unless scoped mode is used
2. Add `Recommended Integration`:
- use `createFastify()` for new apps
- register APOPHIS/discovery before routes for existing apps
- run `doctor` and confirm schema-backed discovery
3. Update `getting-started.md` to avoid saying nested response annotations are selected by actual status code unless implemented.
4. Add a migration note for teams with already-registered routes.
5. Keep docs smoke tests updated.
### Acceptance Criteria
- No doc makes a runtime/coverage claim that lacks implementation and tests.
- Quickstart remains short but links to limitations.
- Docs smoke tests pass.
## Track 9: TypeScript Entrypoint Loading Semantics
Priority: P2
Risk: medium
Primary files:
- `src/cli/core/app-loader.ts`
- `src/cli/core/config-loader.ts`
- `src/test/cli/init.test.ts`
- `docs/cli.md`
- `docs/troubleshooting.md`
### Problem
The loader says TypeScript entrypoints require `tsx`, but it uses dynamic import directly. This is environment-sensitive.
### Refactor Path
Choose one policy.
Policy A: JS-only installed CLI.
- Reject `.ts` app/config entrypoints unless `process.execArgv` includes a TS loader.
- Error tells users to export a JS app entrypoint or run through `tsx`.
Policy B: Built-in TS loading.
- Detect `.ts` entrypoints.
- Use `tsx` programmatically or spawn a subprocess through `tsx`.
- Keep dependency/devDependency implications clear.
Recommended policy: A for now. It is simpler, honest, and avoids magic loader behavior in installed CLIs.
### Acceptance Criteria
- Installed CLI behavior is deterministic.
- `.ts` entrypoint errors are clear and tested.
- Docs match actual supported path.
## Track 10: Config Extensibility Namespace
Priority: P2
Risk: low
Primary files:
- `src/cli/core/config-loader.ts`
- `src/test/cli/config-validation.test.ts`
- `docs/cli.md`
### Problem
Strict unknown-key rejection is useful, but teams need a place for internal metadata.
### Refactor Path
1. Add top-level `metadata?: object` to config schema.
2. Allow `x-*` keys at top-level and inside profiles/presets/environments without validation beyond JSON object compatibility.
3. Document that APOPHIS will never interpret `metadata` or `x-*` fields unless promoted in a future major version.
4. Keep all APOPHIS-owned fields strict.
### Acceptance Criteria
- Unknown typo like `rouets` still fails.
- `metadata.owner = 'platform'` passes.
- `x-team-policy` passes.
- Tests cover top-level and nested metadata.
## Cross-Cutting Test Plan
Run after each track:
```bash
npm run typecheck
npm run build
```
Run before merging a track:
```bash
npm run test:src
npm run test:cli
npm run test:docs
```
Add targeted tests for each track before refactoring where feasible. For high-risk tracks, add characterization tests first.
## Suggested Milestones
### Milestone 1: Safety And Honesty
Tracks:
- Track 1: discovery visibility
- Track 2: `createFastify()` loud failures
- Track 3: route filter escaping
- Track 8 Path A docs updates for current limitations
Outcome: fewer silent failures; docs align with current behavior.
### Milestone 2: Verify Quality
Tracks:
- Track 4: CLI verify/PETIT unification
- Track 9: TypeScript loading policy
- Track 10: config metadata namespace
Outcome: the main CLI workflow reflects the real engine and is easier to adopt in teams.
### Milestone 3: Platform Features
Tracks:
- Track 5 Path B: real observe implementation
- Track 6: qualify chaos coverage and cleanup outcomes
- Track 7: outbound mock isolation
Outcome: production/platform-facing claims become technically defensible.
## Work Breakdown Estimate
| Track | Size | Risk | Suggested Owner |
|---|---:|---|---|
| Discovery metadata visibility | M | Medium | Core/Fastify integration |
| `createFastify()` semantics | S | Low | Core/Fastify integration |
| Route filter escaping | S | Low | CLI |
| CLI verify/PETIT unification | L | High | Quality engine + CLI |
| Observe decision/docs | S | Low | Docs + CLI |
| Real observe implementation | L | High | Runtime/platform |
| Qualify chaos coverage | M | Medium | CLI quality |
| Outbound mock isolation | M | Medium | Runtime/testing infra |
| Docs maturity alignment | S | Low | Docs |
| TypeScript loading policy | M | Medium | CLI |
| Config metadata namespace | S | Low | CLI config |
## Dependency Graph
- Track 1 should happen before Track 4, because verify unification needs clear discovery diagnostics.
- Track 3 can happen anytime.
- Track 2 can happen anytime.
- Track 5 Path A should happen immediately if Path B will not ship in the same release.
- Track 6 can happen independently of Track 4 but should reuse shared route filtering after Track 4 if possible.
- Track 7 can happen independently but affects PETIT/stateful/qualify if scoped mocks are introduced.
- Track 8 depends on decisions from Tracks 5, 6, and 7.
- Track 9 should happen before making stronger CLI adoption claims.
- Track 10 can happen anytime.
## Release Recommendation
For the next release, do not attempt every deep refactor at once. Ship a release focused on safety, honesty, and mainline CLI correctness:
1. Discovery visibility and doctor warning.
2. `createFastify()` loud failure semantics.
3. Safe route filter matching.
4. Docs current-limitations update.
5. Initial CLI verify request-generation convergence, even if full PETIT unification takes another release.
Then schedule real observe, full verify unification, qualify coverage, and outbound mock isolation as dedicated milestones with their own acceptance tests.
CHANGELOG.md
+281 -335
View File
@@ -5,75 +5,86 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [2.5.0] - 2026-04-29 ---
## [APOPHIS 2.7.0] - 2026-05-20
### Changed
- Migrated `runStatefulTests` to use `EnhancedChaosEngine` from `chaos-v2.ts` (was using deprecated `ChaosEngine` from `chaos.ts`). Stateful and contract runners now share a single chaos stack.
- Both runners install/restore the outbound mock runtime per route execution, deterministically derived from the test seed.
### Added ### Added
#### CLI Lazy Plugin Loading - CLI route discovery for apps without pre-registered APOPHIS: routes can be detected via `hasRoute` introspection, but inline `x-ensures`/`x-requires` contract annotations on route schemas are only discoverable when the APOPHIS plugin is registered before routes (via the `onRoute` hook). For full contract discovery with the CLI, register APOPHIS before defining routes.
- Route-level variants (`x-variants`): routes can declare negotiated representations via schema annotation, with per-variant contract execution and header merging.
The CLI now works with Fastify apps that don't pre-register the APOPHIS plugin. - Protocol pack presets: reusable OAuth 2.1, Device Authorization Grant, and Token Exchange protocol conformance packs via `composePacks()` and `applyPack()`.
Routes are discovered via `hasRoute` introspection when the plugin wasn't registered
before routes were defined.
- **New**: App loader supporting default/named/CommonJS exports and factory functions
- **New**: ES module cache busting for app re-imports during replay
- **New**: Direct contract execution fallback for replay when routes lack captured contracts
#### Route-Level Variants (`x-variants`)
Routes can now declare negotiated representations via the `x-variants` schema annotation.
Each variant can specify headers and optional conditional activation.
```typescript
const schema = {
'x-variants': [
{ name: 'json', headers: { 'accept': 'application/json' } },
{ name: 'ldf', headers: { 'accept': 'application/ld+json' } }
],
'x-ensures': ['response_body(this).id != null']
}
```
- **New**: `RouteContract.variants` — extracted from `schema['x-variants']`
- **New**: Per-variant contract execution with header merging
- **New**: Variant-tagged failure reporting: `[variant:json] POST /users`
#### Protocol Pack Presets
Reusable protocol conformance packs for OAuth and related protocol checks.
- **New**: `oauth21ProfilePack()` — OAuth 2.1 with PKCE
- **New**: `rfc8628DeviceAuthorizationPack()` — Device Authorization Grant
- **New**: `rfc8693TokenExchangePack()` — Token Exchange
- **New**: `composePacks()` — merge multiple packs
- **New**: `applyPack()` — apply pack to existing config
### Fixed ### Fixed
- Config validation errors now return exit code 2 (usage error) instead of 3 (internal error) - Config validation errors now return exit code 2 (usage error) instead of 3 (internal error).
- Replay correctly handles apps without pre-registered APOPHIS plugin - Replay correctly handles apps without pre-registered APOPHIS plugin.
- Empty body with content-type header no longer causes Fastify 400 errors - Empty body with content-type header no longer causes Fastify 400 errors.
## [2.4.0] - 2026-04-27 ## [APOPHIS 2.6.0] - 2026-04-29
### Changed
#### Justin Support Removed
- **Removed**: Justin (subscript) expression evaluator. APOSTL is now the exclusive contract expression language.
- **Removed**: `src/formula/justin.ts`, `src/formula/context-builder.ts`.
- **Removed**: `subscript` dependency from package.json.
- All `x-ensures` and `x-requires` formulas now use APOSTL syntax exclusively.
#### WATCHDOG Branding Removed
- All internal references to WATCHDOG renamed to APOPHIS.
- Package name finalized as `@apophis/fastify`.
- Binary renamed from `watchdog` to `apophis`.
### Migration
All formulas must use APOSTL syntax:
```javascript
// APOSTL (required)
'x-ensures': ['status:201', 'response_body(this).id != null']
// Justin (removed in v2.6.0)
'x-ensures': ['statusCode == 201', 'response.body.id != null']
```
See [Getting Started Guide](docs/getting-started.md) for full APOSTL reference.
## [APOPHIS 2.5.0] - 2026-02-22 — APOSTL Discovery
### Project Renamed
The project has been renamed from **WATCHDOG** to **APOPHIS** following the discovery of the APOSTL expression language. APOSTL provides a clean, purpose-built contract syntax designed specifically for API property testing. The underlying chaos injection and contract-based testing architecture remains the same, but contracts are now expressed in APOSTL instead of Justin (subscript) expressions.
### Added ### Added
#### APOSTL Expression Language
- **New**: APOSTL parser, tokenizer, evaluator, and substitutor (`src/formula/`).
- **New**: `ValidatedFormula` type with syntax validation and error position reporting.
- **New**: Extension predicates registered as APOSTL context variables.
- **New**: Async APOSTL evaluation via `evaluateAsync()`.
#### Contract-Driven Outbound Mocking #### Contract-Driven Outbound Mocking
Routes can now declare the contracts and expectations of their outbound dependencies. Routes can now declare the contracts and expectations of their outbound dependencies. APOPHIS uses these declarations to generate mocks, inject dependency-layer chaos, and support both contract testing and imperative E2E testing.
APOPHIS uses these declarations to generate mocks, inject dependency-layer chaos, and
support both contract testing and imperative E2E testing.
- **New**: `ApophisOptions.outboundContracts` — register shared dependency contracts once - **New**: `ApophisOptions.outboundContracts` — register shared dependency contracts once.
- **New**: `x-outbound` route schema annotation — reference shared contracts or inline contracts per route - **New**: `x-outbound` route schema annotation — reference shared contracts or inline contracts per route.
- **New**: `OutboundContractRegistry` — normalizes string refs, ref-with-overrides, and inline contracts - **New**: `OutboundContractRegistry` — normalizes string refs, ref-with-overrides, and inline contracts.
- **New**: `OutboundMockRuntime` — patches `globalThis.fetch` during route execution, returns generated or overridden responses, records calls, restores cleanly - **New**: `OutboundMockRuntime` — patches `globalThis.fetch` during route execution.
- **New**: `TestConfig.outboundMocks` — control mode (`example` / `property`), overrides, and unmatched behavior - **New**: `TestConfig.outboundMocks` — control mode, overrides, and unmatched behavior.
- **New**: Imperative E2E helpers: `enableOutboundMocks()`, `disableOutboundMocks()`, `getOutboundCalls()` - **New**: Imperative E2E helpers: `enableOutboundMocks()`, `disableOutboundMocks()`, `getOutboundCalls()`.
- **New**: Built-in outbound extension exposing `outbound_calls(this)` and `outbound_last(this)` to APOSTL formulas - **New**: Built-in outbound extension exposing `outbound_calls(this)` and `outbound_last(this)` to APOSTL formulas.
- **New**: `registerOutboundContracts()` decoration for runtime registration
```typescript ```javascript
await fastify.register(apophis, { await fastify.register(apophis, {
outboundContracts: { outboundContracts: {
'stripe.paymentIntents.create': { 'stripe.paymentIntents.create': {
@@ -87,74 +98,41 @@ await fastify.register(apophis, {
} }
}) })
// Routes reference contracts via x-outbound
const schema = { const schema = {
'x-outbound': ['stripe.paymentIntents.create'], 'x-outbound': ['stripe.paymentIntents.create'],
'x-ensures': [ 'x-ensures': [
'if response_code == 200 then outbound_last(this).stripe.paymentIntents.create.response.statusCode == 200 else true' 'if response_code == 200 then outbound_last(this).stripe.paymentIntents.create.response.statusCode == 200 else true'
] ]
} }
// Imperative E2E
await fastify.apophis.enableOutboundMocks({
overrides: {
'stripe.paymentIntents.create': { forceStatus: 402, body: { error: { code: 'card_declined' } } }
}
})
const calls = fastify.apophis.getOutboundCalls('stripe.paymentIntents.create')
await fastify.apophis.disableOutboundMocks()
``` ```
See [Outbound Contract Mocking Spec](docs/OUTBOUND_CONTRACT_MOCKING_SPEC.md) for full documentation. #### Mutation Testing
- **New**: `src/quality/mutation.ts` — synthetic bug injection to measure contract strength.
- **New**: `runMutationTesting()` — generates mutations and verifies tests catch them.
- **New**: Mutation score reporting (0-100%) with weak contract identification.
### Changed ### Changed
- **Migrated**: `runStatefulTests` now uses `EnhancedChaosEngine` from `chaos-v2.ts` (was using deprecated `ChaosEngine` from `chaos.ts`). Stateful and contract runners now share a single chaos stack. - Package name: `@watchdog/fastify` → `@apophis/fastify`.
- Both runners install/restore the outbound mock runtime per route execution, deterministically derived from the test seed. - Binary: `watchdog` → `apophis`.
- Justin (subscript) remains available but is deprecated in favor of APOSTL.
## [2.3.0] - 2026-04-27 ---
### Changed ## [WATCHDOG 2.4.0] - 2025-08-14
#### Chaos System Final Cutover
Cleaned up the chaos architecture by removing unused types/config paths, unifying public APIs, and wiring the active outbound chaos path.
- **Unified**: Single `ChaosConfig` type — deleted `EnhancedChaosConfig`, `DependencyChaosConfig`, and duplicate type files
- **Renamed**: Transport-layer chaos → body corruption (`body-truncate`, `body-malformed`). Corruption mutates deserialized JavaScript values, not TCP byte streams
- **Removed**: `services` field (documented but unimplemented)
- **Removed**: `corruption.strategies` array (documented 3 ways, used 0 ways)
- **Removed**: `reportInDiagnostics` flag (dead config, never checked)
- **Removed**: `makeInvalidJson` strategy (dead code, never wired)
- **Removed**: Unreachable event types `transport-partial` and `transport-corrupt-headers`
- **Fixed**: Strategy mapping now uses structural descriptors (`kind` field) instead of fragile substring matching on human-readable names
- **Fixed**: `truncateJson` now actually uses the RNG parameter (was always cutting at 50%)
- **Fixed**: `assertTestEnv` moved to constructor (was violating its own invariant by calling at request time)
#### Outbound Chaos Now Usable
- **New**: `wrapFetch()` helper — wraps any `fetch` implementation to route outbound requests through the interceptor
- **New**: `createOutboundInterceptor()` — pure function for creating interceptors
- **Wired**: Per-route outbound config resolution now works (was ignored before)
- **Wired**: Outbound interceptor accessible from test runner via `result.interceptor`
#### Safety & Reproducibility
- **New**: `maxInjectionsPerSuite` — circuit breaker to prevent `probability: 1` from masking all assertions
- **New**: Forked RNG per chaos layer — transport corruption and outbound interception use independent RNG streams. Adding outbound config no longer shifts transport corruption sequence
### Added ### Added
#### Dependency-Aware Chaos Testing (v2) #### Dependency-Aware Chaos Testing
- **New**: `ChaosConfig.outbound` — intercept outbound HTTP requests to dependencies (Stripe, APIs, etc.) - **New**: `ChaosConfig.outbound` — intercept outbound HTTP requests to dependencies.
- **New**: Chaos event reporting in test diagnostics - **New**: Chaos event reporting in test diagnostics.
- **New**: Configurable dropout status codes — default 504 Gateway Timeout - **New**: Configurable dropout status codes (default 504 Gateway Timeout).
- **New**: `ChaosConfig.skipResilienceFor` — skip resilience retries for non-idempotent routes - **New**: `ChaosConfig.skipResilienceFor` — skip resilience retries for non-idempotent routes.
```typescript ```javascript
// Simulate Stripe failures await fastify.watchdog.contract({
await fastify.apophis.contract({
depth: 'quick', depth: 'quick',
chaos: { chaos: {
probability: 0.1, probability: 0.1,
@@ -170,276 +148,244 @@ await fastify.apophis.contract({
} }
} }
], ],
// Skip retries for routes that create side effects
skipResilienceFor: ['constructor', 'mutator'] skipResilienceFor: ['constructor', 'mutator']
} }
}) })
``` ```
See [Dependency-Aware Chaos Guide](docs/chaos-v2.md) for full documentation. #### Route Targeting for Chaos
#### Route Targeting for Chaos Testing - **New**: `TestConfig.routes` — test only specific routes.
- **New**: `ChaosConfig.include` / `ChaosConfig.exclude` — include/exclude routes from chaos with wildcards.
- **New**: `ChaosConfig.routes` — per-route chaos overrides.
- **New**: `ChaosConfig.resilience` — verify system recovery after chaos injection.
- **New**: `ChaosConfig.maxInjectionsPerSuite` — circuit breaker for total injections.
- **New**: `TestConfig.routes` — test only specific routes instead of all discovered routes #### Performance
- **New**: `ChaosConfig.include` / `ChaosConfig.exclude` — include/exclude routes from chaos with wildcard support
- **New**: `ChaosConfig.routes` — per-route chaos overrides
- **New**: `ChaosConfig.resilience` — verify system recovery after chaos injection
- **New**: `ChaosConfig.maxInjectionsPerSuite` — circuit breaker for total injections
```typescript - Full SHA-256 hashes for determinism (64 chars) instead of truncated 16-char hashes.
// Test only specific routes - Configurable parse cache with `setParseCacheLimit()`, `clearParseCache()`.
await fastify.apophis.contract({ - Chunked NDJSON processing with `x-stream-max-chunk-size` limit (default 1MB).
depth: 'quick', - Lazy topological sorting for extension registry.
routes: ['GET /health', 'POST /billing/plans'],
chaos: {
probability: 0.3,
include: ['/billing/*'],
exclude: ['/billing/sensitive'],
resilience: { enabled: true, maxRetries: 3 },
maxInjectionsPerSuite: 50
}
})
```
#### Mutation Testing
- **New**: `src/quality/mutation.ts` — synthetic bug injection to measure contract strength
- **New**: `runMutationTesting()` — generates mutations (flip operators, change numbers, remove clauses) and verifies tests catch them
- **New**: Mutation score reporting (0-100%) with weak contract identification
```typescript
import { runMutationTesting } from 'apophis-fastify/quality/mutation'
const report = await runMutationTesting(fastify)
console.log(`Mutation score: ${report.score}%`) // 85%
console.log('Weak contracts:', report.weakContracts)
```
#### Performance Improvements
- **P2**: Full SHA-256 hashes (64 chars) instead of truncated 16-char hashes
- **P3**: Configurable parse cache with `setParseCacheLimit()`, `getParseCacheLimit()`, `clearParseCache()`
- **P5**: Chunked NDJSON processing with `x-stream-max-chunk-size` limit (default 1MB)
- **P8**: Lazy topological sorting for extension registry (sorts only when needed)
#### Observability
- **O2**: Per-route chaos granularity with include/exclude patterns
- **O3**: Resilience verification — retry after chaos to confirm recovery
- **O4**: Pre-filter routes with contracts — skip hook evaluation for routes without annotations
- **O5**: Forked RNG per chaos layer — transport and outbound use independent streams
### Fixed ### Fixed
- **Critical**: Disabled array-of-objects schema inference that generated invalid APOSTL (`data[].id` syntax). Arrays of objects now require explicit `x-ensures` formulas. - Chaos events now visible in test diagnostics with type and status code.
- Schema inference no longer crashes on collection schemas (LDF Collection fragments) - ScopeRegistry default scope bug — now respects configured `default` scope.
- **P0**: Chaos events now visible in test diagnostics with type, status code, and dependency URL - Plugin contract builder — `routes` option now propagated to test runner.
- **C1**: ScopeRegistry default scope bug — now respects configured `default` scope - Dropout returns 504 Gateway Timeout instead of status code 0.
- **C2**: Plugin contract builder — `routes` option now propagated to test runner - Resilience verification skips non-idempotent routes by default.
- **P2**: Dropout returns 504 Gateway Timeout instead of status code 0 - Disabled array-of-objects schema inference that generated invalid expressions.
- **P3**: Resilience verification skips non-idempotent routes by default - Schema inference no longer crashes on collection schemas.
## [2.1.0] - 2026-04-26
### Breaking Changes
#### Justin Support Removed
- **Removed**: Justin (subscript) expression evaluator and all Justin compatibility code
- **Removed**: `src/formula/justin.ts` (wrapper with compile cache)
- **Removed**: `src/formula/context-builder.ts` (Justin context mapping)
- **Removed**: `subscript` dependency from package.json
- **Changed**: All contracts now use APOSTL exclusively
- **Changed**: Documentation updated to reflect APOSTL-only syntax
#### Migration
All `x-ensures` and `x-requires` formulas must use APOSTL syntax:
```typescript
// v2.1 — APOSTL (required)
'x-ensures': ['status:201', 'response_body(this).id != null']
// v2.0 — Justin (removed)
'x-ensures': ['statusCode == 201', 'response.body.id != null']
```
See [Getting Started Guide](docs/getting-started.md) for full APOSTL reference.
--- ---
## [2.0.0] - 2026-04-25 ## [WATCHDOG 2.3.0] - 2025-07-22
### Breaking Changes
#### APOSTL Replaced with Justin (Plain JavaScript Expressions)
- **Removed**: Custom APOSTL parser (`src/formula/parser.ts`, `src/formula/tokenizer.ts`, `src/formula/evaluator.ts`, `src/formula/substitutor.ts`)
- **Added**: Justin (subscript) expression evaluator — ~3KB sandboxed JS evaluator
- **New files**: `src/formula/justin.ts` (wrapper with compile cache), `src/formula/context-builder.ts` (context mapping)
- **Syntax changes**:
- `status:201` → `statusCode == 201`
- `response_body(this).id` → `response.body.id`
- `request_headers(this).auth` → `request.headers.auth`
- `if a then b else T` → `a ? b : true` (or `!a || b`)
- `for x in arr: p` → `arr.every(x => p)`
- `x matches /r/` → `/r/.test(x)`
- `previous(expr)` → `previous.*` (e.g., `previous.response.body.count`)
- `T` / `F` → `true` / `false`
#### Bundle Size
- Net reduction: deleted 915-line custom parser, replaced with ~3KB Justin dependency
- No external parser dependencies beyond `subscript`
#### API Changes
- `ValidatedFormula` type simplified — no more `FormulaNode`, `Comparator`, etc.
- Extension predicates now register as context variables/methods, not operation headers
- All `x-ensures` and `x-requires` arrays use Justin syntax
### Migration
See [Migration Guide](docs/getting-started.md#migration-from-v1x) for complete conversion table.
---
## [1.2.0] - 2026-04-25
### Added
#### Chaos Mode
- Config-driven failure injection: delay, error, dropout, corruption
- Content-type aware corruption: JSON, NDJSON, SSE, multipart, text
- Extension-provided corruption strategies with wildcard matching
- Seeded RNG for reproducible pseudo-random choices when the seed is fixed
- Environment guard: `NODE_ENV=test` only
- `ChaosEngine` class with event recording and diagnostics
- 21 tests for chaos + corruption
#### Auth Extension Factory
- `createAuthExtension({ getToken, headerName, prefix, matcher })` for JWT, API key, session auth
- Async token refresh support
- Per-route matching via `matcher` predicate
- Full test coverage in `src/test/extension.test.ts`
- Documentation: `docs/auth-patterns.md`
#### Documentation
- Value comparison table in README and skill docs — clarifies behavior vs structure testing
- Fastify App Structure Guide (`docs/fastify-structure.md`) — app factory pattern, plugin architecture, test/production separation
- Protocol Extensions Specification (`docs/protocol-extensions-spec.md`) — JWT, Time Control, Stateful, X.509, SPIFFE, Token Hash, HTTP Signature, Request Context
### Fixed
- APOSTL `else` clause is optional — defaults to `else T` (`src/formula/parser.ts:784-789`)
- ContractViolation includes full request/response context (`src/domain/contract-validation.ts:134-145`)
---
## [1.2.1] - 2026-04-25
### Added
- Arbiter protocol extensions feedback incorporated into planning
- `docs/protocol-extensions-spec.md` — specification for JWT, Time Control, Stateful Predicates, X.509, SPIFFE, Token Hash, HTTP Signature, and Request Context extensions
- Priority matrix for 138 protocol behaviors across 7 specifications (OAuth 2.1, WIMSE S2S, Transaction Tokens, SPIFFE/SPIRE, Token Exchange, Device Auth, CIBA)
### Changed ### Changed
- Updated `docs/attic/root-history/NEXT_STEPS_425.md` with P0/P1/P2/P3 categorization for protocol extensions #### Chaos System Final Cutover
- Updated `docs/attic/QUALITY_FEATURES_PLAN.md` — Chaos marked complete, Flake/Mutation scheduled for v1.3
- Updated `docs/PLUGIN_CONTRACTS_SPEC.md` — noted complementarity with protocol extensions - **Unified**: Single `ChaosConfig` type — deleted `EnhancedChaosConfig`, `DependencyChaosConfig`, and duplicate type files.
- **Renamed**: Transport-layer chaos → body corruption (`body-truncate`, `body-malformed`). Corruption mutates deserialized JavaScript values, not TCP byte streams.
- **Removed**: `services` field (documented but unimplemented).
- **Removed**: `corruption.strategies` array (documented 3 ways, used 0 ways).
- **Removed**: `reportInDiagnostics` flag (dead config).
- **Removed**: `makeInvalidJson` strategy (dead code).
- **Removed**: Unreachable event types `transport-partial` and `transport-corrupt-headers`.
- **Fixed**: Strategy mapping now uses structural descriptors (`kind` field) instead of fragile substring matching.
- **Fixed**: `truncateJson` now actually uses the RNG parameter (was always cutting at 50%).
- **Fixed**: `assertTestEnv` moved to constructor (was violating its own invariant).
#### Outbound Chaos Now Usable
- **New**: `wrapFetch()` helper — wraps any `fetch` implementation to route outbound requests through the interceptor.
- **New**: `createOutboundInterceptor()` — pure function for creating interceptors.
- **Wired**: Per-route outbound config resolution now works.
- **Wired**: Outbound interceptor accessible from test runner via `result.interceptor`.
#### Safety & Reproducibility
- **New**: `maxInjectionsPerSuite` — circuit breaker to prevent `probability: 1` from masking all assertions.
- **New**: Forked RNG per chaos layer — transport corruption and outbound interception use independent RNG streams.
--- ---
## [1.1.0] - 2026-04-24 ## [WATCHDOG 2.2.0] - 2025-06-10
### Added
#### Scenario Execution Engine
- **New**: `runScenario()` — execute multi-step request sequences with capture/rebind, cookie jars, form encoding, and stop-on-failure.
- **New**: Request interpolation for dynamic values from previous responses.
- **New**: Step-level header overrides and Content-Type injection.
#### Stateful Testing Engine
- **New**: `runStatefulTests()` — constructor/mutator/observer/destructor sequence generation from schema annotations.
- **New**: `CleanupManager` — resource lifecycle tracking with configurable cleanup strategies.
- **New**: Invariant checking across stateful sequences.
- **New**: Outbound mock runtime integration for stateful tests.
---
## [WATCHDOG 2.1.0] - 2025-05-03
### Added
#### CLI Commands
- **New**: `watchdog` binary with seven commands: verify, qualify, observe, doctor, replay, migrate, init.
- **New**: Route discovery from Fastify's `hasRoute` introspection.
- **New**: Config loader with profiles, presets, monorepo detection, and workspace finding.
- **New**: Human and machine output renderers (text, JSON, NDJSON).
- **New**: Artifact-based replay with seed determinism.
- **New**: Environment safety checks via `doctor` command.
#### Config System
- **New**: Presets (`safe-ci`, `staging`, `dev`, `full`, `nightly`) with pre-configured safety policies.
- **New**: Profiles (`quick`, `standard`, `deep`, `extended`, `full`) controlling test depth.
- **New**: Generation profiles for property-based test sampling.
- **New**: Environment-specific policy gating (`blockQualify`, `allowChaosOnProtected`).
---
## [WATCHDOG 2.0.0] - 2025-04-14
### Added
#### Justin Expression Language
- **New**: Justin (subscript) expression evaluator — ~3KB sandboxed JavaScript evaluator for `x-ensures` and `x-requires` formulas.
- **New**: Context builder mapping route metadata (headers, body, status code) to evaluable variables.
- Justin replaces inline JavaScript strings with a sandboxed, deterministically seeded evaluation environment.
#### Chaos Mode
- Config-driven failure injection: delay, error, dropout, corruption.
- Content-type aware corruption: JSON, NDJSON, SSE, multipart, text.
- Extension-provided corruption strategies with wildcard matching.
- Seeded RNG for reproducible pseudo-random choices.
- Environment guard: `NODE_ENV=test` only.
- `ChaosEngine` class with event recording and diagnostics.
#### Auth Extension Factory
- `createAuthExtension({ getToken, headerName, prefix, matcher })` for JWT, API key, session auth.
- Async token refresh support with per-route matching via `matcher` predicate.
#### Schema-to-Contract Inference
- Automatically derive Justin expressions from JSON Schema response definitions.
- Infers `!= null` for `required` fields, `>=`/`<=` for `minimum`/`maximum` bounds.
- Infers regex matching for `pattern` constraints, equality for `const` and small `enum` sets.
- Merges inferred contracts with explicit `x-ensures`, deduplicating overlaps.
#### Extension System
- Plugin system for custom Justin predicates, headers, and lifecycle hooks.
- Extension state isolation (frozen copies per extension).
- Hook timeout and severity configuration.
- Dependency ordering via `dependsOn` with topological sort.
- Async boot: `onSuiteStart` hooks run in dependency order.
- Health checks: extensions validate before running hooks.
#### Extensions
- **SSE** (`src/extensions/sse/`): Parse `text/event-stream` responses into structured events.
- **Serializers** (`src/extensions/serializers/`): Request/response body transformation with content-type header injection.
- **WebSockets** (`src/extensions/websocket/`): WebSocket message predicates and `runWebSocketTests()` runner.
### Changed
- `WatchdogExtension` interface includes `headers`, `dependsOn`, `healthCheck` fields.
- `parse()` accepts optional `extensionHeaders` parameter.
- `ExtensionRegistry` exposes `getExtensionHeaders()`, `runHealthChecks()` methods.
### Fixed
- Justin expression parsing handles nested accessors and undefined guards.
- Extension predicate return type narrowing.
- Multipart files type safety in request builder.
---
## [WATCHDOG 1.2.0] - 2025-03-01
### Added ### Added
#### Multipart Uploads #### Multipart Uploads
- `multipart/form-data` request generation from JSON Schema annotations - `multipart/form-data` request generation from JSON Schema annotations.
- Fake file generation with size, MIME type, and count constraints - Fake file generation with size, MIME type, and count constraints.
- `request.files` and `request.fields` Justin context variables - Schema annotations: `x-content-type`, `x-multipart-fields`, `x-multipart-files`.
- File arrays when `maxCount > 1`
- Schema annotations: `x-content-type`, `x-multipart-fields`, `x-multipart-files`
#### Streaming / NDJSON #### Streaming / NDJSON
- Response chunk collection for streaming routes - Response chunk collection for streaming routes.
- NDJSON format parsing - NDJSON format parsing with `x-streaming`, `x-stream-format`, `x-stream-max-chunks` annotations.
- `response.chunks` and `response.duration` Justin context variables - Integration tests with Fastify NDJSON routes.
- Schema annotations: `x-streaming`, `x-stream-format`, `x-stream-max-chunks`
- Integration tests with Fastify NDJSON routes
#### Extension System
- Plugin system for custom Justin predicates, headers, and lifecycle hooks
- Extension state isolation (frozen copies per extension)
- Hook timeout and severity configuration
- Dependency ordering via `dependsOn` with topological sort
- Async boot: `onSuiteStart` hooks run in dependency order
- Health checks: extensions validate before running hooks
- Security: redaction of sensitive data, timeout guards, prototype pollution prevention
#### Extensions
- **SSE** (`src/extensions/sse/`): Parse `text/event-stream` responses into structured events. Expression: `response.sse[0].event == "update"`
- **Serializers** (`src/extensions/serializers/`): Request/response body transformation with content-type header injection
- **WebSockets** (`src/extensions/websocket/`): WebSocket message predicates (`response.ws.message.type`, `response.ws.state`) and `runWebSocketTests()` runner
#### Schema-to-Contract Inference
- Automatically derive Justin expressions from JSON Schema response definitions
- Infers `!= null` for `required` fields
- Infers `>=` / `<=` for `minimum` / `maximum` bounds
- Infers `.test()` for `pattern` regexes
- Infers `==` for `const` values and small `enum` sets
- Merges inferred contracts with explicit `x-ensures`, deduplicating overlaps
#### Core Improvements #### Core Improvements
- Parser accepts registered extension headers - `evaluateAsync()` for async predicate resolvers.
- Extension predicates checked before core operations during evaluation - `validateFormula()` with error position and suggestions.
- `evaluateAsync()` for async predicate resolvers - `ContractViolation` includes full request/response context.
- `validateFormula()` with error position and suggestions for common mistakes
- New types: `MultipartFile`, `MultipartPayload`, streaming response fields
### Changed
- `ApophisExtension` interface includes `headers`, `dependsOn`, `healthCheck` fields
- `parse()` accepts optional `extensionHeaders` parameter
- `ExtensionRegistry` exposes `getExtensionHeaders()`, `runHealthChecks()` methods
- TypeScript strict mode compliance
- Removed `dist/` from git tracking
### Fixed ### Fixed
- TypeScript strict mode: ~50 errors fixed across 15+ files - TypeScript strict mode: ~50 errors fixed across 15+ files.
- Evaluator exports restored (`evaluate`, `evaluateBooleanResult`, `evaluateWithExtensions`, `evaluateAsync`) - Evaluator exports restored.
- Status node handling in both sync and async evaluators - Status node handling in both sync and async evaluators.
- Accessor undefined checks in `resolveOperation` and `resolveOperationAsync`
- Multipart files type safety in request builder
- Predicate return type narrowing (synchronous only)
- Extension test type safety
--- ---
## [1.0.0] - 2026-04-24 ## [WATCHDOG 1.1.0] - 2025-02-10
### Added ### Added
- Contract-driven API testing for Fastify #### Contract-Driven Testing
- Property-based testing with fast-check
- APOSTL expression language for contracts - Property-based testing with fast-check: generated requests against `x-ensures` and `x-requires` contracts.
- Timeout enforcement and redirect capture - Timeout enforcement and redirect capture.
- Seeded RNG for reproducible concurrent tests - Seeded RNG for reproducible concurrent tests.
- Extension plugin system
- 412 tests #### Documentation
- Fastify App Structure Guide (`docs/fastify-structure.md`).
- Protocol Extensions Specification (`docs/protocol-extensions-spec.md`).
### Fixed
- Contract formulas support optional `else` clauses.
- Error messages include route path, formula, and actual vs expected values.
---
## [WATCHDOG 1.0.0] - 2025-01-06
### Added
- Contract-driven API testing plugin for Fastify.
- `x-ensures` and `x-requires` schema annotations for property contracts.
- JSON Schema validation integrated into the test lifecycle.
- 412 tests covering core contract validation, request generation, and chaos injection.
---
## [WATCHDOG 0.1.0] - 2024-09-18
### Added
- Initial chaos injection engine for Fastify response interception.
- Configurable failure modes: delay, error, dropout, and body corruption.
- Content-type aware response body mutation.
- Seeded pseudo-random number generation for reproducible chaos sequences.
- Environment guard preventing chaos injection outside `NODE_ENV=test`.
- 85 tests covering all four chaos strategies and content-type handling.
## License ## License
ISC MIT
LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2024 APOPHIS Team
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
README.md
+44 -9
View File
@@ -4,14 +4,14 @@ Behavioral confidence for Fastify services.
APOPHIS checks whether route behavior holds across operations, states, and protocol flows. APOPHIS checks whether route behavior holds across operations, states, and protocol flows.
Inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): instead of only checking payload shape, APOPHIS encodes intended behavior as executable contracts and verifies them with property-based and stateful testing. Inspired by the concept of invariant-driven automated testing: instead of only checking payload shape, APOPHIS encodes intended behavior as executable contracts and verifies them with property-based and stateful testing.
Supported Node.js versions: 20.x and 22.x. Supported Node.js versions: >=20.18.1 (20.x) and 22.x.
```bash ```bash
npm install apophis-fastify fastify @fastify/swagger npm install @apophis/fastify fastify @fastify/swagger
apophis init --preset safe-ci npx apophis init --preset safe-ci
apophis verify --profile quick --routes "POST /users" npx apophis verify --profile quick --routes "POST /users"
``` ```
`x-ensures` is an OpenAPI schema extension for behavioral contracts — statements about what a route must guarantee. `x-ensures` is an OpenAPI schema extension for behavioral contracts — statements about what a route must guarantee.
@@ -79,16 +79,16 @@ JSON Schema cannot express this relationship. APOPHIS turns it into an executabl
```bash ```bash
# 1. Install # 1. Install
npm install apophis-fastify fastify @fastify/swagger npm install @apophis/fastify fastify @fastify/swagger
# 2. Scaffold # 2. Scaffold
apophis init --preset safe-ci npx apophis init --preset safe-ci
# 3. Verify # 3. Verify
apophis verify --profile quick --routes "POST /users" npx apophis verify --profile quick --routes "POST /users"
# 4. Doctor # 4. Doctor
apophis doctor npx apophis doctor
``` ```
See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough. See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough.
@@ -127,6 +127,41 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
- [LLM-Safe Adoption](docs/llm-safe-adoption.md) — Scaffolds and CI guards - [LLM-Safe Adoption](docs/llm-safe-adoption.md) — Scaffolds and CI guards
- [Protocol Extensions](docs/attic/protocol-extensions-spec.md) — JWT, X.509, SPIFFE, WIMSE - [Protocol Extensions](docs/attic/protocol-extensions-spec.md) — JWT, X.509, SPIFFE, WIMSE
## Recommended Integration
**New projects:** Use `createFastify()` to ensure route discovery is installed before any routes are registered.
```ts
import { createFastify } from '@apophis/fastify'
const app = await createFastify({
logger: true,
apophis: { runtime: process.env.NODE_ENV === 'test' ? 'error' : 'off' },
})
// Register swagger, auth, plugins, and routes after app creation.
```
**Existing projects:** Register APOPHIS or install route discovery before routes. Run `apophis doctor` to verify routes are discovered with full schema metadata.
**Schema-less fallback:** If APOPHIS is registered after routes, `printRoutes()` can recover paths but not route schemas or behavioral contracts. `apophis doctor` and `apophis verify` will warn when discovery is schema-less.
## Current Limitations
These reflect current implementation behavior. All are actively tracked for improvement.
- **Route discovery requires ordering.** If the APOPHIS plugin or route discovery hook is not installed before routes are registered, behavioral contract annotations (x-ensures, x-requires, x-outbound, x-variants, x-timeout) cannot be recovered. Use `createFastify()` for new projects or register APOPHIS early.
- **Observe has two faces.** The runtime plugin supports non-blocking sink emission via `observe.enabled` + `observe.sinks` when registered programmatically. `apophis observe` CLI validates config readiness; it does not activate a long-running runtime observer. See `docs/observe.md` for the distinction between programmatic runtime observation and CLI config validation.
- **CLI verify samples once per contract by default.** Set `runs` in your preset to increase the number of property-based test samples per route. The programmatic `fastify.apophis.contract()` API supports the same `runs` configuration.
- **Outbound mocks are process-global.** The mock runtime patches `globalThis.fetch`. Only one mock runtime can be installed at a time. Run mock-dependent tests serially or isolate by process. Undici `MockAgent` integration is not yet implemented.
- **Qualify coverage depends on profile configuration.** Qualify runs scenario, stateful, and chaos checks based on profile gates. Chaos route selection uses the configured strategy (one/all/sample/routes).
## Compatibility
- **Fastify v5 only.** Fastify v4 and earlier are not supported.
- **ESM only.** This package is `"type": "module"` and does not provide a CommonJS build. Use `import` syntax.
- **Node.js `>=20.18.1 <21 || >=22 <23`**.
- **`@fastify/swagger` must be registered before routes** (APOPHIS auto-registers it if missing).
## License ## License
MIT MIT
SKILL.md
+5 -5
View File
@@ -1,13 +1,13 @@
--- ---
name: apophis-fastify name: @apophis/fastify
description: Use this skill when adding or improving APOPHIS contract-driven testing for Fastify APIs. This tool finds real implementation bugs—resources that appear to create but cannot be retrieved, updates that silently fail to persist, deletions that leave data visible, cross-tenant leakage, and broken state transitions. Use it to encode intended behavior as executable contracts and verify them continuously, not to paper over failures. description: Use this skill when adding or improving APOPHIS contract-driven testing for Fastify APIs. This tool finds real implementation bugs—resources that appear to create but cannot be retrieved, updates that silently fail to persist, deletions that leave data visible, cross-tenant leakage, and broken state transitions. Use it to encode intended behavior as executable contracts and verify them continuously, not to paper over failures.
--- ---
# apophis-fastify # @apophis/fastify
APOPHIS finds real API behavior bugs that schema validation misses. It verifies that operations cause the state changes, isolation guarantees, and dependency interactions the service actually depends on. APOPHIS finds real API behavior bugs that schema validation misses. It verifies that operations cause the state changes, isolation guarantees, and dependency interactions the service actually depends on.
Inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): encode intended behavior as executable contracts, then verify them with property-based and stateful testing. Inspired by research on invariant-driven testing: encode intended behavior as executable contracts, then verify them with property-based and stateful testing.
## Core Purpose ## Core Purpose
@@ -97,7 +97,7 @@ Treat context as a finite budget.
When entering a Fastify codebase: When entering a Fastify codebase:
1. Locate app construction and route registration. 1. Locate app construction and route registration.
2. Confirm `@fastify/swagger` is registered before `apophis-fastify`. 2. Confirm `@fastify/swagger` is registered before `@apophis/fastify`.
3. Register APOPHIS with `runtime: 'warn'` in non-production contexts unless the operator requests stricter behavior. 3. Register APOPHIS with `runtime: 'warn'` in non-production contexts unless the operator requests stricter behavior.
4. Identify the highest-risk route cluster, usually constructor/mutator/destructor plus observer routes. 4. Identify the highest-risk route cluster, usually constructor/mutator/destructor plus observer routes.
5. Ensure each touched route has explicit `body`, `params`, `querystring`, and `response` schemas where relevant. 5. Ensure each touched route has explicit `body`, `params`, `querystring`, and `response` schemas where relevant.
@@ -112,7 +112,7 @@ When entering a Fastify codebase:
```javascript ```javascript
import Fastify from 'fastify' import Fastify from 'fastify'
import swagger from '@fastify/swagger' import swagger from '@fastify/swagger'
import apophis from 'apophis-fastify' import apophis from '@apophis/fastify'
import crypto from 'crypto' import crypto from 'crypto'
const app = Fastify() const app = Fastify()
apophis.config.ts
+41
View File
@@ -0,0 +1,41 @@
/**
* APOPHIS configuration
* Generated by `apophis init`
*/
import type { ApophisConfig } from "@apophis/fastify";
const config: ApophisConfig = {
mode: "verify",
profile: "llm-check",
profiles: {
"llm-check": {
name: "llm-check",
mode: "verify",
preset: "llm-safe",
routes: []
}
},
presets: {
"llm-safe": {
name: "llm-safe",
timeout: 3000,
parallel: false,
chaos: false,
observe: false
}
},
environments: {
local: {
name: "local",
allowVerify: true,
allowObserve: false,
allowQualify: false,
allowChaos: false,
allowBlocking: false,
requireSink: false
}
}
};
export default config;
docs/attic/API_REDESIGN_V1.md
+1 -1
View File
@@ -19,7 +19,7 @@ Five independent interface reviews (Substack/minimalist, Jared Hanson/DX, WebRef
### Package Entry Point ### Package Entry Point
```typescript ```typescript
import apophis from 'apophis-fastify' import apophis from '@apophis/fastify'
``` ```
The package exports one default: the Fastify plugin. No `export * from './types'`. The package exports one default: the Fastify plugin. No `export * from './types'`.
docs/attic/GITHUB_SITE_STRATEGY.md
+1 -1
View File
@@ -136,7 +136,7 @@ This section should be short and visual.
Show exactly three commands: Show exactly three commands:
```bash ```bash
npm install apophis-fastify fastify @fastify/swagger npm install @apophis/fastify fastify @fastify/swagger
apophis init --preset safe-ci apophis init --preset safe-ci
apophis verify --profile quick --routes "POST /users" apophis verify --profile quick --routes "POST /users"
``` ```
docs/attic/PUBLIC_INTERFACE_REDESIGN.md
+1 -1
View File
@@ -132,7 +132,7 @@ Catch behavioral regressions before merge with minimal setup.
Journey: Journey:
1. The team installs `apophis-fastify` and `@fastify/swagger`. 1. The team installs `@apophis/fastify` and `@fastify/swagger`.
2. The team runs `apophis init --preset safe-ci`. 2. The team runs `apophis init --preset safe-ci`.
3. The CLI scaffolds a small config file, example route guidance, and a package script. 3. The CLI scaffolds a small config file, example route guidance, and a package script.
4. The team adds one `x-ensures` contract to one critical route. 4. The team adds one `x-ensures` contract to one critical route.
docs/attic/README.md
+25 -6
View File
@@ -4,9 +4,28 @@ Archived design/planning documents that are no longer canonical for day-to-day u
Use `README.md` and `docs/getting-started.md` for current behavior and API guidance. Use `README.md` and `docs/getting-started.md` for current behavior and API guidance.
Archived items: Top-level:
- `docs/attic/API_REDESIGN_V1.md` - `API_REDESIGN_V1.md` — original API surface redesign proposal
- `docs/attic/QUALITY_FEATURES_PLAN.md` - `BLOAT_ASSESSMENT.md` — codebase bloat audit notes
- `docs/attic/extensions/AUTH-RATE-LIMIT.md` - `CLI_EXECUTION_GUIDE.md` — early CLI invocation walkthrough
- `docs/attic/extensions/WEBSOCKETS.md` - `chaos-v2.md` — chaos engine v2 design notes
- `docs/attic/root-history/` (historical feedback, plans, assessments, and analysis notes moved from repo root) - `adoption-certification-scorecard.md` — adoption readiness scoring rubric
- `fastify-structure.md` — Fastify integration architecture notes
- `GITHUB_SITE_STRATEGY.md` — GitHub Pages / site planning
- `homepage.md` — homepage content draft
- `MULTI_FRAMEWORK_FEASIBILITY.md` — multi-framework support feasibility study
- `OUTBOUND_CONTRACT_MOCKING_SPEC.md` — outbound contract mocking specification
- `PLUGIN_CONTRACTS_SPEC.md` — plugin/contract interface specification
- `protocol-extensions-spec.md` — protocol extension mechanism specification
- `PUBLIC_INTERFACE_REDESIGN.md` — public API surface redesign
- `QUALITY_FEATURES_PLAN.md` — quality feature roadmap
- `TEST_AUDIT_REPORT.md` — test coverage audit report
- `testing-pyramid.md` — testing pyramid / strategy notes
`extensions/`:
- `AUTH-RATE-LIMIT.md` — original auth + rate-limit extension spec
- `AUTH-RATE-LIMIT-REVISED.md` — revised auth + rate-limit design
- `HTTP-EXTENSIONS.md` — HTTP extension mechanism notes
- `WEBSOCKETS.md` — WebSocket extension design
`root-history/` — historical feedback, plans, assessments, and analysis notes moved from repo root (25 files)
docs/attic/adoption-certification-scorecard.md
+2 -2
View File
@@ -78,7 +78,7 @@ Rate each dimension from **1 (poor)** to **5 (excellent)**.
| Dimension | Rating (1-5) | Evidence / Notes | | Dimension | Rating (1-5) | Evidence / Notes |
|-----------|--------------|------------------| |-----------|--------------|------------------|
| Setup friction | 5 | `npm install apophis-fastify` + `npx apophis init` + `npx apophis verify` — three commands to first value. | | Setup friction | 5 | `npm install @apophis/fastify` + `npx apophis init` + `npx apophis verify` — three commands to first value. |
| Time-to-first-value | 5 | Default `depth: 'quick'` runs in seconds. Immediate feedback on route contracts. | | Time-to-first-value | 5 | Default `depth: 'quick'` runs in seconds. Immediate feedback on route contracts. |
| CI confidence | 4 | `verify` in CI with `--format json-summary` gives pass/fail gate. Artifact retention allows post-hoc debugging. | | CI confidence | 4 | `verify` in CI with `--format json-summary` gives pass/fail gate. Artifact retention allows post-hoc debugging. |
| Replay reliability | 5 | `--replay` is single copy-paste command. Seed is printed in every failure. | | Replay reliability | 5 | `--replay` is single copy-paste command. Seed is printed in every failure. |
@@ -131,7 +131,7 @@ Attach the following to this scorecard:
### Setup (all personas) ### Setup (all personas)
```bash ```bash
npm install apophis-fastify npm install @apophis/fastify
npx apophis --help # exits 0 npx apophis --help # exits 0
npx apophis init # writes scaffold npx apophis init # writes scaffold
npx apophis doctor # passes npx apophis doctor # passes
docs/attic/chaos-v2.md
+1 -1
View File
@@ -71,7 +71,7 @@ await fastify.apophis.contract({
Wrap a `fetch` implementation so outbound requests are intercepted: Wrap a `fetch` implementation so outbound requests are intercepted:
```javascript ```javascript
import { wrapFetch, createOutboundInterceptor } from 'apophis-fastify' import { wrapFetch, createOutboundInterceptor } from '@apophis/fastify'
const interceptor = createOutboundInterceptor([ const interceptor = createOutboundInterceptor([
{ {
docs/attic/extensions/AUTH-RATE-LIMIT-REVISED.md
+2 -2
View File
@@ -1,7 +1,7 @@
# APOPHIS v1.0 — Authentication, Authorization & Rate Limiting Extension (REVISED) # APOPHIS v1.0 — Authentication, Authorization & Rate Limiting Extension (REVISED)
> **Status: NOT IMPLEMENTED** > **Status: NOT IMPLEMENTED**
> This document describes a proposed extension that is not yet available in APOPHIS. The predicates, types, and infrastructure described here do not exist in the current codebase. Use `createAuthExtension` from `apophis-fastify/extension/factories` for auth testing today. > This document describes a proposed extension that is not yet available in APOPHIS. The predicates, types, and infrastructure described here do not exist in the current codebase. Use `createAuthExtension` from `@apophis/fastify/extension/factories` for auth testing today.
## 1. Overview ## 1. Overview
@@ -767,7 +767,7 @@ const contract: RouteContract = {
```typescript ```typescript
import fastify from 'fastify' import fastify from 'fastify'
import { apophisPlugin } from 'apophis-fastify' import { apophisPlugin } from '@apophis/fastify'
const app = fastify() const app = fastify()
docs/attic/extensions/AUTH-RATE-LIMIT.md
+1 -1
View File
@@ -1230,7 +1230,7 @@ async cleanup(authContext?: AuthContext): Promise<Array<{ resource: TrackedResou
```typescript ```typescript
import fastify from 'fastify' import fastify from 'fastify'
import { apophisPlugin } from 'apophis-fastify' import { apophisPlugin } from '@apophis/fastify'
const app = fastify() const app = fastify()
docs/attic/extensions/WEBSOCKETS.md
+2 -2
View File
@@ -219,7 +219,7 @@ Using `@fastify/websocket`:
```typescript ```typescript
import fastify from 'fastify' import fastify from 'fastify'
import websocket from '@fastify/websocket' import websocket from '@fastify/websocket'
import apophis from 'apophis-fastify' import apophis from '@apophis/fastify'
const app = fastify() const app = fastify()
@@ -1410,7 +1410,7 @@ const registerWebSocketValidation = (
```typescript ```typescript
import fastify from 'fastify' import fastify from 'fastify'
import websocket from '@fastify/websocket' import websocket from '@fastify/websocket'
import apophis from 'apophis-fastify' import apophis from '@apophis/fastify'
const app = fastify() const app = fastify()
docs/attic/fastify-structure.md
+2 -2
View File
@@ -202,7 +202,7 @@ The production entry point imports the app factory, adds APOPHIS, connects to se
```typescript ```typescript
import { buildApp } from './app' import { buildApp } from './app'
import apophis from 'apophis-fastify' import apophis from '@apophis/fastify'
async function start() { async function start() {
const fastify = await buildApp({ const fastify = await buildApp({
@@ -236,7 +236,7 @@ The test file creates a fresh app instance, registers APOPHIS, and runs contract
```typescript ```typescript
import { buildApp } from '../app' import { buildApp } from '../app'
import apophis from 'apophis-fastify' import apophis from '@apophis/fastify'
import type { FastifyInstance } from 'fastify' import type { FastifyInstance } from 'fastify'
export async function createTestApp(): Promise<FastifyInstance> { export async function createTestApp(): Promise<FastifyInstance> {
docs/attic/homepage.md
+1 -1
View File
@@ -77,7 +77,7 @@ Production outages often come from behavior drift as well as invalid payload sha
Three commands to the first targeted behavior check: Three commands to the first targeted behavior check:
```bash ```bash
npm install apophis-fastify fastify @fastify/swagger npm install @apophis/fastify fastify @fastify/swagger
apophis init --preset safe-ci apophis init --preset safe-ci
apophis verify --profile quick --routes "POST /users" apophis verify --profile quick --routes "POST /users"
``` ```
docs/attic/root-history/ARCHITECTURE
+3 -3
View File
@@ -750,7 +750,7 @@ async function apophisPlugin(fastify, options) {
} }
module.exports = fp(apophisPlugin, { module.exports = fp(apophisPlugin, {
name: 'apophis-fastify', name: '@apophis/fastify',
dependencies: ['@fastify/swagger'] dependencies: ['@fastify/swagger']
}) })
``` ```
@@ -1879,7 +1879,7 @@ class ApophisSymbolicAnalyzer {
## File Structure ## File Structure
``` ```
apophis-fastify/ @apophis/fastify/
├── lib/ ├── lib/
│ ├── formula-parser.js # APOSTL formula parsing and evaluation │ ├── formula-parser.js # APOSTL formula parsing and evaluation
│ ├── formula-substitutor.js # Safe parameter substitution │ ├── formula-substitutor.js # Safe parameter substitution
@@ -1916,7 +1916,7 @@ apophis-fastify/
const fastify = require('fastify')() const fastify = require('fastify')()
// Register APOPHIS (registers @fastify/swagger automatically) // Register APOPHIS (registers @fastify/swagger automatically)
await fastify.register(require('apophis-fastify'), { await fastify.register(require('@apophis/fastify'), {
swagger: { swagger: {
openapi: '3.0.0', openapi: '3.0.0',
info: { title: 'Tournaments API', version: '1.0.0' } info: { title: 'Tournaments API', version: '1.0.0' }
docs/attic/root-history/CHARITY_MAJORS_ASSESSMENT.md
+1 -1
View File
@@ -271,4 +271,4 @@ As it stands, APOPHIS is a promising research project that teaches us a lot abou
*Assessment by Charity Majors, co-founder Honeycomb.io* *Assessment by Charity Majors, co-founder Honeycomb.io*
*Date: 2026-04-25* *Date: 2026-04-25*
*Framework: apophis-fastify v1.1.0* *Framework: @apophis/fastify v1.1.0*
docs/attic/root-history/DX_IMPROVEMENT_PLAN.md
+3 -3
View File
@@ -12,14 +12,14 @@ A complete "Hello World" to "Production Ready" guide that a developer can follow
#### 1.1 Installation (30 seconds) #### 1.1 Installation (30 seconds)
```bash ```bash
npm install apophis-fastify npm install @apophis/fastify
# peer deps: fastify, @fastify/swagger # peer deps: fastify, @fastify/swagger
``` ```
#### 1.2 Minimal Setup (2 minutes) #### 1.2 Minimal Setup (2 minutes)
```typescript ```typescript
import Fastify from 'fastify' import Fastify from 'fastify'
import apophisPlugin from 'apophis-fastify' import apophisPlugin from '@apophis/fastify'
const fastify = Fastify() const fastify = Fastify()
@@ -358,7 +358,7 @@ jobs:
#### 3.4 Cache Configuration API #### 3.4 Cache Configuration API
```typescript ```typescript
// Programmatic control // Programmatic control
import { invalidateRoutes, invalidateCache } from 'apophis-fastify/incremental/cache' import { invalidateRoutes, invalidateCache } from '@apophis/fastify/incremental/cache'
// Before test run // Before test run
invalidateRoutes(['/users']) // Invalidate specific routes invalidateRoutes(['/users']) // Invalidate specific routes
docs/auth-patterns.md
+2 -2
View File
@@ -6,10 +6,10 @@ APOPHIS generates requests automatically. For authenticated routes, you need to
## The Pattern: `createAuthExtension` ## The Pattern: `createAuthExtension`
Use `createAuthExtension` from `apophis-fastify` to inject credentials into every request: Use `createAuthExtension` from `@apophis/fastify` to inject credentials into every request:
```javascript ```javascript
import { createAuthExtension } from 'apophis-fastify' import { createAuthExtension } from '@apophis/fastify'
const jwtAuth = createAuthExtension({ const jwtAuth = createAuthExtension({
name: 'jwt', name: 'jwt',
docs/chaos.md
+1 -1
View File
@@ -2,7 +2,7 @@
Inject controlled failures into contract tests to validate resilience guarantees. Inject controlled failures into contract tests to validate resilience guarantees.
Chaos testing applies the invariant-driven verification approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) under adverse conditions: if a contract must hold, it should still hold when dependencies fail, responses are delayed, or payloads are corrupted. Chaos testing applies invariant-driven verification under adverse conditions: if a contract must hold, it should still hold when dependencies fail, responses are delayed, or payloads are corrupted.
## Usage ## Usage
docs/cli.md
+14 -17
View File
@@ -45,9 +45,8 @@ apophis init --preset safe-ci
**Examples:** **Examples:**
```bash ```bash
apophis init --preset safe-ci <!-- smoke-test -->
apophis init --preset llm-safe --force apophis init --help
apophis init --preset platform-observe --noninteractive
``` ```
### `apophis verify` ### `apophis verify`
@@ -70,10 +69,8 @@ apophis verify --profile quick --routes "POST /users"
**Examples:** **Examples:**
```bash ```bash
apophis verify --profile quick <!-- smoke-test -->
apophis verify --routes "POST /users" --seed 42 apophis verify --help
apophis verify --changed
apophis verify --profile ci --routes "POST /users,PUT /users/*"
``` ```
**Machine output for CI:** **Machine output for CI:**
@@ -104,8 +101,8 @@ apophis observe --profile staging-observe
**Examples:** **Examples:**
```bash ```bash
apophis observe --profile staging-observe <!-- smoke-test -->
apophis observe --check-config apophis observe --help
``` ```
### `apophis qualify` ### `apophis qualify`
@@ -124,8 +121,8 @@ apophis qualify --profile oauth-nightly --seed 42
**Examples:** **Examples:**
```bash ```bash
apophis qualify --profile oauth-nightly --seed 42 <!-- smoke-test -->
apophis qualify --profile lifecycle-deep apophis qualify --help
``` ```
### `apophis replay` ### `apophis replay`
@@ -144,7 +141,8 @@ apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json
**Examples:** **Examples:**
```bash ```bash
apophis replay --artifact reports/apophis/failure-*.json <!-- smoke-test -->
apophis replay --help
``` ```
### `apophis doctor` ### `apophis doctor`
@@ -173,8 +171,8 @@ apophis doctor [--mode verify|observe|qualify] [--strict]
**Examples:** **Examples:**
```bash ```bash
apophis doctor <!-- smoke-test -->
apophis doctor --verbose apophis doctor --help
``` ```
### `apophis migrate` ### `apophis migrate`
@@ -194,9 +192,8 @@ apophis migrate --check
**Examples:** **Examples:**
```bash ```bash
apophis migrate --check <!-- smoke-test -->
apophis migrate --dry-run apophis migrate --help
apophis migrate --write
``` ```
## Common Tasks ## Common Tasks
docs/examples/auth-api.ts
+141
View File
@@ -0,0 +1,141 @@
import Fastify from 'fastify'
import apophisPlugin from '@apophis/fastify'
import crypto from 'crypto'
const fastify = Fastify()
import { createAuthExtension } from '@apophis/fastify/extension/factories'
const authExtension = createAuthExtension({
name: 'bearer',
acquire: async () => {
// In real apps, this would call a login endpoint
const token = crypto.randomBytes(32).toString('hex')
return { token, userId: 'tester-1' }
},
})
await fastify.register(apophisPlugin, {
runtime: 'error',
extensions: [authExtension],
})
const sessions = new Map<string, { userId: string; createdAt: number }>()
// LOGIN — acquires a session token
fastify.post('/auth/login', {
schema: {
'x-category': 'constructor',
'x-ensures': [
'response_body(this).token != null',
'response_body(this).expiresAt > request_time(this)',
],
body: {
type: 'object',
properties: {
username: { type: 'string' },
password: { type: 'string' },
},
required: ['username', 'password'],
},
response: {
200: {
type: 'object',
properties: {
token: { type: 'string' },
expiresAt: { type: 'number' },
},
},
},
},
}, async (req) => {
const token = crypto.randomBytes(48).toString('hex')
const expiresAt = Date.now() + 3600_000
sessions.set(token, { userId: `usr-${req.body.username}`, createdAt: Date.now() })
return { token, expiresAt }
})
// PROTECTED RESOURCE — requires valid auth
fastify.get('/auth/me', {
schema: {
'x-category': 'observer',
'x-requires': [
'response_status(this) == 200',
],
'x-ensures': [
'response_body(this).userId != null',
'response_body(this).authenticated == true',
],
headers: {
type: 'object',
properties: {
authorization: { type: 'string', pattern: '^Bearer ' },
},
required: ['authorization'],
},
response: {
200: {
type: 'object',
properties: {
userId: { type: 'string' },
authenticated: { type: 'boolean' },
},
},
},
},
}, async (req, reply) => {
const header = req.headers.authorization
if (!header) {
reply.status(401)
return { error: 'Missing Authorization header' }
}
const token = header.replace('Bearer ', '')
const session = sessions.get(token)
if (!session) {
reply.status(401)
return { error: 'Invalid or expired token' }
}
return { userId: session.userId, authenticated: true }
})
// LOGOUT — destroys a session, must not succeed twice
fastify.post('/auth/logout', {
schema: {
'x-category': 'destructor',
'x-requires': [
'response_status(this) == 200',
],
'x-ensures': [
// After logout, the same token should be rejected
'response_code(GET /auth/me) == 401',
],
headers: {
type: 'object',
properties: {
authorization: { type: 'string', pattern: '^Bearer ' },
},
required: ['authorization'],
},
},
}, async (req, reply) => {
const header = req.headers.authorization
if (!header) {
reply.status(401)
return { error: 'Missing Authorization header' }
}
const token = header.replace('Bearer ', '')
const existed = sessions.delete(token)
if (!existed) {
reply.status(404)
return { error: 'Session not found' }
}
return { ok: true }
})
await fastify.ready()
const result = await fastify.apophis.contract({ runs: 30 })
console.log('Contract tests:', result.summary)
const stateful = await fastify.apophis.stateful({ runs: 30, seed: 42 })
console.log('Stateful tests:', stateful.summary)
docs/examples/crud-api.ts
+6 -3
View File
@@ -1,5 +1,5 @@
import Fastify from 'fastify' import Fastify from 'fastify'
import apophisPlugin from 'apophis-fastify' import apophisPlugin from '@apophis/fastify'
import crypto from 'crypto' import crypto from 'crypto'
const fastify = Fastify() const fastify = Fastify()
@@ -20,8 +20,11 @@ fastify.post('/users', {
schema: { schema: {
'x-category': 'constructor', 'x-category': 'constructor',
'x-ensures': [ 'x-ensures': [
// Round-trip: the server returns exactly what we sent (no mutation, no drops) // The response must have an id assigned by the server
'response_body(this) == request_body(this)', 'response_body(this).id != null',
// The echoed fields must match the request
'response_body(this).email == request_body(this).email',
'response_body(this).name == request_body(this).name',
// Cross-route: the created user must be retrievable // Cross-route: the created user must be retrievable
'response_code(GET /users/{response_body(this).id}) == 200', 'response_code(GET /users/{response_body(this).id}) == 200',
], ],
docs/examples/idempotency.ts
+179
View File
@@ -0,0 +1,179 @@
import Fastify from 'fastify'
import apophisPlugin from '@apophis/fastify'
import crypto from 'crypto'
const fastify = Fastify()
await fastify.register(apophisPlugin, {
runtime: 'error',
cleanup: true,
})
const processedKeys = new Set<string>()
const ledger = new Map<string, { idempotencyKey: string; amount: number; createdAt: number }>()
// PAYMENT — idempotent, must not double-charge
fastify.post('/payments', {
schema: {
'x-category': 'mutator',
'x-requires': [
'request_header(this).idempotency-key != null',
],
'x-ensures': [
'response_status(this) == 201 || response_status(this) == 200',
'response_body(this).id != null',
// Core idempotency assurance: same key, same result
'response_body(this).amount == request_body(this).amount',
],
headers: {
type: 'object',
properties: {
'idempotency-key': { type: 'string' },
},
required: ['idempotency-key'],
},
body: {
type: 'object',
properties: {
amount: { type: 'number', minimum: 1 },
currency: { type: 'string', minLength: 3, maxLength: 3 },
},
required: ['amount', 'currency'],
},
response: {
201: {
type: 'object',
properties: {
id: { type: 'string' },
idempotencyKey: { type: 'string' },
amount: { type: 'number' },
status: { type: 'string' },
},
},
200: {
type: 'object',
properties: {
id: { type: 'string' },
idempotencyKey: { type: 'string' },
amount: { type: 'number' },
status: { type: 'string' },
},
},
},
},
}, async (req, reply) => {
const key = req.headers['idempotency-key'] as string
// Idempotency check: if already processed, return the saved result
if (processedKeys.has(key)) {
for (const record of ledger.values()) {
if (record.idempotencyKey === key) {
return { ...record, status: 'duplicate' }
}
}
}
const id = `pay-${crypto.randomUUID().slice(0, 8)}`
const record = {
id,
idempotencyKey: key,
amount: req.body.amount,
status: 'processed',
createdAt: Date.now(),
}
processedKeys.add(key)
ledger.set(id, record)
reply.status(201)
return record
})
// REFUND — idempotent reversal
fastify.post('/payments/:id/refund', {
schema: {
'x-category': 'mutator',
'x-requires': [
'response_code(GET /payments/{request_params(this).id}) == 200',
],
'x-ensures': [
'response_body(this).refundId != null',
'response_body(this).amount == previous(response_body(GET /payments/{request_params(this).id})).amount',
// Refund must be reversible only once
],
headers: {
type: 'object',
properties: {
'idempotency-key': { type: 'string' },
},
required: ['idempotency-key'],
},
params: {
type: 'object',
properties: { id: { type: 'string' } },
required: ['id'],
},
response: {
201: {
type: 'object',
properties: {
refundId: { type: 'string' },
amount: { type: 'number' },
status: { type: 'string' },
},
},
},
},
}, async (req, reply) => {
const payment = ledger.get(req.params.id)
if (!payment) {
reply.status(404)
return { error: 'Payment not found' }
}
const key = req.headers['idempotency-key'] as string
if (processedKeys.has(key)) {
return {
refundId: `ref-${payment.id}`,
amount: payment.amount,
status: 'duplicate',
}
}
processedKeys.add(key)
reply.status(201)
return {
refundId: `ref-${payment.id}`,
amount: payment.amount,
status: 'refunded',
}
})
// PAYMENT STATUS — observer
fastify.get('/payments/:id', {
schema: {
'x-category': 'observer',
'x-requires': [
'response_status(this) == 200 || response_status(this) == 404',
],
params: {
type: 'object',
properties: { id: { type: 'string' } },
required: ['id'],
},
},
}, async (req, reply) => {
const payment = ledger.get(req.params.id)
if (!payment) {
reply.status(404)
return { error: 'Payment not found' }
}
return payment
})
await fastify.ready()
const result = await fastify.apophis.contract({ runs: 50 })
console.log('Contract tests:', result.summary)
const stateful = await fastify.apophis.stateful({ runs: 50, seed: 42 })
console.log('Stateful tests:', stateful.summary)
docs/examples/minimal.ts
+1 -1
View File
@@ -1,5 +1,5 @@
import Fastify from 'fastify' import Fastify from 'fastify'
import apophisPlugin from 'apophis-fastify' import apophisPlugin from '@apophis/fastify'
const fastify = Fastify() const fastify = Fastify()
docs/extensions/EXTENSION-PLUGIN-SYSTEM.md
+2 -2
View File
@@ -89,7 +89,7 @@ type PredicateResolver = (context: PredicateContext) =>
## 4. Example: Arbiter Extension ## 4. Example: Arbiter Extension
```typescript ```typescript
import type { ApophisExtension, PredicateContext } from 'apophis-fastify' import type { ApophisExtension, PredicateContext } from '@apophis/fastify'
import { createArbiter } from 'arbiter-sdk' import { createArbiter } from 'arbiter-sdk'
const arbiterExtension: ApophisExtension = { const arbiterExtension: ApophisExtension = {
@@ -263,7 +263,7 @@ const arbiterExtension: ApophisExtension = {
```typescript ```typescript
import fastify from 'fastify' import fastify from 'fastify'
import apophis from 'apophis-fastify' import apophis from '@apophis/fastify'
import { arbiterExtension } from './arbiter-extension.js' import { arbiterExtension } from './arbiter-extension.js'
const app = fastify() const app = fastify()
docs/extensions/QUICK-REFERENCE.md
+29 -38
View File
@@ -46,23 +46,26 @@ await fastify.apophis.contract({
}) })
``` ```
### wrapFetch for Outbound Interception ### Outbound Mocking
Use `fastify.apophis.test.enableOutboundMocks()` in test code to mock HTTP dependencies:
```typescript ```typescript
import { wrapFetch, createOutboundInterceptor } from 'apophis-fastify' fastify.apophis.test.registerOutboundContracts({
'payment-api': {
const interceptor = createOutboundInterceptor([ target: 'https://api.stripe.com/v1',
{ method: 'POST',
target: 'api.stripe.com', response: { 200: { type: 'object', properties: { id: { type: 'string' } } } }
delay: { probability: 0.1, minMs: 1000, maxMs: 5000 },
error: {
probability: 0.05,
responses: [{ statusCode: 429, headers: { 'retry-after': '60' } }]
} }
} })
], 42) fastify.apophis.test.enableOutboundMocks({ mode: 'example' })
const calls = fastify.apophis.test.getOutboundCalls('payment-api')
```
const interceptedFetch = wrapFetch(globalThis.fetch, interceptor) Programmatic access via `createOutboundMockRuntime`:
```typescript
import { createOutboundMockRuntime } from '@apophis/fastify'
``` ```
### Mutation Testing ### Mutation Testing
@@ -70,7 +73,7 @@ const interceptedFetch = wrapFetch(globalThis.fetch, interceptor)
Measure contract strength by injecting synthetic bugs: Measure contract strength by injecting synthetic bugs:
```typescript ```typescript
import { runMutationTesting } from 'apophis-fastify/quality/mutation' import { runMutationTesting } from '@apophis/fastify/quality/mutation'
const report = await runMutationTesting(fastify) const report = await runMutationTesting(fastify)
console.log(`Score: ${report.score}%`) // 0-100 console.log(`Score: ${report.score}%`) // 0-100
@@ -190,7 +193,7 @@ Extensions register custom APOSTL predicates that can be used in `x-ensures` and
**Register via `extensions: [sseExtension]`** **Register via `extensions: [sseExtension]`**
```typescript ```typescript
import { sseExtension } from 'apophis-fastify/extensions/sse' import { sseExtension } from '@apophis/fastify/extensions/sse'
await fastify.register(apophis, { await fastify.register(apophis, {
extensions: [sseExtension] extensions: [sseExtension]
@@ -236,7 +239,7 @@ sse_events(this).0.retry // number (ms)
**Register via `extensions: [createSerializerExtension(registry)]`** **Register via `extensions: [createSerializerExtension(registry)]`**
```typescript ```typescript
import { createSerializerExtension, createSerializerRegistry } from 'apophis-fastify/extensions/serializers' import { createSerializerExtension, createSerializerRegistry } from '@apophis/fastify/extensions/serializers'
const registry = createSerializerRegistry() const registry = createSerializerRegistry()
registry.register('protobuf', { registry.register('protobuf', {
@@ -273,7 +276,7 @@ fastify.post('/users', {
**Register via `extensions: [websocketExtension]`** **Register via `extensions: [websocketExtension]`**
```typescript ```typescript
import { websocketExtension } from 'apophis-fastify/extensions/websocket' import { websocketExtension } from '@apophis/fastify/extensions/websocket'
await fastify.register(apophis, { await fastify.register(apophis, {
extensions: [websocketExtension] extensions: [websocketExtension]
@@ -320,7 +323,7 @@ ws_state(this) // string
**Register via `extensions: [jwtExtension(config)]`** **Register via `extensions: [jwtExtension(config)]`**
```typescript ```typescript
import { jwtExtension } from 'apophis-fastify/extensions' import { jwtExtension } from '@apophis/fastify/extensions'
await fastify.register(apophis, { await fastify.register(apophis, {
extensions: [ extensions: [
@@ -348,7 +351,7 @@ jwt_format(this) == "compact"
**Register via `extensions: [x509Extension(config)]`** **Register via `extensions: [x509Extension(config)]`**
```typescript ```typescript
import { x509Extension } from 'apophis-fastify/extensions' import { x509Extension } from '@apophis/fastify/extensions'
await fastify.register(apophis, { await fastify.register(apophis, {
extensions: [x509Extension()] extensions: [x509Extension()]
@@ -370,7 +373,7 @@ x509_self_signed(this) == false
**Register via `extensions: [spiffeExtension(config)]`** **Register via `extensions: [spiffeExtension(config)]`**
```typescript ```typescript
import { spiffeExtension } from 'apophis-fastify/extensions' import { spiffeExtension } from '@apophis/fastify/extensions'
await fastify.register(apophis, { await fastify.register(apophis, {
extensions: [spiffeExtension()] extensions: [spiffeExtension()]
@@ -391,7 +394,7 @@ spiffe_validate(this) == true
**Register via `extensions: [tokenHashExtension(config)]`** **Register via `extensions: [tokenHashExtension(config)]`**
```typescript ```typescript
import { tokenHashExtension } from 'apophis-fastify/extensions' import { tokenHashExtension } from '@apophis/fastify/extensions'
await fastify.register(apophis, { await fastify.register(apophis, {
extensions: [tokenHashExtension()] extensions: [tokenHashExtension()]
@@ -412,7 +415,7 @@ token_hash(this, "sha256") == jwt_claims(this).ath
**Register via `extensions: [httpSignatureExtension(config)]`** **Register via `extensions: [httpSignatureExtension(config)]`**
```typescript ```typescript
import { httpSignatureExtension } from 'apophis-fastify/extensions' import { httpSignatureExtension } from '@apophis/fastify/extensions'
await fastify.register(apophis, { await fastify.register(apophis, {
extensions: [httpSignatureExtension()] extensions: [httpSignatureExtension()]
@@ -433,7 +436,7 @@ signature_valid(this) == true
**Register via `extensions: [timeExtension(config)]`** **Register via `extensions: [timeExtension(config)]`**
```typescript ```typescript
import { timeExtension } from 'apophis-fastify/extensions' import { timeExtension } from '@apophis/fastify/extensions'
await fastify.register(apophis, { await fastify.register(apophis, {
extensions: [timeExtension()] extensions: [timeExtension()]
@@ -453,7 +456,7 @@ jwt_claims(this).exp <= now() + 30000
**Register via `extensions: [statefulExtension()]`** **Register via `extensions: [statefulExtension()]`**
```typescript ```typescript
import { statefulExtension } from 'apophis-fastify/extensions' import { statefulExtension } from '@apophis/fastify/extensions'
await fastify.register(apophis, { await fastify.register(apophis, {
extensions: [statefulExtension()] extensions: [statefulExtension()]
@@ -521,7 +524,7 @@ fastify.get('/tenants/:id', {
**Register via `extensions: [requestContextExtension(config)]`** **Register via `extensions: [requestContextExtension(config)]`**
```typescript ```typescript
import { requestContextExtension } from 'apophis-fastify/extensions' import { requestContextExtension } from '@apophis/fastify/extensions'
await fastify.register(apophis, { await fastify.register(apophis, {
extensions: [requestContextExtension()] extensions: [requestContextExtension()]
@@ -555,19 +558,7 @@ await fastify.apophis.contract({
### Outbound Interception ### Outbound Interception
```typescript Outbound interception works through `fastify.apophis.test.enableOutboundMocks()` in test code. See the [Outbound Mocking](#outbound-mocking) section for the supported API.
import { wrapFetch, createOutboundInterceptor } from 'apophis-fastify'
const interceptor = createOutboundInterceptor([{
target: 'api.stripe.com',
error: {
probability: 0.05,
responses: [{ statusCode: 429, headers: { 'retry-after': '60' } }]
}
}], 42)
const interceptedFetch = wrapFetch(globalThis.fetch, interceptor)
```
### Per-Route Overrides ### Per-Route Overrides
docs/getting-started.md
+9 -5
View File
@@ -2,23 +2,25 @@
Get from install to your first behavioral bug in 10 minutes. Get from install to your first behavioral bug in 10 minutes.
APOPHIS is inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): instead of only validating request and response shape, encode intended behavior as executable contracts and let the tool find violations automatically. APOPHIS is inspired by the concept of invariant-driven automated testing: instead of only validating request and response shape, encode intended behavior as executable contracts and let the tool find violations automatically.
## Prerequisites ## Prerequisites
- Node.js 20.x or 22.x - Node.js 20.x or 22.x
- **Fastify v5** (v4 is not supported)
- **ESM project** (`"type": "module"` in package.json)
- A Fastify app with `@fastify/swagger` registered - A Fastify app with `@fastify/swagger` registered
## Step 1: Install ## Step 1: Install
```bash ```bash
npm install apophis-fastify fastify @fastify/swagger npm install @apophis/fastify fastify @fastify/swagger
``` ```
## Step 2: Scaffold ## Step 2: Scaffold
```bash ```bash
apophis init --preset safe-ci npx apophis init --preset safe-ci
``` ```
This creates: This creates:
@@ -55,7 +57,7 @@ app.post('/users', {
## Step 4: Run Verify ## Step 4: Run Verify
```bash ```bash
apophis verify --profile quick --routes "POST /users" npx apophis verify --profile quick --routes "POST /users"
``` ```
## Example Failure ## Example Failure
@@ -232,7 +234,7 @@ APOPHIS reads these OpenAPI schema extensions:
| `x-validate-runtime` | Top-level or `response[statusCode]` | Toggle runtime validation for this route (default: true) | | `x-validate-runtime` | Top-level or `response[statusCode]` | Toggle runtime validation for this route (default: true) |
| `x-extension-config` | Top-level | Per-route config for extensions (e.g., `{ jwt: { verify: false } }`) | | `x-extension-config` | Top-level | Per-route config for extensions (e.g., `{ jwt: { verify: false } }`) |
Annotations can be placed on the top-level schema or nested inside `response[statusCode]`. Nested annotations take precedence for that status code. Annotations can be placed on the top-level schema or nested inside `response[statusCode]`. Nested annotations from the first status code schema are merged with top-level annotations.
## Programmatic API ## Programmatic API
@@ -268,6 +270,8 @@ fastify.apophis.test.disableOutboundMocks()
const calls = fastify.apophis.test.getOutboundCalls('payment-api') const calls = fastify.apophis.test.getOutboundCalls('payment-api')
``` ```
Outbound mocking patches `globalThis.fetch` and is process-global. Only one mock runtime can be installed at a time. Run mock-dependent tests serially or isolate by process.
## Config Reference ## Config Reference
For the full configuration reference, see [CLI Reference](cli.md). For the full configuration reference, see [CLI Reference](cli.md).
docs/llm-safe-adoption.md
+2 -2
View File
@@ -2,7 +2,7 @@
APOPHIS is designed to be safe and predictable for LLM-generated Fastify services. APOPHIS is designed to be safe and predictable for LLM-generated Fastify services.
It applies the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) to LLM-assisted development: constrained vocabulary, deterministic replay, and executable contracts give coding agents a verifiable loop between generated changes and behavioral correctness. It applies an invariant-driven approach to LLM-assisted development: constrained vocabulary, deterministic replay, and executable contracts give coding agents a verifiable loop between generated changes and behavioral correctness.
## Why APOPHIS Is Good for LLM-Generated Services ## Why APOPHIS Is Good for LLM-Generated Services
@@ -164,7 +164,7 @@ console.log(verifyResult);
2. **Run doctor first**: Catch setup issues before running verify. 2. **Run doctor first**: Catch setup issues before running verify.
3. **Use `--changed` in CI**: Only verify routes that changed in the PR. 3. **Use `--changed` in CI**: Only verify routes that changed in the PR.
4. **Commit config**: Store `apophis.config.js` in version control. 4. **Commit config**: Store `apophis.config.js` in version control.
5. **Pin versions**: Pin `apophis-fastify` version in `package.json`. 5. **Pin versions**: Pin `@apophis/fastify` version in `package.json`.
## Troubleshooting ## Troubleshooting
docs/observe.md
+62 -13
View File
@@ -2,16 +2,11 @@
Runtime visibility and drift detection without blocking by default. Runtime visibility and drift detection without blocking by default.
Observe extends the invariant framework from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) to production environments: contracts run continuously against live traffic to detect behavioral drift without affecting requests. APOPHIS observe has two paths:
## What Observe Does 1. **CLI `apophis observe`**: Validates observe configuration readiness (policy, sinks, sampling, safety boundaries). Introduces no service process or runtime hooks. Use this for CI config validation before deployment.
`apophis observe` validates your runtime observe configuration: 2. **Programmatic runtime observation**: Register the APOPHIS plugin with `observe.enabled: true` and `observe.sinks` to emit contract pass/violation/error events from live traffic without blocking responses. Sampling controls the fraction of observed requests.
1. Checks that observe mode is allowed in the current environment
2. Validates reporting sink setup (logs, metrics, traces)
3. Confirms non-blocking semantics
4. Reports what would be observed and why it is safe
## When to Use It ## When to Use It
@@ -164,18 +159,72 @@ export default {
}; };
``` ```
## Sink Endpoint Configuration ## Programmatic Runtime Activation
Configure the reporting sink endpoint in your observe config: The CLI only validates configuration. To activate runtime observation, register
APOPHIS with observe options in your application:
```javascript ```typescript
import Fastify from 'fastify'
import apophisPlugin from '@apophis/fastify'
const app = Fastify({ logger: true })
// Register APOPHIS with observe enabled.
// This emits non-blocking contract pass/violation/error events
// for every covered request, gated by sampling.
await app.register(apophisPlugin, {
runtime: 'warn',
observe: { observe: {
sink: { enabled: true,
endpoint: 'http://collector.internal:4318' sampling: 0.1, // observe 10% of requests
sinks: [metricsSink],
},
})
// Implement the ObserveSink interface.
// Capture events to your preferred observability backend.
import type { ObserveSink, ObserveEvent } from '@apophis/fastify'
const metricsSink: ObserveSink = {
emit(event: ObserveEvent) {
// Emit a counter for each contract evaluation
myMetrics.increment(`apophis.contract.${event.type}`, {
route: event.route,
formula: event.formula,
})
// Record duration as a histogram
myMetrics.histogram('apophis.contract.duration_ms', event.durationMs, {
route: event.route,
})
// Log high-signal violations for immediate triage
if (event.type === 'contract.violation') {
logger.warn({ event }, 'APOPHIS contract violation')
} }
},
} }
``` ```
Key constraints:
- Sink `emit()` can be sync or async (returns `void | Promise<void>`).
- Sink rejections and thrown errors are silently caught — they never affect the route response or status code.
- Sampling is applied per-formula evaluation via `Math.random() < sampling`.
At `sampling: 1` every formula is emitted. At `sampling: 0` nothing is emitted.
- Only routes with APOPHIS annotations (`x-ensures`, `x-requires`) produce events.
Routes without annotations are not evaluated in observe mode.
## Sink Implementations
APOPHIS does not ship with built-in sinks. The `ObserveSink` interface lets you
plug in any backend. Common patterns:
- **OpenTelemetry**: emit counters and histograms via `@opentelemetry/api`.
- **pino logger**: emit structured log records via `pino.info()` / `pino.warn()`.
- **Internal metrics service**: POST events to an internal collector endpoint.
- **In-memory ring buffer**: capture recent events for diagnostics endpoints.
## Monorepo Validation ## Monorepo Validation
For monorepos, use `apophis doctor --workspace` to validate observe configuration across all workspace packages. `observe` itself does not support `--workspace`; use `doctor` to check config in each package. For monorepos, use `apophis doctor --workspace` to validate observe configuration across all workspace packages. `observe` itself does not support `--workspace`; use `doctor` to check config in each package.
docs/qualify.md
+245 -5
View File
@@ -2,7 +2,7 @@
Run scenario, stateful, and chaos checks against non-production Fastify services. Run scenario, stateful, and chaos checks against non-production Fastify services.
Qualify extends the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) with multi-step protocol flows, stateful sequences, and controlled fault injection. Qualify extends invariant-driven testing with multi-step protocol flows, stateful sequences, and controlled fault injection.
## What Qualify Does ## What Qualify Does
@@ -15,9 +15,51 @@ Qualify extends the invariant-driven approach from [Invariant-Driven Automated T
## When to Use It ## When to Use It
- **Nightly CI**: Scenario and stateful checks for critical flows Qualify is heavier than verify. Use it where the depth is worth the runtime cost:
- **Staging**: Protocol flow validation before production
- **Specialist teams**: Auth, billing, workflow systems | Workflow | Recommended | Why |
|---|---|---|
| **Pull request** | No — use `verify` | `verify` is fast (<5s for typical services) and catches behavioral regressions per-route. Qualify adds multi-minute scenario/stateful/chaos runs that are too slow for PR feedback loops. |
| **Nightly** | Yes | Full scenario, stateful, and chaos execution against staging. Catch protocol-level regressions that single-route verification cannot see. |
| **Pre-release** | Yes | Run qualify against the exact artifact that will be promoted to production. Treat a passing qualify run as a release gate for critical flows. |
| **Specialist workflows** | Yes | Auth flows, billing sequences, idempotency guarantees, and pagination consistency need multi-step qualification that verify cannot express. |
| **Chaos engineering** | Nightly or ad-hoc | Chaos injection increases latency. Run it in dedicated CI slots, not on every commit. |
### Quick workflow setup
```javascript
// apophis.config.js — two profiles for different cadences
export default {
mode: 'qualify',
profiles: {
'nightly': {
name: 'nightly',
mode: 'qualify',
preset: 'deep',
features: ['scenario', 'stateful', 'chaos'],
routes: [],
},
'pre-release': {
name: 'pre-release',
mode: 'qualify',
preset: 'deep',
features: ['scenario', 'stateful'],
routes: [],
},
},
presets: {
deep: { timeout: 15000, chaos: false },
},
}
```
Run nightly: `apophis qualify --profile nightly`
Run pre-release: `apophis qualify --profile pre-release --format json-summary`
For pull requests, use verify instead:
```bash
apophis verify --profile ci
```
## Scenario Examples ## Scenario Examples
@@ -246,7 +288,205 @@ export default {
## Gate Execution Counts ## Gate Execution Counts
Human output shows per-gate execution counts (scenario, stateful, chaos, adversity) so you can verify which gates actually ran. Human output shows per-gate execution counts (scenario, stateful, chaos) so you can verify which gates actually ran.
## Custom Scenarios (config-defined)
Define arbitrary multi-step scenarios directly in your `apophis.config.js` without writing code:
```javascript
// apophis.config.js
export default {
mode: 'qualify',
scenarios: [
{
name: 'idempotency-check',
steps: [
{
name: 'create-order',
request: {
method: 'POST',
url: '/orders',
body: { product: 'widget', quantity: 3 },
},
expect: ['status:201', 'response_body(this).id != null'],
capture: { orderId: 'response_body(this).id' },
},
{
name: 'duplicate-create',
request: {
method: 'POST',
url: '/orders',
headers: { 'x-idempotency-key': 'dup-001' },
body: { product: 'widget', quantity: 3 },
},
expect: ['status:200', 'response_body(this).id == "$create-order.orderId"'],
},
],
},
{
name: 'pagination-flow',
steps: [
{
name: 'list-page-1',
request: { method: 'GET', url: '/items?page=1&limit=5' },
expect: ['status:200', 'response_body(this).items != null'],
capture: { firstPageCount: 'response_body(this).items.length' },
},
{
name: 'list-page-2',
request: { method: 'GET', url: '/items?page=2&limit=5' },
expect: ['status:200'],
},
],
},
],
profiles: {
'nightly': {
name: 'nightly',
mode: 'qualify',
preset: 'deep',
routes: ['POST /orders', 'GET /orders', 'GET /items'],
},
},
presets: {
deep: { name: 'deep', timeout: 15000, chaos: true },
},
environments: {
local: { name: 'local', allowQualify: true, allowChaos: true },
},
};
```
Scenario step fields:
| Field | Required | Description |
|---|---|---|
| `name` | yes | Human-readable step label |
| `request.method` | yes | HTTP method (GET, POST, PUT, DELETE, PATCH) |
| `request.url` | yes | URL path (e.g. `/orders`, `/items?page=1`) |
| `request.body` | no | JSON request body |
| `request.headers` | no | Custom headers (e.g. `x-idempotency-key`) |
| `expect` | yes | APOSTL formulas that must return truthy for step to pass |
| `capture` | no | Map of `{ key: "apostl_formula" }` — captured values are substituted via `$stepName.key` in later steps |
Captured values are interpolated in subsequent step URLs, bodies, and headers using `$stepName.key` syntax.
## Chaos Configuration
Fine-tune chaos behavior via preset fields:
```javascript
presets: {
'chaos-lab': {
name: 'chaos-lab',
timeout: 10000,
chaos: true,
chaosStrategy: 'sample', // 'one' | 'all' | 'sample' | 'routes'
chaosSampleSize: 5, // routes to sample when strategy = 'sample'
chaosSampleRoutes: [ // explicit routes when strategy = 'routes'
'GET /api/users',
'POST /api/orders',
],
},
}
```
| Field | Default | Description |
|---|---|---|
| `chaosStrategy` | `'one'` | Route selection strategy |
| `chaosSampleSize` | `3` | Routes to sample (strategy `sample`) |
| `chaosSampleRoutes` | — | Explicit route list (strategy `routes`) |
## Artifact Interpretation
Each qualify run produces an artifact JSON document. Key sections:
### executionSummary
```json
{
"executionSummary": {
"totalPlanned": 15,
"totalExecuted": 12,
"totalPassed": 10,
"totalFailed": 2,
"scenariosRun": 3,
"statefulTestsRun": 5,
"chaosRunsRun": 4,
"chaosRoutesPlanned": 2,
"chaosRoutesExecuted": 2,
"totalSteps": 12
}
}
```
Use `totalExecuted` vs `totalPlanned` to see how many checks actually ran (gate gating, route filtering, chaos selection). A non-zero `totalPlanned` with zero `totalExecuted` means all gates were disabled or no routes matched.
### executedRoutes / skippedRoutes
```json
{
"executedRoutes": ["POST /orders", "GET /orders/:id", "GET /items"],
"skippedRoutes": [
{ "route": "DELETE /items/:id", "reason": "No scenario covers this route" },
{ "route": "GET /health", "reason": "Not selected by chaos strategy: one" }
]
}
```
`executedRoutes` lists every route that had at least one scenario step, stateful command, or chaos injection. `skippedRoutes` explains why every other discovered route was excluded.
### profileGates
```json
{
"profileGates": {
"scenario": true,
"stateful": true,
"chaos": false
}
}
```
Shows which gates were active. Combine with `executionSummary` per-gate counts to verify each active gate produced results.
### stepTraces
Each entry records an individual step execution:
```json
{
"stepTraces": [
{
"step": 0,
"name": "create-order",
"route": "POST /orders",
"durationMs": 12,
"status": "passed"
}
]
}
```
Filter by `status` to isolate failures. Look at `durationMs` for performance regressions.
### failures
```json
{
"failures": [
{
"route": "POST /orders",
"contract": "status:201",
"category": "runtime",
"replayCommand": "apophis replay --artifact reports/apophis/qualify-2026-05-21T...json"
}
]
}
```
`replayCommand` gives a copy-pasteable command to re-run the exact same seed with the stored artifact for triage.
## Zero-Execution Guardrail ## Zero-Execution Guardrail
docs/quality.md
+4 -4
View File
@@ -44,7 +44,7 @@ import {
applyChaosToExecution, applyChaosToExecution,
createChaosEventArbitrary, createChaosEventArbitrary,
formatChaosEvents, formatChaosEvents,
} from 'apophis-fastify' } from '@apophis/fastify'
// Apply pre-generated chaos events to a context // Apply pre-generated chaos events to a context
const result = applyChaosToExecution(ctx, events) const result = applyChaosToExecution(ctx, events)
@@ -71,7 +71,7 @@ Automatically rerun failing tests with varied seeds to detect non-deterministic
### Usage ### Usage
```javascript ```javascript
import { FlakeDetector } from 'apophis-fastify' import { FlakeDetector } from '@apophis/fastify'
const detector = new FlakeDetector({ const detector = new FlakeDetector({
sameSeedReruns: 1, // Rerun with same seed sameSeedReruns: 1, // Rerun with same seed
@@ -121,7 +121,7 @@ Measure contract strength by injecting synthetic bugs. A "mutation" is a small c
### Usage ### Usage
```javascript ```javascript
import { runMutationTesting } from 'apophis-fastify/quality/mutation' import { runMutationTesting } from '@apophis/fastify/quality/mutation'
const report = await runMutationTesting(fastify, { const report = await runMutationTesting(fastify, {
runs: 10, runs: 10,
@@ -176,7 +176,7 @@ console.log('Weak contracts:', report.weakContracts)
Test a specific mutation without running the full suite: Test a specific mutation without running the full suite:
```javascript ```javascript
import { testMutation } from 'apophis-fastify/quality/mutation' import { testMutation } from '@apophis/fastify/quality/mutation'
const killed = await testMutation(fastify, contract, mutation, { const killed = await testMutation(fastify, contract, mutation, {
runs: 10, runs: 10,
docs/verify.md
+6 -2
View File
@@ -2,7 +2,7 @@
Deterministic contract verification for CI and local development. Deterministic contract verification for CI and local development.
APOPHIS implements the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): encode intended behavior as executable formulas, then verify them automatically with property-based generation and deterministic replay. APOPHIS implements invariant-driven testing: encode intended behavior as executable formulas, then verify them automatically with property-based generation and deterministic replay.
## When to Use It ## When to Use It
@@ -83,6 +83,8 @@ apophis verify --profile ci --changed
If no routes changed, exits 2 with a message. If no routes changed, exits 2 with a message.
`--changed` is a heuristic: it maps changed file paths to routes by checking route path segments against file names. This is useful as a developer convenience, but for strict CI gating, prefer explicit `--routes` filters or full verification.
## Failure Output Format ## Failure Output Format
When a contract fails, APOPHIS prints: When a contract fails, APOPHIS prints:
@@ -186,7 +188,7 @@ Output includes per-package pass/fail summaries. Fails if any package fails.
## Test Budget ## Test Budget
The `runs` field in your preset controls how many property-based tests execute per route. Default is 50. Lower for faster CI feedback, higher for deeper exploration: The `runs` field in your preset controls how many property-based test samples execute per route. Default is 50. Lower for faster CI feedback, higher for deeper exploration:
```javascript ```javascript
profiles: { profiles: {
@@ -203,3 +205,5 @@ presets: {
} }
} }
``` ```
CLI verify generates one property-based test sample per contract by default when no `runs` is specified. Set `runs` in the preset to increase sampled inputs per route.
examples/app/src/app.ts
+1 -1
View File
@@ -1,6 +1,6 @@
import Fastify from 'fastify' import Fastify from 'fastify'
import swagger from '@fastify/swagger' import swagger from '@fastify/swagger'
import { apophisPlugin } from 'apophis-fastify' import { apophisPlugin } from '@apophis/fastify'
import { databasePlugin } from './plugins/database.js' import { databasePlugin } from './plugins/database.js'
import { userRoutes } from './routes/users.js' import { userRoutes } from './routes/users.js'
no_commit_paper.md
-2170
View File
File diff suppressed because it is too large Load Diff
package-lock.json
Generated
+9 -112
View File
@@ -1,13 +1,13 @@
{ {
"name": "apophis-fastify", "name": "@apophis/fastify",
"version": "2.0.0", "version": "2.7.0",
"lockfileVersion": 3, "lockfileVersion": 3,
"requires": true, "requires": true,
"packages": { "packages": {
"": { "": {
"name": "apophis-fastify", "name": "@apophis/fastify",
"version": "2.0.0", "version": "2.7.0",
"license": "ISC", "license": "MIT",
"dependencies": { "dependencies": {
"@clack/prompts": "^0.10.1", "@clack/prompts": "^0.10.1",
"cac": "^6.7.14", "cac": "^6.7.14",
@@ -15,7 +15,6 @@
"fastify-plugin": "^5.0.0", "fastify-plugin": "^5.0.0",
"picocolors": "^1.0.0", "picocolors": "^1.0.0",
"pino": "^10.3.1", "pino": "^10.3.1",
"recheck": "^4.5.0",
"safe-regex": "^2.1.1", "safe-regex": "^2.1.1",
"undici": "^7.0.0" "undici": "^7.0.0"
}, },
@@ -33,6 +32,9 @@
"tsx": "^4.0.0", "tsx": "^4.0.0",
"typescript": "^6.0.3" "typescript": "^6.0.3"
}, },
"engines": {
"node": ">=20.18.1 <21 || >=22 <23"
},
"peerDependencies": { "peerDependencies": {
"@fastify/swagger": "^9.0.0", "@fastify/swagger": "^9.0.0",
"fastify": "^5.0.0" "fastify": "^5.0.0"
@@ -1573,18 +1575,6 @@
"integrity": "sha512-k2ENnmBugE/rzQfEcdWHcCY+/FM3VLzH9cYEsbdsoqrvzAKRhUZeRNhAZvB8OitQJ1TBed3yqWtdjzS6wJKBwg==", "integrity": "sha512-k2ENnmBugE/rzQfEcdWHcCY+/FM3VLzH9cYEsbdsoqrvzAKRhUZeRNhAZvB8OitQJ1TBed3yqWtdjzS6wJKBwg==",
"license": "MIT" "license": "MIT"
}, },
"node_modules/@pkgr/core": {
"version": "0.1.2",
"resolved": "https://registry.npmjs.org/@pkgr/core/-/core-0.1.2.tgz",
"integrity": "sha512-fdDH1LSGfZdTH2sxdpVMw31BanV28K/Gry0cVFxaNP77neJSkd82mM8ErPNYs9e+0O7SdHBLTDzDgwUuy18RnQ==",
"license": "MIT",
"engines": {
"node": "^12.20.0 || ^14.18.0 || >=16.0.0"
},
"funding": {
"url": "https://opencollective.com/unts"
}
},
"node_modules/@rollup/rollup-android-arm-eabi": { "node_modules/@rollup/rollup-android-arm-eabi": {
"version": "4.60.2", "version": "4.60.2",
"resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.60.2.tgz", "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.60.2.tgz",
@@ -3721,84 +3711,6 @@
"node": ">= 12.13.0" "node": ">= 12.13.0"
} }
}, },
"node_modules/recheck": {
"version": "4.5.0",
"resolved": "https://registry.npmjs.org/recheck/-/recheck-4.5.0.tgz",
"integrity": "sha512-kPnbOV6Zfx9a25AZ++28fI1q78L/UVRQmmuazwVRPfiiqpMs+WbOU69Shx820XgfKWfak0JH75PUvZMFtRGSsw==",
"license": "MIT",
"dependencies": {
"synckit": "0.9.2"
},
"engines": {
"node": ">=20"
},
"optionalDependencies": {
"recheck-jar": "4.5.0",
"recheck-linux-x64": "4.5.0",
"recheck-macos-arm64": "4.5.0",
"recheck-macos-x64": "4.5.0",
"recheck-windows-x64": "4.5.0"
}
},
"node_modules/recheck-jar": {
"version": "4.5.0",
"resolved": "https://registry.npmjs.org/recheck-jar/-/recheck-jar-4.5.0.tgz",
"integrity": "sha512-Ad7oCQmY8cQLzd3QVNXjzZ+S6MbImGhR4AaW2yiGzteOfMV45522rt6nSzFyt8p3mCEaMcm/4MoZrMSxUcCbrA==",
"license": "MIT",
"optional": true
},
"node_modules/recheck-linux-x64": {
"version": "4.5.0",
"resolved": "https://registry.npmjs.org/recheck-linux-x64/-/recheck-linux-x64-4.5.0.tgz",
"integrity": "sha512-52kXsR/v+IbGIKYYFZfSZcgse/Ci9IA2HnuzrtvRRcfODkcUGe4n72ESQ8nOPwrdHFg9i4j9/YyPh1HWWgpJ6A==",
"cpu": [
"x64"
],
"license": "MIT",
"optional": true,
"os": [
"linux"
]
},
"node_modules/recheck-macos-arm64": {
"version": "4.5.0",
"resolved": "https://registry.npmjs.org/recheck-macos-arm64/-/recheck-macos-arm64-4.5.0.tgz",
"integrity": "sha512-qIyK3dRuLkORQvv0b59fZZRXweSmjjWaoA4K8Kgifz0anMBH4pqsDV6plBlgjcRmW9yC12wErIRzifREaKnk2w==",
"cpu": [
"arm64"
],
"license": "MIT",
"optional": true,
"os": [
"darwin"
]
},
"node_modules/recheck-macos-x64": {
"version": "4.5.0",
"resolved": "https://registry.npmjs.org/recheck-macos-x64/-/recheck-macos-x64-4.5.0.tgz",
"integrity": "sha512-1wp/eiLxcjC/Ex4wurlrS/LGzt8IiF4TiK5sEjldu4HVAKdNCnnmsS9a5vFpfcikDz4ZuZlLlTi1VbQTxHlwZg==",
"cpu": [
"x64"
],
"license": "MIT",
"optional": true,
"os": [
"darwin"
]
},
"node_modules/recheck-windows-x64": {
"version": "4.5.0",
"resolved": "https://registry.npmjs.org/recheck-windows-x64/-/recheck-windows-x64-4.5.0.tgz",
"integrity": "sha512-ekBKwAp0oKkMULn5zgmHEYLwSJfkfb95AbTtbDkQazNkqYw9PRD/mVyFUR6Ff2IeRyZI0gxy+N2AKBISWydhug==",
"cpu": [
"x64"
],
"license": "MIT",
"optional": true,
"os": [
"win32"
]
},
"node_modules/regexp-tree": { "node_modules/regexp-tree": {
"version": "0.1.27", "version": "0.1.27",
"resolved": "https://registry.npmjs.org/regexp-tree/-/regexp-tree-0.1.27.tgz", "resolved": "https://registry.npmjs.org/regexp-tree/-/regexp-tree-0.1.27.tgz",
@@ -4208,22 +4120,6 @@
"node": ">= 6" "node": ">= 6"
} }
}, },
"node_modules/synckit": {
"version": "0.9.2",
"resolved": "https://registry.npmjs.org/synckit/-/synckit-0.9.2.tgz",
"integrity": "sha512-vrozgXDQwYO72vHjUb/HnFbQx1exDjoKzqx23aXEg2a9VIg2TSFZ8FmeZpTjUCFMYw7mpX4BE2SFu8wI7asYsw==",
"license": "MIT",
"dependencies": {
"@pkgr/core": "^0.1.0",
"tslib": "^2.6.2"
},
"engines": {
"node": "^14.18.0 || >=16.0.0"
},
"funding": {
"url": "https://opencollective.com/unts"
}
},
"node_modules/thenify": { "node_modules/thenify": {
"version": "3.3.1", "version": "3.3.1",
"resolved": "https://registry.npmjs.org/thenify/-/thenify-3.3.1.tgz", "resolved": "https://registry.npmjs.org/thenify/-/thenify-3.3.1.tgz",
@@ -4314,6 +4210,7 @@
"version": "2.8.1", "version": "2.8.1",
"resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz",
"integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
"dev": true,
"license": "0BSD" "license": "0BSD"
}, },
"node_modules/tsup": { "node_modules/tsup": {
package.json
+25 -8
View File
@@ -1,9 +1,9 @@
{ {
"name": "@apophis/fastify", "name": "@apophis/fastify",
"version": "2.0.0", "version": "2.7.0",
"description": "Contract-driven API testing plugin for Fastify with property-based testing, timeout enforcement, redirect capture, and deterministic concurrency", "description": "Contract-driven API testing plugin for Fastify with property-based testing, timeout enforcement, redirect capture, and deterministic concurrency",
"main": "dist/index.js", "main": "dist/index.js",
"types": "index.d.ts", "types": "dist/index.d.ts",
"type": "module", "type": "module",
"bin": { "bin": {
"apophis": "dist/cli/index.js" "apophis": "dist/cli/index.js"
@@ -11,12 +11,28 @@
"exports": { "exports": {
".": { ".": {
"import": "./dist/index.js", "import": "./dist/index.js",
"types": "./index.d.ts" "types": "./dist/index.d.ts"
},
"./extension/factories": {
"import": "./dist/extension/factories.js",
"types": "./dist/extension/factories.d.ts"
}, },
"./extensions": { "./extensions": {
"import": "./dist/extensions/index.js", "import": "./dist/extensions/index.js",
"types": "./dist/extensions/index.d.ts" "types": "./dist/extensions/index.d.ts"
}, },
"./extensions/sse": {
"import": "./dist/extensions/sse/index.js",
"types": "./dist/extensions/sse/index.d.ts"
},
"./extensions/websocket": {
"import": "./dist/extensions/websocket/index.js",
"types": "./dist/extensions/websocket/index.d.ts"
},
"./extensions/serializers": {
"import": "./dist/extensions/serializers/index.js",
"types": "./dist/extensions/serializers/index.d.ts"
},
"./extensions/*": { "./extensions/*": {
"import": "./dist/extensions/*.js", "import": "./dist/extensions/*.js",
"types": "./dist/extensions/*.d.ts" "types": "./dist/extensions/*.d.ts"
@@ -28,18 +44,18 @@
}, },
"files": [ "files": [
"dist", "dist",
"index.d.ts",
"README.md", "README.md",
"LICENSE", "LICENSE",
"docs" "docs"
], ],
"engines": { "engines": {
"node": "^20.0.0 || ^22.0.0" "node": ">=20.18.1 <21 || >=22 <23"
}, },
"scripts": { "scripts": {
"build": "tsc", "prepare": "npm run build",
"build": "tsc -p tsconfig.build.json && chmod +x dist/cli/index.js",
"test": "npm run build && npm run test:src && npm run test:cli", "test": "npm run build && npm run test:src && npm run test:cli",
"test:dist": "NODE_ENV=test node --test dist/test/*.test.js",
"test:src": "tsx --test src/test/*.test.ts", "test:src": "tsx --test src/test/*.test.ts",
"test:cli": "tsx --test src/test/cli/*.test.ts", "test:cli": "tsx --test src/test/cli/*.test.ts",
"test:cli:goldens": "tsx --test src/test/cli/goldens.test.ts", "test:cli:goldens": "tsx --test src/test/cli/goldens.test.ts",
@@ -52,6 +68,8 @@
"profile:qualify": "npm run build && mkdir -p .profiles && node --cpu-prof --cpu-prof-dir=.profiles --cpu-prof-name=qualify.cpuprofile dist/cli/index.js qualify --cwd src/cli/__fixtures__/protocol-lab --profile oauth-nightly --seed 42 --quiet", "profile:qualify": "npm run build && mkdir -p .profiles && node --cpu-prof --cpu-prof-dir=.profiles --cpu-prof-name=qualify.cpuprofile dist/cli/index.js qualify --cwd src/cli/__fixtures__/protocol-lab --profile oauth-nightly --seed 42 --quiet",
"profile:qualify:quick": "npm run build && mkdir -p .profiles && node --cpu-prof --cpu-prof-dir=.profiles --cpu-prof-name=qualify-quick.cpuprofile dist/cli/index.js qualify --cwd src/cli/__fixtures__/protocol-lab --profile oauth-nightly --seed 42 --quiet", "profile:qualify:quick": "npm run build && mkdir -p .profiles && node --cpu-prof --cpu-prof-dir=.profiles --cpu-prof-name=qualify-quick.cpuprofile dist/cli/index.js qualify --cwd src/cli/__fixtures__/protocol-lab --profile oauth-nightly --seed 42 --quiet",
"clean": "rm -rf dist", "clean": "rm -rf dist",
"typecheck": "tsc --noEmit",
"lint": "tsc --noEmit",
"apophis:verify": "apophis verify --profile quick", "apophis:verify": "apophis verify --profile quick",
"apophis:doctor": "apophis doctor" "apophis:doctor": "apophis doctor"
}, },
@@ -83,7 +101,6 @@
"fastify-plugin": "^5.0.0", "fastify-plugin": "^5.0.0",
"picocolors": "^1.0.0", "picocolors": "^1.0.0",
"pino": "^10.3.1", "pino": "^10.3.1",
"recheck": "^4.5.0",
"safe-regex": "^2.1.1", "safe-regex": "^2.1.1",
"undici": "^7.0.0" "undici": "^7.0.0"
}, },
src/augmentations.ts
+11
View File
@@ -0,0 +1,11 @@
/**
* Fastify module augmentation — declares the apophis property on FastifyInstance.
* This makes fastify.apophis typed correctly in consumer TypeScript projects.
*/
import type { ApophisDecorations } from './types/core.js'
declare module 'fastify' {
interface FastifyInstance {
readonly apophis: ApophisDecorations
}
}
src/cli/__goldens__/help.txt
+1 -1
View File
@@ -6,7 +6,7 @@ Options:
--config <path> Path to config file --config <path> Path to config file
--profile <name> Profile name from config --profile <name> Profile name from config
--cwd <path> Working directory --cwd <path> Working directory
--format <human|json|ndjson> Output format (default: human) --format <human|json|ndjson|json-summary|ndjson-summary> Output format (default: human)
--color <auto|always|never> Color mode (default: auto) --color <auto|always|never> Color mode (default: auto)
--quiet Suppress non-essential output --quiet Suppress non-essential output
--verbose Verbose output --verbose Verbose output
src/cli/commands/doctor/checks/dependencies.ts
+20 -6
View File
@@ -33,7 +33,7 @@ export interface DependencyCheckOptions {
// Constants // Constants
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
const MIN_NODE_VERSION = 18; const MIN_NODE_VERSION = 20;
const REQUIRED_PEER_DEPS = ['fastify', '@fastify/swagger']; const REQUIRED_PEER_DEPS = ['fastify', '@fastify/swagger'];
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -54,13 +54,27 @@ function parseNodeMajor(version: string): number {
export function checkNodeVersion(nodeVersion: string): DependencyCheckResult { export function checkNodeVersion(nodeVersion: string): DependencyCheckResult {
const major = parseNodeMajor(nodeVersion); const major = parseNodeMajor(nodeVersion);
if (major < MIN_NODE_VERSION) { const tooOld = major < MIN_NODE_VERSION;
const untested = major >= 23 || major === 21;
if (tooOld) {
return { return {
name: 'node-version', name: 'node-version',
status: 'fail', status: 'fail',
message: `Node.js ${nodeVersion} is not supported. Minimum required: ${MIN_NODE_VERSION}.x`, message: `Node.js ${nodeVersion} is not supported. APOPHIS requires Node.js >=${MIN_NODE_VERSION}.18.1.`,
detail: `APOPHIS requires Node.js ${MIN_NODE_VERSION} or higher for ESM and modern features.`, detail: `Detected Node.js ${nodeVersion} (major ${major}). APOPHIS requires Node ${MIN_NODE_VERSION} LTS or 22 LTS.`,
remediation: `Upgrade Node.js to ${MIN_NODE_VERSION}.x or higher (use nvm, fnm, or your package manager).`, remediation: `Install Node.js ${MIN_NODE_VERSION}.x or 22.x (use nvm, fnm, or your package manager).`,
mode: 'all',
};
}
if (untested) {
return {
name: 'node-version',
status: 'warn',
message: `Node.js ${nodeVersion} is not in the tested range (20.x or 22.x).`,
detail: `Detected Node.js ${nodeVersion} (major ${major}). APOPHIS is tested on Node 20 LTS and 22 LTS.`,
remediation: `Use Node.js 20.x or 22.x for best compatibility. Current version may work but is untested.`,
mode: 'all', mode: 'all',
}; };
} }
@@ -68,7 +82,7 @@ export function checkNodeVersion(nodeVersion: string): DependencyCheckResult {
return { return {
name: 'node-version', name: 'node-version',
status: 'pass', status: 'pass',
message: `Node.js ${nodeVersion} meets minimum requirement (${MIN_NODE_VERSION}+)`, message: `Node.js ${nodeVersion} is supported (>=${MIN_NODE_VERSION}.18.1 <21 || >=22 <23)`,
mode: 'all', mode: 'all',
}; };
} }
src/cli/commands/doctor/checks/routes.ts
+36 -44
View File
@@ -7,8 +7,7 @@
* - Is the app file loadable? * - Is the app file loadable?
*/ */
import { existsSync } from 'node:fs'; import { APP_CANDIDATES, findAppFile } from '../../../core/app-loader.js';
import { resolve } from 'node:path';
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Types // Types
@@ -28,38 +27,6 @@ export interface RouteCheckOptions {
configPath?: string; configPath?: string;
} }
// ---------------------------------------------------------------------------
// App file detection
// ---------------------------------------------------------------------------
const APP_CANDIDATES = [
'app.js',
'app.ts',
'server.js',
'server.ts',
'index.js',
'index.ts',
'src/app.js',
'src/app.ts',
'src/server.js',
'src/server.ts',
'src/index.js',
'src/index.ts',
];
/**
* Find the Fastify app entrypoint file.
*/
function findAppFile(cwd: string): string | null {
for (const candidate of APP_CANDIDATES) {
const fullPath = resolve(cwd, candidate);
if (existsSync(fullPath)) {
return candidate;
}
}
return null;
}
/** /**
* Check if app file exists and is readable. * Check if app file exists and is readable.
*/ */
@@ -106,7 +73,7 @@ export async function checkRouteDiscovery(options: RouteCheckOptions): Promise<R
} }
try { try {
const appPath = resolve(options.cwd, appFile); const appPath = appFile;
const appModule = await import(appPath); const appModule = await import(appPath);
const app = appModule.default || appModule; const app = appModule.default || appModule;
@@ -147,18 +114,32 @@ export async function checkRouteDiscovery(options: RouteCheckOptions): Promise<R
} }
// Check for routes // Check for routes
let routeCount = 0; let routeCount = 0
let discoverySource: string | undefined
// Fastify 5+ routes access // Fastify 5+ routes access
if (app.routes && typeof app.routes === 'function') { if (app.routes && typeof app.routes === 'function') {
const routes = app.routes(); const routes = app.routes()
routeCount = Array.isArray(routes) ? routes.length : 0; routeCount = Array.isArray(routes) ? routes.length : 0
} }
// Fallback: check if we can get routes via inject or other methods // Fallback: check if we can get routes via inject or other methods
if (routeCount === 0 && app.hasRoute) { if (routeCount === 0 && app.hasRoute) {
// We can't enumerate, but we can at least verify the app is functional routeCount = -1
routeCount = -1; // Unknown but app seems functional }
// Use discoverRouteDetails for metadata-aware reporting
try {
const { discoverRouteDetails } = await import('../../../../domain/discovery.js')
const discovery = discoverRouteDetails(app as Parameters<typeof discoverRouteDetails>[0])
if (discovery.source === 'print-routes') {
discoverySource = 'print-routes'
if (routeCount === 0 && discovery.routes.length > 0) {
routeCount = discovery.routes.length
}
}
} catch {
// discovery diagnostics are optional
} }
if (routeCount === 0) { if (routeCount === 0) {
@@ -170,7 +151,7 @@ export async function checkRouteDiscovery(options: RouteCheckOptions): Promise<R
'APOPHIS discovers routes via the onRoute hook.', 'APOPHIS discovers routes via the onRoute hook.',
remediation: 'Register routes before exporting the app, or ensure the APOPHIS plugin is registered.', remediation: 'Register routes before exporting the app, or ensure the APOPHIS plugin is registered.',
mode: 'all', mode: 'all',
}; }
} }
if (routeCount < 0) { if (routeCount < 0) {
@@ -180,7 +161,18 @@ export async function checkRouteDiscovery(options: RouteCheckOptions): Promise<R
message: `App loaded from ${appFile}. Route enumeration not available (app is functional).`, message: `App loaded from ${appFile}. Route enumeration not available (app is functional).`,
detail: 'Route count could not be determined, but the app appears to be a valid Fastify instance.', detail: 'Route count could not be determined, but the app appears to be a valid Fastify instance.',
mode: 'all', mode: 'all',
}; }
}
if (discoverySource === 'print-routes') {
return {
name: 'route-discovery',
status: 'warn',
message: `Discovered ${routeCount} route(s) from ${appFile} but without schema metadata.`,
detail: 'Routes were discovered through printRoutes() fallback. Schema annotations (x-ensures, x-requires, x-outbound, x-timeout, x-variants) are unavailable. APOPHIS can detect paths but not behavioral contracts.',
remediation: 'Register APOPHIS or install route discovery before defining routes, or use createFastify() from @apophis/fastify.',
mode: 'all',
}
} }
return { return {
@@ -233,8 +225,8 @@ export async function checkSwaggerRegistration(options: RouteCheckOptions): Prom
} }
try { try {
const appPath = resolve(options.cwd, appFile); const { readFileSync } = await import('node:fs');
const content = (await import('node:fs')).readFileSync(appPath, 'utf-8'); const content = readFileSync(appFile, 'utf-8');
if (content.includes('@fastify/swagger') || content.includes('fastify-swagger')) { if (content.includes('@fastify/swagger') || content.includes('fastify-swagger')) {
return { return {
src/cli/commands/doctor/index.ts
-6
View File
@@ -78,12 +78,6 @@ export interface DoctorResult {
// Check filtering // Check filtering
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
function shouldRunCheck(checkMode: string | undefined, modeFilter: DoctorMode): boolean {
if (!modeFilter) return true;
if (!checkMode || checkMode === 'all') return true;
return checkMode === modeFilter;
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Monorepo detection // Monorepo detection
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
src/cli/commands/init/index.ts
+3 -17
View File
@@ -8,6 +8,7 @@ import { resolve } from 'node:path';
import type { CliContext } from '../../core/types.js'; import type { CliContext } from '../../core/types.js';
import { USAGE_ERROR, SUCCESS } from '../../core/exit-codes.js'; import { USAGE_ERROR, SUCCESS } from '../../core/exit-codes.js';
import { getScaffoldForPreset, getPresetNames, type ScaffoldResult } from './scaffolds/index.js'; import { getScaffoldForPreset, getPresetNames, type ScaffoldResult } from './scaffolds/index.js';
import { APP_CANDIDATES } from '../../core/app-loader.js';
// ───────────────────────────────────────────────────────────────────────────── // ─────────────────────────────────────────────────────────────────────────────
// Types // Types
@@ -63,22 +64,7 @@ function renderInstallCommand(
* - Common server file names (server.js, app.js, index.js, etc.) * - Common server file names (server.js, app.js, index.js, etc.)
*/ */
export async function detectFastifyEntrypoint(cwd: string): Promise<string | null> { export async function detectFastifyEntrypoint(cwd: string): Promise<string | null> {
const candidates = [ for (const candidate of APP_CANDIDATES) {
'app.js',
'app.ts',
'server.js',
'server.ts',
'index.js',
'index.ts',
'src/app.js',
'src/app.ts',
'src/server.js',
'src/server.ts',
'src/index.js',
'src/index.ts',
];
for (const candidate of candidates) {
const fullPath = resolve(cwd, candidate); const fullPath = resolve(cwd, candidate);
if (!existsSync(fullPath)) continue; if (!existsSync(fullPath)) continue;
@@ -217,7 +203,7 @@ function generateConfigContent(config: ScaffoldResult['config'], isTypeScript: b
lines.push(''); lines.push('');
if (isTypeScript) { if (isTypeScript) {
lines.push('import type { ApophisConfig } from "apophis-fastify/cli";'); lines.push('import type { ApophisConfig } from "@apophis/fastify";');
lines.push(''); lines.push('');
lines.push('const config: ApophisConfig = ' + stringifyConfig(config) + ';'); lines.push('const config: ApophisConfig = ' + stringifyConfig(config) + ';');
lines.push(''); lines.push('');
src/cli/commands/observe/index.ts
+10 -1
View File
@@ -264,7 +264,16 @@ function formatActivationOutput(
} }
lines.push(''); lines.push('');
lines.push('To activate observation, run without --check-config.'); lines.push('The CLI validates configuration and reports readiness.');
lines.push('To activate runtime observation, register the APOPHIS plugin in your');
lines.push('application with observe options:');
lines.push('');
lines.push(' await fastify.register(apophisPlugin, {');
lines.push(' runtime: \'warn\',');
lines.push(' observe: { enabled: true, sinks: [...] }');
lines.push(' });');
lines.push('');
lines.push('See docs/observe.md for the full programmatic activation guide.');
return lines.join('\n'); return lines.join('\n');
} }
src/cli/commands/qualify/chaos-handler.ts
+38 -14
View File
@@ -11,7 +11,7 @@
* - No optional imports — everything is passed via parameters * - No optional imports — everything is passed via parameters
*/ */
import { applyChaosToExecution, createChaosEventArbitrary, formatChaosEvents } from '../../../quality/chaos-v3.js' import { applyChaosToExecution, formatChaosEvents, extractDelays, sleep as chaosSleep } from '../../../quality/chaos-v3.js'
import { SeededRng } from '../../../infrastructure/seeded-rng.js' import { SeededRng } from '../../../infrastructure/seeded-rng.js'
import type { import type {
RouteContract, RouteContract,
@@ -23,6 +23,12 @@ import type { QualifyRunnerDeps, ChaosRunResult } from './runner.js'
/** /**
* Run a single route with chaos injection and collect traces. * Run a single route with chaos injection and collect traces.
* Uses chaos-v3 pure functions for deterministic adversity. * Uses chaos-v3 pure functions for deterministic adversity.
*
* Flow:
* 1. Generate deterministic chaos events
* 2. Apply transport-level delays (sleep before HTTP request)
* 3. Execute the HTTP request
* 4. Apply remaining chaos (error, dropout, corruption) to the response context
*/ */
export async function runChaosOnRoute( export async function runChaosOnRoute(
deps: QualifyRunnerDeps, deps: QualifyRunnerDeps,
@@ -31,12 +37,16 @@ export async function runChaosOnRoute(
): Promise<{ ctx: EvalContext; chaosResult: ChaosRunResult }> { ): Promise<{ ctx: EvalContext; chaosResult: ChaosRunResult }> {
const started = Date.now() const started = Date.now()
// Generate chaos events using seeded RNG via fast-check // 1. Generate deterministic chaos events
// For CLI qualify, we use a deterministic subset const events = generateDeterministicChaosEvents(chaosConfig, deps.seed)
const rng = new SeededRng(deps.seed)
const contractNames: string[] = []
// Build a minimal request for the route // 2. Apply transport-level delays BEFORE the HTTP request
const { totalMs: delayMs } = extractDelays(events)
if (delayMs > 0) {
await chaosSleep(delayMs)
}
// 3. Build a minimal request and execute it
const request = { const request = {
method: route.method, method: route.method,
url: route.path, url: route.path,
@@ -45,25 +55,39 @@ export async function runChaosOnRoute(
body: undefined as unknown, body: undefined as unknown,
} }
// Execute the request
const { executeHttp } = await import('../../../infrastructure/http-executor.js') const { executeHttp } = await import('../../../infrastructure/http-executor.js')
const ctx = await executeHttp(deps.fastify, route, request, undefined, deps.timeout) const ctx = await executeHttp(deps.fastify, route, request, undefined, deps.timeout)
// Generate and apply chaos events // 4. Apply remaining chaos (error, dropout, corruption) to response context
const chaosArb = createChaosEventArbitrary(chaosConfig, contractNames) // Filter out delay events since they were already applied at the transport level
// For deterministic CLI runs, we generate a fixed small set of events const nonDelayEvents = events.filter(e => e.type !== 'inbound-delay')
// In practice, fast-check would be used in property tests; here we simulate const application = applyChaosToExecution(ctx, nonDelayEvents)
const events = generateDeterministicChaosEvents(chaosConfig, deps.seed)
const application = applyChaosToExecution(ctx, events) // 5. Validate post-chaos contracts: did the route still satisfy its contracts?
let contractsPassed = true
if (application.applied && route.ensures.length > 0) {
const { validatePostconditionsAsync } = await import('../../../domain/contract-validation.js')
const result = await validatePostconditionsAsync(
route.ensures,
application.ctx,
route,
undefined as unknown as import('../../../extension/types.js').ExtensionRegistry
)
contractsPassed = result.success
}
// Consider delay as "applied chaos" for reporting purposes
const hadDelay = delayMs > 0
const chaosApplied = application.applied || hadDelay
const chaosResult: ChaosRunResult = { const chaosResult: ChaosRunResult = {
applied: application.applied, applied: chaosApplied,
events: application.events events: application.events
.filter(e => e.type !== 'none') .filter(e => e.type !== 'none')
.map(e => formatChaosEvents([e])), .map(e => formatChaosEvents([e])),
route: `${route.method} ${route.path}`, route: `${route.method} ${route.path}`,
durationMs: Date.now() - started, durationMs: Date.now() - started,
contractsPassed,
} }
return { ctx: application.ctx, chaosResult } return { ctx: application.ctx, chaosResult }
src/cli/commands/qualify/index.ts
+88 -144
View File
@@ -37,7 +37,6 @@ import { renderJson, renderJsonArtifact, renderJsonSummaryArtifact } from '../..
import { renderNdjsonArtifact, renderNdjsonSummaryArtifact } from '../../renderers/ndjson.js' import { renderNdjsonArtifact, renderNdjsonSummaryArtifact } from '../../renderers/ndjson.js'
import type { OutputContext } from '../../renderers/shared.js' import type { OutputContext } from '../../renderers/shared.js'
import { resolve } from 'node:path' import { resolve } from 'node:path'
import { pathToFileURL } from 'node:url'
const ROUTE_IDENTITY_PATTERN = /^[A-Z]+\s+\/\S*$/ const ROUTE_IDENTITY_PATTERN = /^[A-Z]+\s+\/\S*$/
@@ -98,24 +97,11 @@ export function generateSeed(): number {
/** /**
* Discover routes from the Fastify app for chaos execution. * Discover routes from the Fastify app for chaos execution.
* Injected fastify instance must have routes registered. * Uses the shared discovery module for capture + fallback support.
*/ */
async function discoverAppRoutes(fastify: unknown): Promise<RouteContract[]> { async function discoverAppRoutes(fastify: unknown): Promise<RouteContract[]> {
// Cast to access routes const { discoverRoutes } = await import('../../../domain/discovery.js')
const app = fastify as { routes?: Array<{ method: string; url: string; schema?: Record<string, unknown> }> } return discoverRoutes(fastify as { routes?: Array<{ method: string; url: string; schema?: Record<string, unknown> } >; hasRoute?: (opts: { method: string; url: string }) => boolean; printRoutes?: () => string })
if (!app.routes) return []
return app.routes.map(r => ({
path: r.url,
method: r.method as RouteContract['method'],
category: 'observer',
requires: [],
ensures: [],
invariants: [],
regexPatterns: {},
validateRuntime: false,
schema: r.schema,
}))
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -288,12 +274,51 @@ export function buildArtifact(
// Build execution summary from runner result // Build execution summary from runner result
const executionSummary = runResult.executionSummary const executionSummary = runResult.executionSummary
// Build per-gate coverage breakdown for clear artifact interpretation
const scenarioRoutes = [...new Set(
runResult.stepTraces
.filter(t => t.status === 'passed' || t.status === 'failed')
.map(t => t.route)
.filter(Boolean)
)]
const statefulRoutes = runResult.statefulResult
? [...new Set(runResult.statefulResult.tests.map(t => normalizeRouteIdentity(t.name)))]
: []
const chaosRoutesPlanned: string[] = []
const chaosRoutesExecuted: string[] = []
for (const r of runResult.chaosResults ?? []) {
if (r.applied) chaosRoutesExecuted.push(r.route)
}
// Infer planned from chaos results or executionSummary
if (executionSummary.chaosRoutesPlanned > 0 && chaosRoutesExecuted.length > 0) {
chaosRoutesPlanned.push(...chaosRoutesExecuted)
}
const coverageBreakdown = {
scenario: {
routesCovered: scenarioRoutes,
stepsTotal: executionSummary.totalSteps,
stepsPassed: runResult.scenarioResults.reduce((sum, s) => sum + s.summary.passed, 0),
},
stateful: {
routesCovered: statefulRoutes,
testsTotal: executionSummary.statefulTestsRun,
testsPassed: runResult.statefulResult?.summary.passed ?? 0,
},
chaos: {
routesPlanned: chaosRoutesPlanned,
routesExecuted: chaosRoutesExecuted,
runsTotal: executionSummary.chaosRunsRun,
runsPassed: (runResult.chaosResults ?? []).filter(r => r.contractsPassed).length,
},
}
// Build profile gates from the result context // Build profile gates from the result context
// We need to pass gates through or infer from results // We need to pass gates through or infer from results
const profileGates = { const profileGates = {
scenario: runResult.scenarioResults.length > 0 || executionSummary.scenariosRun > 0, scenario: runResult.scenarioResults.length > 0 || executionSummary.scenariosRun > 0,
stateful: (runResult.statefulResult?.tests.length ?? 0) > 0 || executionSummary.statefulTestsRun > 0, stateful: (runResult.statefulResult?.tests.length ?? 0) > 0 || executionSummary.statefulTestsRun > 0,
chaos: (runResult.chaosResult !== undefined) || executionSummary.chaosRunsRun > 0, chaos: executionSummary.chaosRoutesPlanned > 0,
} }
// Deterministic parameters for audit // Deterministic parameters for audit
@@ -320,6 +345,7 @@ export function buildArtifact(
failed: failures.length, failed: failures.length,
}, },
executionSummary, executionSummary,
coverageBreakdown,
executedRoutes: (runResult.executedRoutes || []).map(normalizeRouteIdentity), executedRoutes: (runResult.executedRoutes || []).map(normalizeRouteIdentity),
skippedRoutes: (runResult.skippedRoutes || []).map(sr => ({ skippedRoutes: (runResult.skippedRoutes || []).map(sr => ({
route: sr.route, route: sr.route,
@@ -386,117 +412,6 @@ async function emitArtifact(
// Output formatting // Output formatting
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
function formatHumanOutput(
result: QualifyRunResult,
options: { profile?: string; seed: number; env: string },
): string {
const lines: string[] = []
lines.push(`Qualify run for environment "${options.env}"`)
if (options.profile) {
lines.push(`Profile: ${options.profile}`)
}
lines.push(`Seed: ${options.seed}`)
lines.push('')
// Scenario results
for (const scenario of result.scenarioResults) {
lines.push(`Scenario: ${scenario.name}`)
for (const step of scenario.steps) {
const icon = step.ok ? '✓' : '✗'
lines.push(` ${icon} ${step.name} (${step.statusCode ?? 'no-status'})`)
if (!step.ok && step.diagnostics) {
lines.push(` Expected: ${step.diagnostics.expected || 'success'}`)
lines.push(` Observed: ${step.diagnostics.error || 'failure'}`)
if (step.diagnostics.actual) {
lines.push(` Actual: ${step.diagnostics.actual}`)
}
if (step.diagnostics.diff) {
lines.push(` Diff:`)
for (const line of String(step.diagnostics.diff).split('\n')) {
lines.push(` ${line}`)
}
}
}
}
lines.push('')
}
// Stateful results
if (result.statefulResult) {
lines.push(`Stateful: ${result.statefulResult.summary.passed} passed, ${result.statefulResult.summary.failed} failed`)
lines.push('')
}
// Chaos results
if (result.chaosResult) {
lines.push(`Chaos: ${result.chaosResult.applied ? 'applied' : 'none'}`)
if (result.chaosResult.events.length > 0) {
for (const event of result.chaosResult.events) {
lines.push(` ${event}`)
}
}
lines.push('')
}
// Step traces
if (result.stepTraces.length > 0) {
lines.push('Step traces:')
for (const trace of result.stepTraces.slice(0, 20)) {
const icon = trace.status === 'passed' ? '✓' : trace.status === 'skipped' ? '⊘' : '✗'
lines.push(` ${icon} ${trace.name} (${trace.durationMs}ms)`)
}
if (result.stepTraces.length > 20) {
lines.push(` ... and ${result.stepTraces.length - 20} more`)
}
lines.push('')
}
// Cleanup failures
if (result.cleanupFailures.length > 0) {
lines.push('Cleanup failures (reported separately):')
for (const cf of result.cleanupFailures) {
lines.push(` ⚠ ${cf.resource}: ${cf.error}`)
}
lines.push('')
}
// Per-profile gate execution counts
lines.push('Profile gate execution counts:')
lines.push(` Scenario: ${result.executionSummary.scenariosRun} run`)
lines.push(` Stateful: ${result.executionSummary.statefulTestsRun} tests run`)
lines.push(` Chaos: ${result.executionSummary.chaosRunsRun} runs run`)
lines.push('')
// Executed routes
if (result.executedRoutes.length > 0) {
lines.push(`Executed routes (${result.executedRoutes.length}):`)
for (const route of result.executedRoutes) {
lines.push(` ${route}`)
}
lines.push('')
}
// Skipped routes
if (result.skippedRoutes.length > 0) {
lines.push(`Skipped routes (${result.skippedRoutes.length}):`)
for (const sr of result.skippedRoutes) {
lines.push(` ${sr.route}: ${sr.reason}`)
}
lines.push('')
}
// Summary
if (result.passed) {
lines.push('All qualifications passed.')
} else {
lines.push('Qualification failed.')
lines.push(`Replay: apophis replay --artifact <artifact-path>`)
}
return lines.join('\n')
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Main command handler // Main command handler
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -583,9 +498,26 @@ export async function qualifyCommand(
const profileDef = profile ? config.profiles?.[profile] : undefined const profileDef = profile ? config.profiles?.[profile] : undefined
const gates = resolveProfileGates(profileDef?.features) const gates = resolveProfileGates(profileDef?.features)
// 5. Build scenario configs from profile routes // 5. Build scenario configs from profile routes and config
const routes = profileDef?.routes ?? [] const routes = profileDef?.routes ?? []
const scenarios = buildScenarioConfigs(routes, seed) const fixtureScenarios = buildScenarioConfigs(routes, seed)
const configScenarios: ScenarioConfig[] = (config.scenarios ?? []).map(s => ({
name: s.name,
steps: s.steps.map(step => ({
name: step.name,
request: {
method: step.request.method as ScenarioConfig['steps'][0]['request']['method'],
url: step.request.url,
body: step.request.body,
headers: step.request.headers,
},
expect: step.expect,
capture: step.capture,
})),
}))
const scenarios = [...fixtureScenarios, ...configScenarios]
// 6. Build stateful config // 6. Build stateful config
const presetName = profileDef?.preset const presetName = profileDef?.preset
@@ -603,6 +535,9 @@ export async function qualifyCommand(
const chaosConfig: ChaosConfig | undefined = gates.chaos && preset?.chaos const chaosConfig: ChaosConfig | undefined = gates.chaos && preset?.chaos
? { ? {
probability: 0.5, probability: 0.5,
strategy: preset.chaosStrategy as ChaosConfig['strategy'],
sampleSize: preset.chaosSampleSize,
sampleRoutes: preset.chaosSampleRoutes,
delay: { probability: 0.3, minMs: 100, maxMs: 500 }, delay: { probability: 0.3, minMs: 100, maxMs: 500 },
error: { probability: 0.2, statusCode: 503 }, error: { probability: 0.2, statusCode: 503 },
dropout: { probability: 0.2, statusCode: 504 }, dropout: { probability: 0.2, statusCode: 504 },
@@ -611,32 +546,41 @@ export async function qualifyCommand(
: undefined : undefined
// 8. Load the Fastify app for execution // 8. Load the Fastify app for execution
// Try to import the app from the fixture
let fastify: FastifyAppLike | undefined let fastify: FastifyAppLike | undefined
let appEntrypoint: string | undefined
try { try {
const appPath = resolve(workingDir, 'app.js') const { loadApp } = await import('../../core/app-loader.js')
const appUrl = pathToFileURL(appPath) const loaded = await loadApp(workingDir)
appUrl.searchParams.set('apophisRun', String(Date.now())) fastify = loaded.fastify as FastifyAppLike
const appModule = await import(appUrl.href) appEntrypoint = loaded.entrypoint
fastify = (appModule.default || appModule) as FastifyAppLike
if (fastify && typeof fastify.ready === 'function') { if (fastify && typeof fastify.ready === 'function') {
await fastify.ready() await fastify.ready()
} }
} catch (err) { } catch (err) {
// App not available — return a result indicating no app to test const errorMessage = err instanceof Error ? err.message : String(err)
if (process.env.APOPHIS_DEBUG === '1') {
console.error('Failed to load app:', err)
}
return { return {
exitCode: USAGE_ERROR, exitCode: USAGE_ERROR,
message: 'No Fastify app found. Ensure app.js exports a Fastify instance or a factory function.\n\nSupported patterns:\n export default app\n export const createApp = () => app\n module.exports = app', message: `Failed to load Fastify app.\n\nError: ${errorMessage}\n\nApp file candidates (searched in order):\n app.js, app.ts, server.js, server.ts, index.js, index.ts\n src/app.js, src/app.ts, src/server.js, src/server.ts, src/index.js, src/index.ts\n\nNext:\n Run \`apophis init\` to scaffold a working app.js and config.`,
} }
} }
try { try {
// 9. Discover routes for chaos // 9. Discover routes for chaos (with discovery metadata)
const appRoutes = await discoverAppRoutes(fastify) const appRoutes = await discoverAppRoutes(fastify)
// Collect discovery warnings
const { discoverRouteDetails } = await import('../../../domain/discovery.js')
const discoveryResult = discoverRouteDetails(fastify as { routes?: Array<{ method: string; url: string; schema?: Record<string, unknown> } >; hasRoute?: (opts: { method: string; url: string }) => boolean; printRoutes?: () => string })
const discoveryWarnings: string[] = [
...discoveryResult.warnings,
]
if (discoveryResult.source === 'print-routes' && !discoveryResult.hasSchemaMetadata) {
discoveryWarnings.push(
'Routes were discovered without schema metadata. Behavioral contracts and chaos injection may be limited. ' +
'Register APOPHIS before defining routes or use createFastify().'
)
}
// 10. Run qualify execution // 10. Run qualify execution
const deps = { const deps = {
fastify: fastify as any, fastify: fastify as any,
@@ -658,7 +602,7 @@ export async function qualifyCommand(
// 12. Signal quality guardrails — fail if zero checks executed // 12. Signal quality guardrails — fail if zero checks executed
const execSummary = runResult.executionSummary const execSummary = runResult.executionSummary
const warnings: string[] = [...artifact.warnings] const warnings: string[] = [...artifact.warnings, ...discoveryWarnings]
if (execSummary.totalExecuted === 0) { if (execSummary.totalExecuted === 0) {
await emitArtifact(artifact, { await emitArtifact(artifact, {
src/cli/commands/qualify/runner.ts
+90 -18
View File
@@ -48,6 +48,7 @@ export interface QualifyRunResult {
scenarioResults: ScenarioResult[] scenarioResults: ScenarioResult[]
statefulResult?: TestSuite statefulResult?: TestSuite
chaosResult?: ChaosRunResult chaosResult?: ChaosRunResult
chaosResults: ChaosRunResult[]
stepTraces: StepTrace[] stepTraces: StepTrace[]
cleanupFailures: CleanupFailure[] cleanupFailures: CleanupFailure[]
durationMs: number durationMs: number
@@ -60,6 +61,8 @@ export interface QualifyRunResult {
scenariosRun: number scenariosRun: number
statefulTestsRun: number statefulTestsRun: number
chaosRunsRun: number chaosRunsRun: number
chaosRoutesPlanned: number
chaosRoutesExecuted: number
totalSteps: number totalSteps: number
} }
executedRoutes: string[] executedRoutes: string[]
@@ -71,6 +74,7 @@ export interface ChaosRunResult {
events: string[] events: string[]
route: string route: string
durationMs: number durationMs: number
contractsPassed: boolean
} }
export interface CleanupFailure { export interface CleanupFailure {
@@ -110,6 +114,45 @@ export function resolveProfileGates(features?: string[]): ProfileGates {
} }
} }
/**
* Select routes for chaos injection based on strategy.
*
* - 'one': picks one route deterministically (default, backward-compatible)
* - 'all': every provided route
* - 'sample': randomly sampled subset
* - 'routes': explicit list matched against route strings "METHOD /path"
*/
export function selectChaosRoutes(
allRoutes: RouteContract[],
config: ChaosConfig,
seed: number,
): RouteContract[] {
const strategy = config.strategy ?? 'one'
const rng = new SeededRng(seed)
if (strategy === 'all') {
return [...allRoutes]
}
if (strategy === 'sample') {
const size = Math.min(config.sampleSize ?? 3, allRoutes.length)
const shuffled = [...allRoutes].sort(() => rng.next() - 0.5)
return shuffled.slice(0, size)
}
if (strategy === 'routes' && config.sampleRoutes) {
return allRoutes.filter(r => {
const id = `${r.method} ${r.path}`
return config.sampleRoutes!.some(p => id === p)
})
}
// 'one' — default: pick one route deterministically
const idx = Math.floor(rng.next() * allRoutes.length)
const route = allRoutes[idx]
return route ? [route] : []
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Main qualify runner // Main qualify runner
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -125,13 +168,14 @@ export async function runQualify(
statefulConfig?: TestConfig, statefulConfig?: TestConfig,
chaosConfig?: ChaosConfig, chaosConfig?: ChaosConfig,
routes?: RouteContract[], routes?: RouteContract[],
cleanupManager?: import('../../../infrastructure/cleanup-manager.js').CleanupManager,
): Promise<QualifyRunResult> { ): Promise<QualifyRunResult> {
const started = Date.now() const started = Date.now()
const scenarioResults: ScenarioResult[] = [] const scenarioResults: ScenarioResult[] = []
const allTraces: StepTrace[] = [] const allTraces: StepTrace[] = []
const cleanupFailures: CleanupFailure[] = [] const cleanupFailures: CleanupFailure[] = []
let statefulResult: TestSuite | undefined let statefulResult: TestSuite | undefined
let chaosResult: ChaosRunResult | undefined const chaosResults: ChaosRunResult[] = []
// Run scenarios // Run scenarios
if (gates.scenario) { if (gates.scenario) {
@@ -149,37 +193,59 @@ export async function runQualify(
allTraces.push(...traces) allTraces.push(...traces)
} }
// Run chaos on routes // Run chaos on selected routes
let chaosRoutesPlanned = 0
let chaosRoutesExecuted = 0
if (gates.chaos && chaosConfig && routes && routes.length > 0) { if (gates.chaos && chaosConfig && routes && routes.length > 0) {
// Pick one route deterministically for CLI chaos demo const selectedRoutes = selectChaosRoutes(routes, chaosConfig, deps.seed)
const rng = new SeededRng(deps.seed) chaosRoutesPlanned = selectedRoutes.length
const route = routes[Math.floor(rng.next() * routes.length)]
if (route) { for (const route of selectedRoutes) {
try {
const { chaosResult: cr } = await runChaosOnRoute(deps, route, chaosConfig) const { chaosResult: cr } = await runChaosOnRoute(deps, route, chaosConfig)
chaosResult = cr chaosResults.push(cr)
if (cr.applied) {
chaosRoutesExecuted++
}
} catch {
// Individual chaos run failure should not abort the qualify run
}
} }
} }
// Simulate cleanup tracking // Run cleanup and track failures
// In real usage, cleanupManager would be injected and tracked if (cleanupManager) {
// For now, cleanup failures are empty unless injected by caller try {
const outcomes = await cleanupManager.cleanup()
for (const outcome of outcomes) {
if (outcome.error) {
cleanupFailures.push({
resource: `${outcome.resource.type}/${outcome.resource.id} (${outcome.resource.url})`,
error: outcome.error,
})
}
}
} catch {
// Cleanup failures are tracked but should not block the qualify result
}
}
const durationMs = Date.now() - started const durationMs = Date.now() - started
// Determine overall pass/fail // Determine overall pass/fail
const scenarioPassed = scenarioResults.every(r => r.ok) const scenarioPassed = scenarioResults.every(r => r.ok)
const statefulPassed = !statefulResult || statefulResult.summary.failed === 0 const statefulPassed = !statefulResult || statefulResult.summary.failed === 0
const chaosPassed = !chaosResult || chaosResult.applied // chaos "passes" if it applied const chaosPassed = chaosResults.every(r => !r.applied || r.contractsPassed)
// Count execution metrics // Count execution metrics
const scenariosRun = scenarioResults.length const scenariosRun = scenarioResults.length
const statefulTestsRun = statefulResult?.tests.length ?? 0 const statefulTestsRun = statefulResult?.tests.length ?? 0
const chaosRunsRun = chaosResult ? 1 : 0 const chaosRunsRun = chaosResults.length
const totalSteps = allTraces.length const totalSteps = allTraces.length
const totalExecuted = scenariosRun + statefulTestsRun + chaosRunsRun const totalExecuted = scenariosRun + statefulTestsRun + chaosRunsRun
const totalPassed = scenarioResults.reduce((sum, r) => sum + r.summary.passed, 0) + const totalPassed = scenarioResults.reduce((sum, r) => sum + r.summary.passed, 0) +
(statefulResult?.summary.passed ?? 0) + (statefulResult?.summary.passed ?? 0) +
(chaosResult?.applied ? 1 : 0) chaosResults.reduce((sum, r) => sum + (r.applied && r.contractsPassed ? 1 : 0), 0)
const totalFailed = scenarioResults.reduce((sum, r) => sum + r.summary.failed, 0) + const totalFailed = scenarioResults.reduce((sum, r) => sum + r.summary.failed, 0) +
(statefulResult?.summary.failed ?? 0) (statefulResult?.summary.failed ?? 0)
@@ -204,9 +270,9 @@ export async function runQualify(
} }
} }
// Track chaos route // Track chaos routes
if (chaosResult) { for (const cr of chaosResults) {
executedRoutes.push(chaosResult.route) executedRoutes.push(cr.route)
} }
// Track skipped routes from profile filters // Track skipped routes from profile filters
@@ -224,6 +290,9 @@ export async function runQualify(
reason = 'Stateful config missing or invalid' reason = 'Stateful config missing or invalid'
} else if (gates.chaos && !chaosConfig) { } else if (gates.chaos && !chaosConfig) {
reason = 'Chaos config missing or invalid' reason = 'Chaos config missing or invalid'
} else if (gates.chaos && chaosConfig) {
const strategy = chaosConfig.strategy ?? 'one'
reason = `Not selected by chaos strategy: ${strategy}`
} }
skippedRoutes.push({ route: routeStr, reason }) skippedRoutes.push({ route: routeStr, reason })
} }
@@ -234,19 +303,22 @@ export async function runQualify(
passed: scenarioPassed && statefulPassed && chaosPassed, passed: scenarioPassed && statefulPassed && chaosPassed,
scenarioResults, scenarioResults,
statefulResult, statefulResult,
chaosResult, chaosResult: chaosResults[0],
chaosResults,
stepTraces: allTraces, stepTraces: allTraces,
cleanupFailures, cleanupFailures,
durationMs, durationMs,
seed: deps.seed, seed: deps.seed,
executionSummary: { executionSummary: {
totalPlanned: scenarios.length + (statefulConfig ? 1 : 0) + (chaosConfig && routes && routes.length > 0 ? 1 : 0), totalPlanned: scenarios.length + (statefulConfig ? 1 : 0) + chaosRoutesPlanned,
totalExecuted, totalExecuted,
totalPassed, totalPassed,
totalFailed, totalFailed,
scenariosRun, scenariosRun,
statefulTestsRun, statefulTestsRun,
chaosRunsRun, chaosRunsRun,
chaosRoutesPlanned,
chaosRoutesExecuted,
totalSteps, totalSteps,
}, },
executedRoutes: [...new Set(executedRoutes)], executedRoutes: [...new Set(executedRoutes)],
src/cli/commands/qualify/scenario-handler.ts
+1 -1
View File
@@ -10,7 +10,7 @@
* - No optional imports — everything is passed via parameters * - No optional imports — everything is passed via parameters
*/ */
import { runScenario } from '../../../test/scenario-runner.js' import { runScenario } from '../../../quality/scenario-runner.js'
import type { import type {
ScenarioConfig, ScenarioConfig,
ScenarioResult, ScenarioResult,
src/cli/commands/qualify/stateful-handler.ts
+1 -1
View File
@@ -10,7 +10,7 @@
* - No optional imports — everything is passed via parameters * - No optional imports — everything is passed via parameters
*/ */
import { runStatefulTests } from '../../../test/stateful-runner.js' import { runStatefulTests } from '../../../quality/stateful-runner.js'
import { CleanupManager } from '../../../infrastructure/cleanup-manager.js' import { CleanupManager } from '../../../infrastructure/cleanup-manager.js'
import type { import type {
TestConfig, TestConfig,
src/cli/commands/replay/index.ts
+1 -1
View File
@@ -256,7 +256,7 @@ async function executeReplay(
const errorMessage = err instanceof Error ? err.message : String(err) const errorMessage = err instanceof Error ? err.message : String(err)
return { return {
exitCode: USAGE_ERROR, exitCode: USAGE_ERROR,
message: `Cannot load Fastify app from ${workingDir}/app.js: ${errorMessage}`, message: `Cannot load Fastify app: ${errorMessage}`,
warnings, warnings,
reproduced: false, reproduced: false,
originalFailure: failure, originalFailure: failure,
src/cli/commands/replay/loader.ts
+1 -1
View File
@@ -26,7 +26,7 @@ import type { Artifact, FailureRecord } from '../../core/types.js';
const SUPPORTED_ARTIFACT_VERSION = 'apophis-artifact/1'; const SUPPORTED_ARTIFACT_VERSION = 'apophis-artifact/1';
/** Current CLI version for compatibility checks */ /** Current CLI version for compatibility checks */
const CLI_VERSION = '2.0.0'; const CLI_VERSION = '2.7.0';
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Types // Types
src/cli/commands/verify/index.ts
+15 -104
View File
@@ -117,20 +117,26 @@ function buildArtifact(
} }
return { return {
route, route,
contract: f.contract, contract: f.formula || f.contract,
expected: f.expected, expected: f.expected,
observed: f.observed, observed: f.observed,
seed: options.seed, seed: options.seed,
replayCommand: `apophis replay --artifact ${f.artifactPath || '<artifact-path-unavailable>'}`, replayCommand: `apophis replay --artifact ${f.artifactPath || '<artifact-path-unavailable>'}`,
category: f.observed ? classifyError(f.observed) : ErrorTaxonomy.RUNTIME, category: f.category ?? (f.observed ? classifyError(f.observed) : ErrorTaxonomy.RUNTIME),
} }
}) })
if (runResult.noContractsFound) { if (runResult.noContractsFound) {
warnings.push('No behavioral contracts found. Schema-only routes are not enough for verify. Add x-ensures or x-requires to route schemas. See docs/getting-started.md for examples.') warnings.push('No behavioral contracts found. Schema-only routes are not enough for verify. Add x-ensures or x-requires to route schemas. See docs/getting-started.md for examples.')
if (runResult.discoveryWarnings && runResult.discoveryWarnings.length > 0) {
warnings.push(...runResult.discoveryWarnings)
}
} }
if (runResult.noRoutesMatched) { if (runResult.noRoutesMatched) {
warnings.push(`No routes matched the filter. Available routes: ${runResult.availableRoutes?.join(', ') || 'none'}`) warnings.push(`No routes matched the filter. Available routes: ${runResult.availableRoutes?.join(', ') || 'none'}`)
if (runResult.discoveryWarnings && runResult.discoveryWarnings.length > 0) {
warnings.push(...runResult.discoveryWarnings)
}
} }
if (runResult.notGitRepo) { if (runResult.notGitRepo) {
warnings.push('--changed requires a git repository. Current directory is not inside a git repo.') warnings.push('--changed requires a git repository. Current directory is not inside a git repo.')
@@ -149,7 +155,7 @@ function buildArtifact(
return { return {
version: 'apophis-artifact/1', version: 'apophis-artifact/1',
cliVersion: '2.0.0', cliVersion: '2.7.0',
command: 'verify', command: 'verify',
mode: 'verify', mode: 'verify',
cwd: options.cwd, cwd: options.cwd,
@@ -255,106 +261,6 @@ function formatHumanFailure(failure: FailureRecord, profile?: string): string {
} }
/** /**
* Format human-readable output for verify results.
*/
function formatHumanOutput(
runResult: VerifyRunResult,
options: { profile?: string; seed: number; env: string; routeFilters?: string[] },
): string {
const lines: string[] = []
if (runResult.notGitRepo) {
lines.push(`--changed requires a git repository.`)
lines.push(`Current directory is not inside a git repo.`)
lines.push('')
lines.push('Next:')
lines.push(` Initialize git with \`git init\`, or run verify without --changed.`)
lines.push('')
return lines.join('\n')
}
if (runResult.noRelevantChanges) {
lines.push(`No relevant changes detected.`)
lines.push(`Git shows no modified files that match any route.`)
lines.push('')
return lines.join('\n')
}
if (runResult.noRoutesMatched) {
lines.push(`No routes matched the filter.`)
lines.push(`Filters applied: ${options.routeFilters?.join(', ') || 'none'}`)
lines.push(`Available routes:`)
for (const r of runResult.availableRoutes || []) {
lines.push(` ${r}`)
}
lines.push('')
lines.push('Next:')
lines.push(` Adjust --routes filter or add routes to your app.`)
lines.push('')
return lines.join('\n')
}
if (runResult.noContractsFound) {
lines.push('No behavioral contracts found.')
lines.push('')
lines.push('APOPHIS discovered routes, but none have behavioral contracts.')
lines.push('Schema-only routes (with response schemas) are not enough.')
lines.push('You must add x-ensures or x-requires clauses that check behavior.')
lines.push('')
lines.push('Example — add this to your route schema:')
lines.push(' "x-ensures": [')
lines.push(' "response_code(GET /users/{response_body(this).id}) == 200"')
lines.push(' ]')
lines.push('')
lines.push('Next steps:')
lines.push(' 1. Open your route file (e.g., app.js or src/routes/users.js)')
lines.push(' 2. Find the route you want to test')
lines.push(' 3. Add an "x-ensures" array inside the schema object')
lines.push(' 4. Run: apophis verify --profile quick --routes "POST /users"')
lines.push('')
lines.push('For more examples, see docs/getting-started.md')
lines.push('')
return lines.join('\n')
}
// Print failures using canonical format
for (const failure of runResult.failures) {
const failureRecord: FailureRecord = {
route: failure.route,
contract: failure.contract,
expected: failure.expected,
observed: failure.observed,
seed: options.seed,
replayCommand: `apophis replay --artifact ${failure.artifactPath || 'reports/apophis/failure-*.json'}`,
}
lines.push(formatHumanFailure(failureRecord, options.profile))
lines.push('')
}
// Summary
if (runResult.passed) {
lines.push(`All ${runResult.total} contract(s) passed.`)
} else {
lines.push(`Failed: ${runResult.failed} of ${runResult.total} contract(s) failed.`)
}
lines.push(`Seed: ${options.seed}`)
// Replay command on failure
if (!runResult.passed && runResult.failures.length > 0) {
lines.push('')
lines.push('Replay')
lines.push(` apophis replay --artifact <path-to-artifact>`)
lines.push('')
lines.push('Determinism')
lines.push(` This run used seed ${options.seed}.`)
lines.push(` Same seed + same app state = same results.`)
lines.push(` If results differ on re-run, the app has nondeterministic behavior.`)
lines.push(` Stabilize: reset app state, mock external services, avoid time-dependent logic.`)
}
return lines.join('\n')
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Main command handler // Main command handler
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -454,10 +360,12 @@ export async function verifyCommand(
// 5. Load the Fastify app // 5. Load the Fastify app
let fastify: unknown let fastify: unknown
let appEntrypoint: string | undefined
try { try {
const { loadApp } = await import('../../core/app-loader.js') const { loadApp } = await import('../../core/app-loader.js')
const loaded = await loadApp(workingDir) const loaded = await loadApp(workingDir)
fastify = loaded.fastify fastify = loaded.fastify
appEntrypoint = loaded.entrypoint
if (fastify && typeof (fastify as any).ready === 'function') { if (fastify && typeof (fastify as any).ready === 'function') {
await (fastify as any).ready() await (fastify as any).ready()
} }
@@ -465,7 +373,7 @@ export async function verifyCommand(
const errorMessage = err instanceof Error ? err.message : String(err) const errorMessage = err instanceof Error ? err.message : String(err)
return { return {
exitCode: USAGE_ERROR, exitCode: USAGE_ERROR,
message: `No Fastify app found. Ensure app.js exports a Fastify instance or a factory function.\n\nSupported patterns:\n export default app\n export const createApp = () => app\n module.exports = app\n\nError: ${errorMessage}\n\nNext:\n Run \`apophis init\` to scaffold a working app.js and config.`, message: `Failed to load Fastify app.\n\nError: ${errorMessage}\n\nApp file candidates (searched in order):\n app.js, app.ts, server.js, server.ts, index.js, index.ts\n src/app.js, src/app.ts, src/server.js, src/server.ts, src/index.js, src/index.ts\n\nNext:\n Run \`apophis init\` to scaffold a working app.js and config.`,
} }
} }
@@ -479,6 +387,9 @@ export async function verifyCommand(
routeFilters, routeFilters,
changed, changed,
profileRoutes: config.profiles?.[profile || '']?.routes, profileRoutes: config.profiles?.[profile || '']?.routes,
runs: typeof config.presets?.[loadResult.presetName || '']?.runs === 'number'
? (config.presets[loadResult.presetName || ''] as { runs?: number }).runs
: undefined,
}) })
// 7. Build artifact // 7. Build artifact
src/cli/commands/verify/runner.ts
+121 -21
View File
@@ -20,6 +20,10 @@ import { executeHttp } from '../../../infrastructure/http-executor.js'
import { parse } from '../../../formula/parser.js' import { parse } from '../../../formula/parser.js'
import { evaluateAsync } from '../../../formula/evaluator.js' import { evaluateAsync } from '../../../formula/evaluator.js'
import { createOperationResolver } from '../../../formula/runtime.js' import { createOperationResolver } from '../../../formula/runtime.js'
import { buildRequest } from '../../../domain/request-builder.js'
import { convertSchema } from '../../../domain/schema-to-arbitrary.js'
import * as fc from 'fast-check'
import { resolveRuns } from '../../../types.js'
import type { EvalContext, RouteContract, FastifyInjectInstance } from '../../../types.js' import type { EvalContext, RouteContract, FastifyInjectInstance } from '../../../types.js'
import type { RouteResult } from '../../core/types.js' import type { RouteResult } from '../../core/types.js'
@@ -33,6 +37,8 @@ export interface VerifyFailure {
expected: string expected: string
observed: string observed: string
artifactPath?: string artifactPath?: string
formula?: string
category?: string
} }
export interface VerifyRunResult { export interface VerifyRunResult {
@@ -48,6 +54,8 @@ export interface VerifyRunResult {
noRelevantChanges?: boolean noRelevantChanges?: boolean
availableRoutes?: string[] availableRoutes?: string[]
artifactPaths: string[] artifactPaths: string[]
discoveryWarnings?: string[]
runs: number
} }
export interface VerifyRunnerDeps { export interface VerifyRunnerDeps {
@@ -57,6 +65,7 @@ export interface VerifyRunnerDeps {
routeFilters?: string[] routeFilters?: string[]
changed?: boolean changed?: boolean
profileRoutes?: string[] profileRoutes?: string[]
runs?: number
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -203,28 +212,58 @@ async function filterChangedRoutes(
/** /**
* Build a request for a route. * Build a request for a route.
*/ */
function buildRouteRequest(route: RouteContract): { function buildRouteRequest(
route: RouteContract,
seed?: number,
runIndex?: number,
variant?: { name: string; headers?: Record<string, string> },
): {
method: string method: string
url: string url: string
body?: unknown body?: unknown
query?: Record<string, string>
headers: Record<string, string> headers: Record<string, string>
} { } {
const headers: Record<string, string> = { let generatedData: Record<string, unknown> = {}
'content-type': 'application/json',
const bodySchema = route.schema?.body as Record<string, unknown> | undefined
if (bodySchema && seed !== undefined) {
try {
const bodyArb = convertSchema(bodySchema, { context: 'request' })
const bodySeed = seed + (runIndex ?? 0) * 31
const samples = fc.sample(bodyArb, { numRuns: 1, seed: bodySeed })
const bodySample = samples[0]
if (bodySample !== null && typeof bodySample === 'object') {
generatedData = bodySample as Record<string, unknown>
}
} catch {
// fall through to example-based generation
}
} }
// Build body from schema if available if (!generatedData || Object.keys(generatedData).length === 0) {
let body: unknown = undefined generatedData = buildExampleBody(bodySchema ?? {}) as Record<string, unknown> ?? {}
const bodySchema = route.schema?.body as Record<string, unknown> | undefined }
if (bodySchema && route.method === 'POST') {
body = buildExampleBody(bodySchema) const request = buildRequest(
route,
generatedData,
variant?.headers ?? ({} as Record<string, string>),
{ resources: new Map(), counters: new Map() },
)
const headers: Record<string, string> = {
'content-type': 'application/json',
...request.headers,
...(variant?.headers ?? {}),
} }
return { return {
method: route.method, method: request.method,
url: route.path, url: request.url,
body, body: request.body,
headers, headers,
query: request.query,
} }
} }
@@ -268,18 +307,62 @@ function buildExampleValue(schema: Record<string, unknown>): unknown {
return undefined return undefined
} }
/** function buildFailureDiagnostic(
* Execute a single contract for a route. contract: string,
* Returns the evaluation context and any failure. evalCtx: EvalContext,
*/ evalError?: string,
): Pick<VerifyFailure, 'expected' | 'observed' | 'formula' | 'category'> {
if (evalError) {
return { expected: 'true', observed: evalError, formula: contract, category: 'runtime' }
}
const status = evalCtx.response.statusCode
const statusMatch = contract.match(/^status:(\d+)$/i)
if (statusMatch) {
return { expected: `HTTP ${statusMatch[1]}`, observed: `HTTP ${status}`, formula: contract, category: 'runtime' }
}
const fieldMatch = contract.match(/response_body\(this\)\.([\w.]+)\s*(!=|==)\s*null/i)
if (fieldMatch) {
const path = fieldMatch[1]!
const negated = fieldMatch[2] === '!='
const parts = path.split('.')
let value: unknown = evalCtx.response.body
for (const p of parts) {
if (value != null && typeof value === 'object') {
value = (value as Record<string, unknown>)[p]
} else { value = undefined; break }
}
const actual = value === undefined ? 'undefined' : value === null ? 'null' : `"${String(value)}"`
return {
expected: negated ? `${path} != null` : `${path} == null`,
observed: `${path} is ${actual}`,
formula: contract, category: 'runtime',
}
}
const codeMatch = contract.match(/response_code\(this\)\s*(==|!=|<|>|<=|>=)\s*(\d+)/i)
if (codeMatch) {
return {
expected: `response_code == ${codeMatch[2]}`,
observed: `response_code is ${status}`,
formula: contract, category: 'runtime',
}
}
return { expected: contract, observed: `false`, formula: contract, category: 'runtime' }
}
async function executeContract( async function executeContract(
fastify: FastifyInjectInstance, fastify: FastifyInjectInstance,
route: RouteContract, route: RouteContract,
contract: string, contract: string,
timeout?: number, timeout?: number,
variant?: { name: string; headers?: Record<string, string> }, variant?: { name: string; headers?: Record<string, string> },
seed?: number,
runIndex?: number,
): Promise<{ ctx: EvalContext; failure?: VerifyFailure }> { ): Promise<{ ctx: EvalContext; failure?: VerifyFailure }> {
const request = buildRouteRequest(route) const request = buildRouteRequest(route, seed, runIndex, variant)
// Merge variant headers if provided // Merge variant headers if provided
const headers = variant?.headers const headers = variant?.headers
@@ -307,6 +390,7 @@ async function executeContract(
const result = await evaluateAsync(parsed.ast, evalCtx) const result = await evaluateAsync(parsed.ast, evalCtx)
if (!result.success || !result.value) { if (!result.success || !result.value) {
const diagnostic = buildFailureDiagnostic(contract, evalCtx, result.success ? undefined : result.error)
return { return {
ctx: evalCtx, ctx: evalCtx,
failure: { failure: {
@@ -314,14 +398,17 @@ async function executeContract(
? `[variant:${variant.name}] ${route.method} ${route.path}` ? `[variant:${variant.name}] ${route.method} ${route.path}`
: `${route.method} ${route.path}`, : `${route.method} ${route.path}`,
contract, contract,
expected: 'true', expected: diagnostic.expected,
observed: result.success ? String(result.value) : result.error, observed: diagnostic.observed,
formula: diagnostic.formula,
category: diagnostic.category,
}, },
} }
} }
return { ctx: evalCtx } return { ctx: evalCtx }
} catch (error) { } catch (error) {
const diagnostic = buildFailureDiagnostic(contract, evalCtx, error instanceof Error ? error.message : String(error))
return { return {
ctx: evalCtx, ctx: evalCtx,
failure: { failure: {
@@ -329,8 +416,10 @@ async function executeContract(
? `[variant:${variant.name}] ${route.method} ${route.path}` ? `[variant:${variant.name}] ${route.method} ${route.path}`
: `${route.method} ${route.path}`, : `${route.method} ${route.path}`,
contract, contract,
expected: 'true', expected: diagnostic.expected,
observed: error instanceof Error ? error.message : String(error), observed: diagnostic.observed,
formula: diagnostic.formula,
category: diagnostic.category,
}, },
} }
} }
@@ -399,6 +488,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
availableRoutes, availableRoutes,
artifactPaths: [], artifactPaths: [],
notGitRepo: true, notGitRepo: true,
runs: 0,
} }
} }
routes = await filterChangedRoutes(routes, cwd) routes = await filterChangedRoutes(routes, cwd)
@@ -417,6 +507,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
noContractsFound: false, noContractsFound: false,
availableRoutes, availableRoutes,
artifactPaths: [], artifactPaths: [],
runs: 0,
} }
} }
@@ -437,10 +528,13 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
noContractsFound: true, noContractsFound: true,
availableRoutes, availableRoutes,
artifactPaths: [], artifactPaths: [],
runs: 0,
} }
} }
// 4. Execute contracts (with variant expansion) // 4. Execute contracts (with variant expansion)
const runConfig = resolveRuns(deps.runs)
const runs = runConfig.contractRuns
const failures: VerifyFailure[] = [] const failures: VerifyFailure[] = []
let total = 0 let total = 0
let passedCount = 0 let passedCount = 0
@@ -453,8 +547,12 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
for (const variant of variants) { for (const variant of variants) {
for (const contract of contracts) { for (const contract of contracts) {
for (let runIndex = 0; runIndex < runs; runIndex++) {
total++ total++
const result = await executeContract(fastify, route, contract, deps.timeout, variant) const result = await executeContract(
fastify, route, contract, deps.timeout,
variant, deps.seed, runIndex,
)
if (result.failure) { if (result.failure) {
failures.push(result.failure) failures.push(result.failure)
@@ -464,6 +562,7 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
} }
} }
} }
}
const durationMs = Date.now() - started const durationMs = Date.now() - started
@@ -485,5 +584,6 @@ export async function runVerify(deps: VerifyRunnerDeps): Promise<VerifyRunResult
noContractsFound: false, noContractsFound: false,
availableRoutes, availableRoutes,
artifactPaths: [], artifactPaths: [],
runs,
} }
} }
src/cli/core/app-loader.ts
+146 -34
View File
@@ -1,99 +1,211 @@
/** /**
* App loader utility for CLI commands. * App loader utility for CLI commands.
* Handles various app export patterns and module systems. * Handles various app export patterns and module systems.
*
* Supports auto-detection of entrypoint files:
* app.js, app.ts, server.js, server.ts, index.js, index.ts,
* src/app.js, src/app.ts, src/server.js, src/server.ts,
* src/index.js, src/index.ts
*/ */
import { existsSync } from 'node:fs'
import { resolve } from 'node:path' import { resolve, extname } from 'node:path'
import { pathToFileURL } from 'node:url' import { pathToFileURL } from 'node:url'
function isTsxAvailable(): boolean {
return process.execArgv.some(arg => arg.includes('tsx') || arg.includes('ts-node')) ||
process.argv[0]?.includes('tsx') ||
!!process.env.TSX_TSCONFIG_PATH
}
export const APP_CANDIDATES = [
'app.js',
'app.ts',
'server.js',
'server.ts',
'index.js',
'index.ts',
'src/app.js',
'src/app.ts',
'src/server.js',
'src/server.ts',
'src/index.js',
'src/index.ts',
]
export interface LoadedApp { export interface LoadedApp {
fastify: unknown fastify: unknown
source: 'default' | 'named' | 'commonjs' source: 'default' | 'named' | 'commonjs'
entrypoint: string
} }
/** /**
* Load a Fastify app from app.js in the given directory. * Find the first existing app entrypoint file in the given directory.
* Supports: * Returns the resolved absolute path, or null if no candidate exists.
*/
export function findAppFile(cwd: string): string | null {
for (const candidate of APP_CANDIDATES) {
const fullPath = resolve(cwd, candidate)
if (existsSync(fullPath)) {
return fullPath
}
}
return null
}
/**
* Load a Fastify app from the given directory.
*
* Auto-detects the entrypoint by searching APP_CANDIDATES for the first
* existing file. Supports:
* - ESM default export: export default fastifyInstance * - ESM default export: export default fastifyInstance
* - ESM named export: export const createApp = () => fastifyInstance * - ESM named export: export const createApp = () => fastifyInstance
* - CommonJS: module.exports = fastifyInstance * - CommonJS: module.exports = fastifyInstance
* - CommonJS named: exports.createApp = () => fastifyInstance * - CommonJS named: exports.createApp = () => fastifyInstance
*
* If an explicit entrypoint is provided, it takes precedence over auto-detection.
* For .ts entrypoints, a clear error message suggests installing tsx.
*
* After loading the app, this function ensures route discovery works even if the
* APOPHIS plugin is not registered or was registered after routes.
*/ */
export async function loadApp(cwd: string): Promise<LoadedApp> { export async function loadApp(cwd: string, entrypoint?: string): Promise<LoadedApp> {
const appPath = resolve(cwd, 'app.js') const resolvedEntrypoint = entrypoint
const appUrl = pathToFileURL(appPath).href + '?t=' + Date.now() ? resolve(cwd, entrypoint)
: findAppFile(cwd)
if (!resolvedEntrypoint) {
throw new AppLoadError(
`No Fastify app entrypoint found. Searched for: ${APP_CANDIDATES.join(', ')}.\n` +
'Create an app.js, server.js, or similar that exports a Fastify instance.',
'not_found',
)
}
const ext = extname(resolvedEntrypoint)
const isTypeScript = ext === '.ts'
if (isTypeScript && !isTsxAvailable()) {
throw new AppLoadError(
`TypeScript entrypoint ${resolvedEntrypoint} requires a TS loader.\n` +
'Install tsx and run:\n' +
' npm install -D tsx\n' +
' npx tsx ./node_modules/.bin/apophis verify\n\n' +
'Or convert your entrypoint to JavaScript.',
'import_failed',
)
}
const appUrl = pathToFileURL(resolvedEntrypoint).href + '?t=' + Date.now()
let appModule: Record<string, unknown> let appModule: Record<string, unknown>
try { try {
appModule = await import(appUrl) as Record<string, unknown> appModule = await import(appUrl) as Record<string, unknown>
} catch (err) { } catch (err) {
const errMessage = err instanceof Error ? err.message : String(err)
if (isTypeScript) {
throw new AppLoadError( throw new AppLoadError(
`Cannot load app.js: ${err instanceof Error ? err.message : String(err)}`, `Cannot load TypeScript entrypoint ${resolvedEntrypoint}: ${errMessage}\n\n` +
'TypeScript entrypoints require tsx. Install tsx and run:\n' +
' npm install -D tsx\n' +
' npx tsx ./node_modules/.bin/apophis verify\n\n' +
'Or convert your entrypoint to JavaScript.',
'import_failed',
)
}
throw new AppLoadError(
`Cannot load ${resolvedEntrypoint}: ${errMessage}`,
'import_failed', 'import_failed',
) )
} }
// Try default export first let fastify: unknown
if (appModule.default && isFastifyInstance(appModule.default)) { let source: 'default' | 'named' | 'commonjs' = 'default'
return { fastify: appModule.default, source: 'default' }
}
// Try named exports that look like Fastify instances or factory functions if (appModule.default && isFastifyInstance(appModule.default)) {
fastify = appModule.default
source = 'default'
} else {
let found = false
for (const [key, value] of Object.entries(appModule)) { for (const [key, value] of Object.entries(appModule)) {
if (key === 'default') continue if (key === 'default') continue
if (isFastifyInstance(value)) { if (isFastifyInstance(value)) {
return { fastify: value, source: 'named' } fastify = value
source = 'named'
found = true
break
} }
// Try calling factory functions
if (typeof value === 'function' && !isClass(value)) { if (typeof value === 'function' && !isClass(value)) {
try { try {
const result = await value() const result = await value()
if (isFastifyInstance(result)) { if (isFastifyInstance(result)) {
return { fastify: result, source: 'named' } fastify = result
source = 'named'
found = true
break
} }
} catch { } catch {
// Factory function failed, try next // Factory function failed, try next
} }
} }
} }
if (!found && isFastifyInstance(appModule)) {
// If module itself is a Fastify instance (CommonJS) fastify = appModule
if (isFastifyInstance(appModule)) { source = 'commonjs'
return { fastify: appModule, source: 'commonjs' } found = true
} }
if (!found) {
throw new AppLoadError( throw new AppLoadError(
'No Fastify instance found in app.js. Ensure app.js exports a Fastify instance or a factory function.', `No Fastify instance found in ${resolvedEntrypoint}. ` +
'Ensure the file exports a Fastify instance or a factory function.\n\n' +
'Supported patterns:\n' +
' export default app\n' +
' export { app }\n' +
' export const createApp = () => app\n' +
' module.exports = app',
'no_fastify', 'no_fastify',
) )
} }
}
/** await ensureRouteDiscovery(fastify as Record<string, unknown>)
* Check if a value looks like a Fastify instance.
*/ return { fastify, source, entrypoint: resolvedEntrypoint }
function isFastifyInstance(value: unknown): boolean {
return value !== null &&
typeof value === 'object' &&
typeof (value as Record<string, unknown>).ready === 'function'
} }
/** /**
* Check if a function is a class constructor. * Install a lightweight onRoute hook for route discovery.
* This runs before avvio's boot phase, so routes registered
* inside deferred plugins will be captured even if apophis
* is not registered first.
*/ */
async function ensureRouteDiscovery(f: Record<string, unknown>): Promise<void> {
if (typeof f.addHook !== 'function') return
try {
const { installRouteDiscovery } = await import('../../infrastructure/discovery-hook.js')
installRouteDiscovery(f as { addHook: Function })
} catch {
// discovery module not importable, skip auto-capture
}
}
function isFastifyInstance(value: unknown): boolean {
if (value === null || typeof value !== 'object') return false
const obj = value as Record<string, unknown>
return typeof obj.ready === 'function'
}
function isClass(fn: unknown): boolean { function isClass(fn: unknown): boolean {
return typeof fn === 'function' && return typeof fn === 'function' &&
fn.toString().startsWith('class ') fn.toString().startsWith('class ')
} }
/**
* Error type for app loading failures.
*/
export class AppLoadError extends Error { export class AppLoadError extends Error {
constructor( constructor(
message: string, message: string,
public readonly code: 'import_failed' | 'no_fastify', public readonly code: 'import_failed' | 'no_fastify' | 'not_found',
) { ) {
super(message) super(message)
this.name = 'AppLoadError' this.name = 'AppLoadError'
src/cli/core/config-loader.ts
+63 -3
View File
@@ -30,6 +30,7 @@ export interface Config {
environments?: Record<string, EnvironmentPolicy>; environments?: Record<string, EnvironmentPolicy>;
profiles?: Record<string, ProfileDefinition>; profiles?: Record<string, ProfileDefinition>;
presets?: Record<string, PresetDefinition>; presets?: Record<string, PresetDefinition>;
scenarios?: ScenarioConfigDef[];
[key: string]: unknown; [key: string]: unknown;
} }
@@ -53,9 +54,28 @@ export interface PresetDefinition {
routes?: string[]; routes?: string[];
seed?: number; seed?: number;
features?: string[]; features?: string[];
chaos?: boolean;
chaosStrategy?: 'one' | 'all' | 'sample' | 'routes';
chaosSampleSize?: number;
chaosSampleRoutes?: string[];
[key: string]: unknown; [key: string]: unknown;
} }
export interface ScenarioConfigDef {
name: string;
steps: Array<{
name: string;
request: {
method: string;
url: string;
body?: unknown;
headers?: Record<string, string>;
};
expect: string[];
capture?: Record<string, string>;
}>;
}
export interface LoadConfigOptions { export interface LoadConfigOptions {
cwd: string; cwd: string;
configPath?: string; configPath?: string;
@@ -111,6 +131,20 @@ const CONFIG_SCHEMA: Record<string, SchemaField> = {
optional: true, optional: true,
items: { type: 'string' }, items: { type: 'string' },
}, },
metadata: {
type: 'object',
optional: true,
},
scenarios: {
type: 'array',
optional: true,
items: { type: 'object' },
},
chaos: {
type: 'object',
optional: true,
properties: {},
},
}; };
// Schema for EnvironmentPolicy values (inside environments.<name>) // Schema for EnvironmentPolicy values (inside environments.<name>)
@@ -140,6 +174,10 @@ const PROFILE_SCHEMA: Record<string, SchemaField> = {
sampling: { type: 'number', optional: true }, sampling: { type: 'number', optional: true },
blocking: { type: 'boolean', optional: true }, blocking: { type: 'boolean', optional: true },
sinks: { type: 'object', optional: true }, sinks: { type: 'object', optional: true },
chaos: { type: 'object', optional: true },
chaosStrategy: { type: 'string', optional: true, enumValues: ['one', 'all', 'sample', 'routes'] },
chaosSampleSize: { type: 'number', optional: true, min: 1 },
chaosSampleRoutes: { type: 'array', optional: true, items: { type: 'string' } },
}; };
// Schema for PresetDefinition values (inside presets.<name>) // Schema for PresetDefinition values (inside presets.<name>)
@@ -148,6 +186,9 @@ const PRESET_SCHEMA: Record<string, SchemaField> = {
timeout: { type: 'number', optional: true, min: 0 }, timeout: { type: 'number', optional: true, min: 0 },
parallel: { type: 'boolean', optional: true }, parallel: { type: 'boolean', optional: true },
chaos: { type: 'boolean', optional: true }, chaos: { type: 'boolean', optional: true },
chaosStrategy: { type: 'string', optional: true, enumValues: ['one', 'all', 'sample', 'routes'] },
chaosSampleSize: { type: 'number', optional: true, min: 1 },
chaosSampleRoutes: { type: 'array', optional: true, items: { type: 'string' } },
observe: { type: 'boolean', optional: true }, observe: { type: 'boolean', optional: true },
features: { type: 'array', optional: true, items: { type: 'string' } }, features: { type: 'array', optional: true, items: { type: 'string' } },
sampling: { type: 'number', optional: true }, sampling: { type: 'number', optional: true },
@@ -201,7 +242,7 @@ export function loadPackageJsonConfig(cwd: string): { config: Config | null; pat
/** /**
* Load a config file by path. * Load a config file by path.
* Supports .js, .ts (via dynamic import, assumes tsx available), and .json. * Supports .js, .ts (via dynamic import when tsx is available), and .json.
*/ */
export async function loadConfigFile(configPath: string): Promise<Config> { export async function loadConfigFile(configPath: string): Promise<Config> {
if (configPath.endsWith('.json')) { if (configPath.endsWith('.json')) {
@@ -209,6 +250,14 @@ export async function loadConfigFile(configPath: string): Promise<Config> {
return JSON.parse(content) as Config; return JSON.parse(content) as Config;
} }
if (configPath.endsWith('.ts') && !process.execArgv.some(a => a.includes('tsx') || a.includes('ts-node'))) {
throw new Error(
`TypeScript config file ${configPath} requires a TS loader.\n` +
'Convert to apophis.config.js or apophis.config.json, or run through tsx:\n' +
' npx tsx ./node_modules/.bin/apophis verify'
);
}
// For .js and .ts, use dynamic import. // For .js and .ts, use dynamic import.
// tsx handles .ts files in dev environments. // tsx handles .ts files in dev environments.
const fileUrl = pathToFileURL(configPath).href; const fileUrl = pathToFileURL(configPath).href;
@@ -259,6 +308,15 @@ function isInsideDynamicContainer(path: string): boolean {
return path.startsWith('profiles.') || path.startsWith('presets.') || path.startsWith('environments.'); return path.startsWith('profiles.') || path.startsWith('presets.') || path.startsWith('environments.');
} }
/**
* User-managed extension keys that pass through schema validation.
* APOPHIS will never interpret these fields; they are reserved for team metadata.
* Keys starting with 'x-' are always allowed at any level.
*/
function isUserExtensionKey(key: string): boolean {
return key.startsWith('x-')
}
/** /**
* Validate that a value matches the expected type for a schema field. * Validate that a value matches the expected type for a schema field.
* Throws ConfigValidationError on type mismatch. * Throws ConfigValidationError on type mismatch.
@@ -406,13 +464,14 @@ export function validateConfigAgainstSchema(
} }
} }
} else { } else {
if (isUserExtensionKey(key)) continue
// Unknown key inside a profile/preset/environment object // Unknown key inside a profile/preset/environment object
throw new ConfigValidationError( throw new ConfigValidationError(
`Unknown config key at ${currentPath}`, `Unknown config key at ${currentPath}`,
currentPath, currentPath,
key, key,
obj[key], obj[key],
`Valid keys for ${parentContainer} entries: ${Object.keys(childSchema || {}).join(', ')}.`, `Valid keys for ${parentContainer} entries: ${Object.keys(childSchema || {}).join(', ')}. Use x- prefixed keys for team metadata.`,
); );
} }
} }
@@ -420,12 +479,13 @@ export function validateConfigAgainstSchema(
} }
if (!fieldSchema) { if (!fieldSchema) {
if (isUserExtensionKey(key)) continue
throw new ConfigValidationError( throw new ConfigValidationError(
`Unknown config key at ${currentPath}`, `Unknown config key at ${currentPath}`,
currentPath, currentPath,
key, key,
obj[key], obj[key],
`Valid top-level keys: ${Object.keys(CONFIG_SCHEMA).join(', ')}.`, `Valid top-level keys: ${Object.keys(CONFIG_SCHEMA).join(', ')}. Use the metadata field or x- prefixed keys for team data.`,
); );
} }
src/cli/core/index.ts
+5 -5
View File
@@ -2,7 +2,7 @@ import { cac } from 'cac';
import pc from 'picocolors'; import pc from 'picocolors';
import { createContext, type CliContext } from './context.js'; import { createContext, type CliContext } from './context.js';
const CLI_VERSION = '2.0.0'; const CLI_VERSION = '2.7.0';
const HELP_HEADER = ` const HELP_HEADER = `
${pc.bold('apophis')} — Contract-driven API testing for Fastify ${pc.bold('apophis')} — Contract-driven API testing for Fastify
@@ -23,7 +23,7 @@ const HELP_HEADER = `
--config <path> Config file path --config <path> Config file path
--profile <name> Profile name from config --profile <name> Profile name from config
--cwd <path> Working directory override --cwd <path> Working directory override
--format <mode> Output format: human | json | ndjson (default: human) --format <mode> Output format: human | json | ndjson | json-summary | ndjson-summary (default: human)
--color <mode> Color mode: auto | always | never (default: auto) --color <mode> Color mode: auto | always | never (default: auto)
--quiet Suppress non-error output --quiet Suppress non-error output
--verbose Enable verbose logging --verbose Enable verbose logging
@@ -223,7 +223,7 @@ export async function main(argv: string[] = process.argv.slice(2)): Promise<numb
cli.option('--config <path>', 'Config file path'); cli.option('--config <path>', 'Config file path');
cli.option('--profile <name>', 'Profile name from config'); cli.option('--profile <name>', 'Profile name from config');
cli.option('--cwd <path>', 'Working directory override'); cli.option('--cwd <path>', 'Working directory override');
cli.option('--format <mode>', 'Output format: human | json | ndjson', { default: 'human' }); cli.option('--format <mode>', 'Output format: human | json | ndjson | json-summary | ndjson-summary', { default: 'human' });
cli.option('--color <mode>', 'Color mode: auto | always | never', { default: 'auto' }); cli.option('--color <mode>', 'Color mode: auto | always | never', { default: 'auto' });
cli.option('--quiet', 'Suppress non-error output'); cli.option('--quiet', 'Suppress non-error output');
cli.option('--verbose', 'Enable verbose logging'); cli.option('--verbose', 'Enable verbose logging');
@@ -372,11 +372,11 @@ export async function main(argv: string[] = process.argv.slice(2)): Promise<numb
]); ]);
const commandSpecificFlags: Record<string, Set<string>> = { const commandSpecificFlags: Record<string, Set<string>> = {
init: new Set(['--preset', '--force', '--noninteractive']), init: new Set(['--preset', '-p', '--force', '-f', '--noninteractive']),
verify: new Set(['--profile', '--routes', '--seed', '--changed', '--workspace']), verify: new Set(['--profile', '--routes', '--seed', '--changed', '--workspace']),
observe: new Set(['--profile', '--check-config', '--workspace']), observe: new Set(['--profile', '--check-config', '--workspace']),
qualify: new Set(['--profile', '--seed', '--workspace']), qualify: new Set(['--profile', '--seed', '--workspace']),
replay: new Set(['--artifact']), replay: new Set(['--artifact', '--route']),
doctor: new Set(['--mode', '--strict', '--workspace']), doctor: new Set(['--mode', '--strict', '--workspace']),
migrate: new Set(['--check', '--dry-run', '--write']), migrate: new Set(['--check', '--dry-run', '--write']),
}; };
src/cli/core/types.ts
+5
View File
@@ -299,6 +299,11 @@ export interface Artifact {
failed: number; failed: number;
}; };
executionSummary?: ExecutionSummary; executionSummary?: ExecutionSummary;
coverageBreakdown?: {
scenario: { routesCovered: string[]; stepsTotal: number; stepsPassed: number }
stateful: { routesCovered: string[]; testsTotal: number; testsPassed: number }
chaos: { routesPlanned: string[]; routesExecuted: string[]; runsTotal: number; runsPassed: number }
}
executedRoutes?: string[]; executedRoutes?: string[];
skippedRoutes?: RouteExecutionInfo[]; skippedRoutes?: RouteExecutionInfo[];
stepTraces?: StepTrace[]; stepTraces?: StepTrace[];
src/cli/renderers/human.ts
+2 -26
View File
@@ -21,7 +21,7 @@ import { shouldUseColor, getColors, truncate, indent, formatDuration } from './s
// Types // Types
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
export interface HumanRendererOptions { interface HumanRendererOptions {
ctx: OutputContext; ctx: OutputContext;
profile?: string; profile?: string;
seed?: number; seed?: number;
@@ -180,34 +180,10 @@ function generateNextSteps(failure: FailureRecord): string {
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Progress and summary rendering
// ---------------------------------------------------------------------------
/**
* Render a simple ASCII progress bar.
*/
function renderProgressBar(
current: number,
total: number,
width: number,
ctx: OutputContext,
): string {
const c = getColorizer(ctx);
if (total === 0) return c.dim('[' + ' '.repeat(width) + ']');
const filled = Math.round((current / total) * width);
const empty = width - filled;
const filledChar = '█';
const emptyChar = '░';
return '[' + c.green(filledChar.repeat(filled)) + c.dim(emptyChar.repeat(empty)) + ']';
}
/** /**
* Render summary for verify/observe/qualify results. * Render summary for verify/observe/qualify results.
*/ */
export function renderSummary( function renderSummary(
artifact: Artifact, artifact: Artifact,
ctx: OutputContext, ctx: OutputContext,
): string { ): string {
src/cli/renderers/json.ts
+1 -19
View File
@@ -14,7 +14,7 @@ import type { Artifact, CommandResult } from '../core/types.js';
// Types // Types
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
export interface JsonRendererOptions { interface JsonRendererOptions {
indent?: number; indent?: number;
} }
@@ -141,24 +141,6 @@ export function renderJsonArtifact(
* If an artifact is present, it is rendered. * If an artifact is present, it is rendered.
* Otherwise, a minimal JSON with the message and exit code is returned. * Otherwise, a minimal JSON with the message and exit code is returned.
*/ */
export function renderJsonResult(
result: CommandResult,
options: JsonRendererOptions = {},
): string {
if (result.artifact) {
return renderJsonArtifact(result.artifact, options);
}
// Minimal JSON for results without artifacts
const minimal = {
exitCode: result.exitCode,
message: result.message,
warnings: result.warnings,
};
return JSON.stringify(minimal, null, options.indent ?? 2);
}
/** /**
* Render a concise summary artifact for CI/machine parsers. * Render a concise summary artifact for CI/machine parsers.
* Omits stepTraces, cleanupOutcomes, and profileGates to reduce noise. * Omits stepTraces, cleanupOutcomes, and profileGates to reduce noise.
src/cli/renderers/ndjson.ts
+2 -23
View File
@@ -14,7 +14,7 @@ import type { Artifact, FailureRecord, NdjsonEvent } from '../core/types.js';
// Types // Types
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
export interface NdjsonRendererOptions { interface NdjsonRendererOptions {
/** Output stream to write to (defaults to process.stdout) */ /** Output stream to write to (defaults to process.stdout) */
output?: NodeJS.WriteStream; output?: NodeJS.WriteStream;
} }
@@ -118,7 +118,7 @@ export function renderNdjsonEvent(event: NdjsonEvent): string {
* Write an NDJSON event to the output stream. * Write an NDJSON event to the output stream.
* Flushes after each write. * Flushes after each write.
*/ */
export function writeNdjsonEvent( function writeNdjsonEvent(
event: NdjsonEvent, event: NdjsonEvent,
options: NdjsonRendererOptions = {}, options: NdjsonRendererOptions = {},
): void { ): void {
@@ -216,25 +216,4 @@ export function renderNdjsonSummaryArtifact(
writeNdjsonEvent(createRunCompletedEvent(artifact.summary), options); writeNdjsonEvent(createRunCompletedEvent(artifact.summary), options);
} }
/**
* Create concise NDJSON events for an artifact without writing.
* Useful for testing summary mode.
*/
export function createNdjsonSummaryEvents(artifact: Artifact): NdjsonEvent[] {
const events: NdjsonEvent[] = [];
events.push(createRunStartedEvent(artifact.command, artifact.seed));
events.push({
type: 'run.summary',
summary: artifact.summary,
executionSummary: artifact.executionSummary,
profileGates: artifact.profileGates,
deterministicParams: artifact.deterministicParams,
timestamp: getTimestamp(),
} as unknown as NdjsonEvent);
events.push(createRunCompletedEvent(artifact.summary));
return events;
}
src/cli/renderers/shared.ts
+1 -71
View File
@@ -15,7 +15,7 @@ import pc from 'picocolors';
// Types // Types
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
export interface TruncationOptions { interface TruncationOptions {
maxLength?: number; maxLength?: number;
suffix?: string; suffix?: string;
} }
@@ -77,15 +77,6 @@ export function truncate(str: string, options: TruncationOptions = {}): string {
return str.slice(0, truncatedLength) + suffix; return str.slice(0, truncatedLength) + suffix;
} }
/**
* Truncate an object for terminal display.
* Converts to JSON and truncates.
*/
export function truncateObject(obj: unknown, options: TruncationOptions = {}): string {
const str = typeof obj === 'string' ? obj : JSON.stringify(obj, null, 2);
return truncate(str, options);
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Indentation // Indentation
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -113,13 +104,6 @@ export function formatDuration(ms: number): string {
return `${(ms / 1000).toFixed(2)}s`; return `${(ms / 1000).toFixed(2)}s`;
} }
/**
* Format a timestamp as ISO string.
*/
export function formatTimestamp(date?: Date): string {
return (date || new Date()).toISOString();
}
/** /**
* Strip ANSI escape codes from a string. * Strip ANSI escape codes from a string.
*/ */
@@ -136,58 +120,4 @@ export function hasAnsi(str: string): boolean {
return /\u001b\[[0-9;]*m/.test(str); return /\u001b\[[0-9;]*m/.test(str);
} }
// ---------------------------------------------------------------------------
// TTY/CI output helpers
// ---------------------------------------------------------------------------
/**
* Determine if spinners should be shown.
* Never show spinners in CI or non-TTY environments.
*/
export function shouldShowSpinner(ctx: OutputContext): boolean {
return ctx.isTTY && !ctx.isCI;
}
/**
* Write to stdout with optional flushing.
* In non-TTY mode, always flush.
*/
export function writeStdout(str: string): void {
process.stdout.write(str);
}
/**
* Write line to stdout.
*/
export function writeLine(str: string = ''): void {
process.stdout.write(str + '\n');
}
// ---------------------------------------------------------------------------
// Progress helpers
// ---------------------------------------------------------------------------
/**
* Format a progress indicator (no spinner, just text).
* Safe for CI/non-TTY.
*/
export function formatProgress(current: number, total: number, label?: string): string {
const pct = total > 0 ? Math.round((current / total) * 100) : 0;
const prefix = label ? `${label} ` : '';
return `${prefix}[${current}/${total}] ${pct}%`;
}
// ---------------------------------------------------------------------------
// Summary formatting
// ---------------------------------------------------------------------------
/**
* Format a summary block for human output.
*/
export function formatSummary(total: number, passed: number, failed: number): string {
const lines: string[] = [];
lines.push(`Total: ${total}`);
lines.push(`Passed: ${passed}`);
lines.push(`Failed: ${failed}`);
return lines.join('\n');
}
src/domain/contract.ts
+15 -2
View File
@@ -8,6 +8,18 @@ const EMPTY_INVARIANTS: ValidatedFormula[] = []
// Two-level cache: WeakMap<schema, Map<"METHOD path", RouteContract>> // Two-level cache: WeakMap<schema, Map<"METHOD path", RouteContract>>
// Preserves automatic GC of schema objects while correctly caching per-route contracts // Preserves automatic GC of schema objects while correctly caching per-route contracts
const contractCache = new WeakMap<Record<string, unknown>, Map<string, RouteContract>>() const contractCache = new WeakMap<Record<string, unknown>, Map<string, RouteContract>>()
const getFirstSuccessSchema = (responseSchema: Record<string, Record<string, unknown>>): Record<string, unknown> => {
const keys = Object.keys(responseSchema).sort((a, b) => Number(a) - Number(b))
for (const key of keys) {
const status = Number(key)
if (status >= 200 && status < 300) {
return responseSchema[key] ?? {}
}
}
return Object.values(responseSchema)[0] ?? {}
}
export const extractContract = ( export const extractContract = (
path: string, path: string,
method: string, method: string,
@@ -32,8 +44,9 @@ export const extractContract = (
// APOPHIS annotations may live on the top-level schema OR nested inside // APOPHIS annotations may live on the top-level schema OR nested inside
// response.statusCode (e.g. schema.response[200]['x-ensures']). // response.statusCode (e.g. schema.response[200]['x-ensures']).
// We merge both levels so contracts are never silently dropped. // We merge both levels so contracts are never silently dropped.
const responseSchema = (s.response ?? {}) as Record<string, Record<string, unknown>> const rawResponse = s.response
const firstStatus = Object.values(responseSchema)[0] ?? {} const responseSchema = (typeof rawResponse === 'object' && rawResponse !== null ? rawResponse : {}) as Record<string, Record<string, unknown>>
const firstStatus = getFirstSuccessSchema(responseSchema)
const topRequires = s['x-requires'] const topRequires = s['x-requires']
const nestedRequires = firstStatus['x-requires'] const nestedRequires = firstStatus['x-requires']
const requires = Array.isArray(topRequires) && topRequires.length > 0 const requires = Array.isArray(topRequires) && topRequires.length > 0
src/domain/discovery.ts
+125 -40
View File
@@ -14,6 +14,14 @@ interface CapturedRoute {
schema?: Record<string, unknown> schema?: Record<string, unknown>
prefix?: string prefix?: string
} }
export interface DiscoveryResult {
routes: RouteContract[]
source: 'captured' | 'legacy-routes-array' | 'print-routes' | 'none'
hasSchemaMetadata: boolean
warnings: string[]
}
// WeakMap to store captured routes per Fastify instance (no memory leaks) // WeakMap to store captured routes per Fastify instance (no memory leaks)
const capturedRoutes = new WeakMap<object, CapturedRoute[]>() const capturedRoutes = new WeakMap<object, CapturedRoute[]>()
/** /**
@@ -32,36 +40,127 @@ export const captureRoute = (
/** /**
* Fallback route discovery for Fastify 5 when routes were registered before * Fallback route discovery for Fastify 5 when routes were registered before
* the APOPHIS plugin (e.g., external apps loaded by CLI). * the APOPHIS plugin (e.g., external apps loaded by CLI).
* Uses hasRoute to test known route patterns. * Parses printRoutes() output to discover route paths and methods.
*
* printRoutes() output looks like:
* └── /
* ├── users (GET, HEAD)
* └── items/
* └── :id (POST)
*/
function parsePrintRoutesOutput(output: string): Array<{ method: string; path: string }> {
const routes: Array<{ method: string; path: string }> = []
const lines = output.split('\n')
const depthPrefixes: string[] = []
for (const line of lines) {
const trimmed = line.trimStart()
if (!trimmed) continue
const indent = line.length - line.trimStart().length
const depth = Math.floor(indent / 2)
depthPrefixes.length = depth
const branchMatch = trimmed.match(/^(├──|└──)\s+(.+)$/)
if (!branchMatch) continue
let segment = branchMatch[2]!
const isIntermediate = segment.endsWith('/')
if (isIntermediate) {
segment = segment.slice(0, -1)
}
const methodMatch = segment.match(/^(.+?)\s+\(([^)]+)\)$/)
if (methodMatch) {
const rawPath = methodMatch[1]!
const methodList = methodMatch[2]!.split(',').map(m => m.trim())
const fullPath = '/' + [...depthPrefixes, rawPath].filter(Boolean).join('/')
for (const method of methodList) {
if (method === 'HEAD') continue
routes.push({ method, path: fullPath })
}
} else {
depthPrefixes.push(segment)
}
}
return routes
}
/**
* Fallback route discovery for Fastify 5 when onRoute hook wasn't installed
* before route registration.
*/ */
function discoverRoutesFallback( function discoverRoutesFallback(
instance: { hasRoute?: (opts: { method: string; url: string }) => boolean } instance: {
): RouteContract[] { hasRoute?: (opts: { method: string; url: string }) => boolean
if (typeof instance.hasRoute !== 'function') { printRoutes?: () => string
return [] }
): DiscoveryResult {
let routes: RouteContract[] = []
let source: DiscoveryResult['source'] = 'none'
let warnings: string[] = []
if (typeof instance.printRoutes === 'function') {
const output = instance.printRoutes()
const parsed = parsePrintRoutesOutput(output)
if (parsed.length > 0) {
routes = parsed.map(r => extractContract(r.path, r.method, undefined))
source = 'print-routes'
warnings.push(
'Route schemas were not available during discovery. ' +
'Behavioral contracts (x-ensures, x-requires, x-outbound, x-variants, x-timeout) ' +
'will not be detected. Register APOPHIS or install route discovery before defining routes, ' +
'or use createFastify() from @apophis/fastify.'
)
}
} }
// Common HTTP methods to test return { routes, source, hasSchemaMetadata: false, warnings }
const methods = ['GET', 'POST', 'PUT', 'DELETE', 'PATCH', 'HEAD', 'OPTIONS']
// We can't enumerate all possible routes, but we can check if the instance
// has any routes at all by testing a few common patterns
// This is a best-effort fallback
const routes: RouteContract[] = []
// Try to extract routes from the instance's internal state
// Fastify stores routes in find-my-way router, but it's not directly accessible
// We'll use a heuristic: check if the instance responds to common route methods
// Check if instance has any routes by looking at prototype methods
const hasRouting = typeof (instance as any).routing === 'function'
if (!hasRouting) {
return []
} }
// Since we can't enumerate routes in Fastify 5 without the onRoute hook, export interface DiscoverRouteDetailsInput {
// we return empty and let the caller handle the "no routes" case routes?: Array<{ method: string; url: string; schema?: Record<string, unknown> }>
return [] hasRoute?: (opts: { method: string; url: string }) => boolean
printRoutes?: () => string
}
/**
* Discover routes from a Fastify instance with full metadata about discovery quality.
*
* Use this function when you need to know whether routes were discovered
* with full schema metadata (captured / legacy-routes-array) or as
* schema-less fallback paths (print-routes / none).
*/
export const discoverRouteDetails = (instance: DiscoverRouteDetailsInput): DiscoveryResult => {
const captured = capturedRoutes.get(instance)
if (captured && captured.length > 0) {
return {
routes: captured.map((route) =>
extractContract(route.url, route.method, route.schema)
),
source: 'captured',
hasSchemaMetadata: true,
warnings: [],
}
}
if (Array.isArray(instance.routes) && instance.routes.length > 0) {
return {
routes: instance.routes.map((route) =>
extractContract(route.url, route.method, route.schema)
),
source: 'legacy-routes-array',
hasSchemaMetadata: true,
warnings: [],
}
}
return discoverRoutesFallback(instance)
} }
/** /**
@@ -70,20 +169,6 @@ function discoverRoutesFallback(
* First checks captured routes (from onRoute hook), then falls back to * First checks captured routes (from onRoute hook), then falls back to
* the legacy `routes` array for Fastify 4 compatibility. * the legacy `routes` array for Fastify 4 compatibility.
*/ */
export const discoverRoutes = (instance: { routes?: Array<{ method: string; url: string; schema?: Record<string, unknown> }>; hasRoute?: (opts: { method: string; url: string }) => boolean }): RouteContract[] => { export const discoverRoutes = (instance: DiscoverRouteDetailsInput): RouteContract[] => {
// Fastify 5: routes captured via onRoute hook return discoverRouteDetails(instance).routes
const captured = capturedRoutes.get(instance)
if (captured && captured.length > 0) {
return captured.map((route) =>
extractContract(route.url, route.method, route.schema)
)
}
// Fastify 4 fallback
if (Array.isArray(instance.routes) && instance.routes.length > 0) {
return instance.routes.map((route) =>
extractContract(route.url, route.method, route.schema)
)
}
// Fastify 5 fallback: routes registered before plugin
return discoverRoutesFallback(instance)
} }
src/extension/timeout.ts
+1 -1
View File
@@ -4,7 +4,7 @@
* Wraps async operations with a timeout to prevent indefinite hangs. * Wraps async operations with a timeout to prevent indefinite hangs.
*/ */
export class HookTimeoutError extends Error { class HookTimeoutError extends Error {
constructor(extensionName: string, hookName: string, timeoutMs: number) { constructor(extensionName: string, hookName: string, timeoutMs: number) {
super( super(
`Extension '${extensionName}' ${hookName} timed out after ${timeoutMs}ms. ` + `Extension '${extensionName}' ${hookName} timed out after ${timeoutMs}ms. ` +
src/extension/types.ts
+2
View File
@@ -112,6 +112,8 @@ export type PredicateResolver = (context: PredicateContext) => PredicateResult
export interface ApophisExtension { export interface ApophisExtension {
/** Unique extension name (used for logging and state isolation) */ /** Unique extension name (used for logging and state isolation) */
readonly name: string readonly name: string
/** Whether this extension is safe for production observe mode. Default: false (conservative). */
readonly productionSafe?: boolean
/** /**
* APOSTL operation headers this extension adds. * APOSTL operation headers this extension adds.
* Used by the parser to validate extension operations in formulas. * Used by the parser to validate extension operations in formulas.
src/extensions/http-signature.ts
+2 -1
View File
@@ -6,7 +6,7 @@
* *
* Example: * Example:
* ```typescript * ```typescript
* import { httpSignatureExtension } from 'apophis-fastify/extensions/http-signature' * import { httpSignatureExtension } from '@apophis/fastify/extensions/http-signature'
* *
* await fastify.register(apophis, { * await fastify.register(apophis, {
* extensions: [httpSignatureExtension()] * extensions: [httpSignatureExtension()]
@@ -167,6 +167,7 @@ function extractSignature(ctx: PredicateContext): string | null {
export function httpSignatureExtension(config: HttpSignatureExtensionConfig = {}): ApophisExtension { export function httpSignatureExtension(config: HttpSignatureExtensionConfig = {}): ApophisExtension {
return { return {
name: 'httpSignature', name: 'httpSignature',
productionSafe: true,
headers: ['signature_input', 'signature', 'signature_valid', 'signature_covers'], headers: ['signature_input', 'signature', 'signature_valid', 'signature_covers'],
src/extensions/index.ts
+1 -1
View File
@@ -14,7 +14,7 @@
* tokenHashExtension, * tokenHashExtension,
* httpSignatureExtension, * httpSignatureExtension,
* requestContextExtension, * requestContextExtension,
* } from 'apophis-fastify/extensions' * } from '@apophis/fastify/extensions'
* ``` * ```
*/ */
src/extensions/jwt.ts
+2 -1
View File
@@ -7,7 +7,7 @@
* *
* Example: * Example:
* ```typescript * ```typescript
* import { jwtExtension } from 'apophis-fastify/extensions/jwt' * import { jwtExtension } from '@apophis/fastify/extensions/jwt'
* *
* await fastify.register(apophis, { * await fastify.register(apophis, {
* extensions: [ * extensions: [
@@ -286,6 +286,7 @@ function resolveFormat(ctx: PredicateContext, config: JwtExtensionConfig): unkno
export function jwtExtension(config: JwtExtensionConfig = {}): ApophisExtension { export function jwtExtension(config: JwtExtensionConfig = {}): ApophisExtension {
return { return {
name: 'jwt', name: 'jwt',
productionSafe: true,
headers: ['jwt_claims', 'jwt_header', 'jwt_valid', 'jwt_format'], headers: ['jwt_claims', 'jwt_header', 'jwt_valid', 'jwt_format'],
src/extensions/request-context.ts
+2 -1
View File
@@ -6,7 +6,7 @@
* *
* Example: * Example:
* ```typescript * ```typescript
* import { requestContextExtension } from 'apophis-fastify/extensions/request-context' * import { requestContextExtension } from '@apophis/fastify/extensions/request-context'
* *
* await fastify.register(apophis, { * await fastify.register(apophis, {
* extensions: [requestContextExtension()] * extensions: [requestContextExtension()]
@@ -135,6 +135,7 @@ function hashBody(body: unknown, algorithm: string): string | null {
export function requestContextExtension(config: RequestContextExtensionConfig = {}): ApophisExtension { export function requestContextExtension(config: RequestContextExtensionConfig = {}): ApophisExtension {
return { return {
name: 'requestContext', name: 'requestContext',
productionSafe: true,
headers: ['request_url', 'request_tls', 'request_body_hash'], headers: ['request_url', 'request_tls', 'request_body_hash'],
src/extensions/serializers/index.ts
+3
View File
@@ -0,0 +1,3 @@
export { createSerializerExtension } from './extension.js'
export { createSerializerRegistry, type Serializer, type SerializerRegistry } from './types.js'
export { transformRequest, transformResponse } from './transformer.js'
src/extensions/spiffe.ts
+2 -1
View File
@@ -5,7 +5,7 @@
* *
* Example: * Example:
* ```typescript * ```typescript
* import { spiffeExtension } from 'apophis-fastify/extensions/spiffe' * import { spiffeExtension } from '@apophis/fastify/extensions/spiffe'
* *
* await fastify.register(apophis, { * await fastify.register(apophis, {
* extensions: [spiffeExtension()] * extensions: [spiffeExtension()]
@@ -150,6 +150,7 @@ function extractSpiffeId(ctx: PredicateContext, config: SpiffeExtensionConfig):
export function spiffeExtension(config: SpiffeExtensionConfig = {}): ApophisExtension { export function spiffeExtension(config: SpiffeExtensionConfig = {}): ApophisExtension {
return { return {
name: 'spiffe', name: 'spiffe',
productionSafe: true,
headers: ['spiffe_parse', 'spiffe_validate', 'spiffe_id', 'spiffe_trust_domain'], headers: ['spiffe_parse', 'spiffe_validate', 'spiffe_id', 'spiffe_trust_domain'],
src/extensions/sse/index.ts
+3
View File
@@ -0,0 +1,3 @@
export { sseExtension } from './extension.js'
export { parseSSEEvents } from './transformer.js'
export type { SSEEvent, SSEConfig } from './types.js'
src/extensions/stateful.ts
+1 -1
View File
@@ -8,7 +8,7 @@
* *
* Example: * Example:
* ```typescript * ```typescript
* import { statefulExtension } from 'apophis-fastify/extensions/stateful' * import { statefulExtension } from '@apophis/fastify/extensions/stateful'
* *
* await fastify.register(apophis, { * await fastify.register(apophis, {
* extensions: [statefulExtension()] * extensions: [statefulExtension()]
src/extensions/time.ts
+1 -1
View File
@@ -7,7 +7,7 @@
* *
* Example: * Example:
* ```typescript * ```typescript
* import { timeExtension } from 'apophis-fastify/extensions/time' * import { timeExtension } from '@apophis/fastify/extensions/time'
* *
* await fastify.register(apophis, { * await fastify.register(apophis, {
* extensions: [timeExtension()] * extensions: [timeExtension()]
src/extensions/token-hash.ts
+2 -1
View File
@@ -6,7 +6,7 @@
* *
* Example: * Example:
* ```typescript * ```typescript
* import { tokenHashExtension } from 'apophis-fastify/extensions/token-hash' * import { tokenHashExtension } from '@apophis/fastify/extensions/token-hash'
* *
* await fastify.register(apophis, { * await fastify.register(apophis, {
* extensions: [tokenHashExtension()] * extensions: [tokenHashExtension()]
@@ -110,6 +110,7 @@ export function tokenHashExtension(config: TokenHashExtensionConfig = {}): Apoph
return { return {
name: 'tokenHash', name: 'tokenHash',
productionSafe: true,
headers: ['ath_valid', 'tth_valid', 'oth_valid', 'token_hash'], headers: ['ath_valid', 'tth_valid', 'oth_valid', 'token_hash'],
src/extensions/websocket/index.ts
+3
View File
@@ -0,0 +1,3 @@
export { websocketExtension } from './extension.js'
export { runWebSocketTests, type WebSocketTestConfig, type WebSocketTestResult } from './runner.js'
export type { WebSocketMessage, WebSocketConnection, WebSocketContract } from './types.js'
src/extensions/x509.ts
+2 -1
View File
@@ -6,7 +6,7 @@
* *
* Example: * Example:
* ```typescript * ```typescript
* import { x509Extension } from 'apophis-fastify/extensions/x509' * import { x509Extension } from '@apophis/fastify/extensions/x509'
* *
* await fastify.register(apophis, { * await fastify.register(apophis, {
* extensions: [x509Extension()] * extensions: [x509Extension()]
@@ -157,6 +157,7 @@ function extractCertificate(ctx: PredicateContext, config: X509ExtensionConfig):
export function x509Extension(config: X509ExtensionConfig = {}): ApophisExtension { export function x509Extension(config: X509ExtensionConfig = {}): ApophisExtension {
return { return {
name: 'x509', name: 'x509',
productionSafe: true,
headers: [ headers: [
'x509_uri_sans', 'x509_uri_sans',
src/fastify-factory.ts
+25
View File
@@ -0,0 +1,25 @@
import type { FastifyInstance, FastifyServerOptions } from 'fastify'
import Fastify from 'fastify'
import { installRouteDiscovery } from './infrastructure/discovery-hook.js'
export interface CreateFastifyOptions extends FastifyServerOptions {
apophis?: {
runtime?: 'off' | 'warn' | 'error'
discoveryOnly?: boolean
}
}
export async function createFastify(
opts?: CreateFastifyOptions,
): Promise<FastifyInstance> {
const { apophis: apophisOpts, ...fastifyOpts } = opts ?? {}
const fastify = Fastify(fastifyOpts)
installRouteDiscovery(fastify as unknown as { addHook: Function })
if (apophisOpts?.discoveryOnly !== true && apophisOpts?.runtime && apophisOpts.runtime !== 'off') {
const { apophisPlugin } = await import('./plugin/index.js')
await fastify.register(apophisPlugin, { runtime: apophisOpts.runtime })
}
return fastify
}
src/formula/evaluator.ts
-12
View File
@@ -267,18 +267,6 @@ function evaluateComparison(op: string, left: unknown, right: unknown): boolean
throw new Error(`Unknown comparator: ${op}`) throw new Error(`Unknown comparator: ${op}`)
} }
} }
function evaluateBoolean(op: string, left: boolean, right: boolean): boolean {
switch (op) {
case '&&':
return left && right
case '||':
return left || right
case '=>':
return !left || right
default:
throw new Error(`Unknown boolean operator: ${op}`)
}
}
function evaluateQuantified( function evaluateQuantified(
quantifier: 'for' | 'exists', quantifier: 'for' | 'exists',
collection: unknown, collection: unknown,
src/index.ts
+34 -3
View File
@@ -1,14 +1,15 @@
/** /**
* apophis-fastify - Package entry point. * @apophis/fastify - Package entry point.
* Exports the plugin as a Fastify plugin with proper metadata. * Exports the plugin as a Fastify plugin with proper metadata.
*/ */
import './augmentations.js'
import fp from 'fastify-plugin' import fp from 'fastify-plugin'
import { apophisPlugin } from './plugin/index.js' import { apophisPlugin } from './plugin/index.js'
export default fp(apophisPlugin, { export default fp(apophisPlugin, {
name: 'apophis-fastify', name: '@apophis/fastify',
dependencies: ['@fastify/swagger'],
}) })
export * from './types.js' export * from './types.js'
@@ -43,3 +44,33 @@ export {
type MutationReport, type MutationReport,
type MutationConfig, type MutationConfig,
} from './quality/mutation.js' } from './quality/mutation.js'
export type {
ApophisConfig,
ProfileDefinition,
PresetDefinition,
EnvironmentPolicy,
} from './cli/core/types.js'
// Extension factories
export {
createAuthExtension,
createHeaderExtension,
createConditionalHeaderExtension,
createPredicateExtension,
createLoggingExtension,
createMetricsExtension,
} from './extension/factories.js'
// Outbound mock runtime
export {
createOutboundMockRuntime,
type OutboundMockOptions,
type OutboundMockRuntime,
} from './infrastructure/outbound-mock-runtime.js'
// Fastify factory with built-in route discovery
export {
createFastify,
type CreateFastifyOptions,
} from './fastify-factory.js'
src/infrastructure/discovery-hook.ts
+26
View File
@@ -0,0 +1,26 @@
import { captureRoute } from '../domain/discovery.js'
const installed = new WeakSet<object>()
export function installRouteDiscovery(fastify: { addHook?: Function }): void {
if (typeof fastify.addHook !== 'function') return
const instance = fastify as object
if (installed.has(instance)) return
installed.add(instance)
fastify.addHook('onRoute', (routeOptions: Record<string, unknown>) => {
const methods = Array.isArray(routeOptions.method)
? routeOptions.method as string[]
: [routeOptions.method as string]
const schema = routeOptions.schema as Record<string, unknown> | undefined
const prefix = (routeOptions as Record<string, unknown>).prefix as string | undefined
const url = prefix && !(routeOptions.url as string).startsWith(prefix)
? `${prefix}${routeOptions.url}`
: routeOptions.url as string
for (const method of methods) {
if (!method) continue
captureRoute(instance, { method, url, schema, prefix })
}
})
}
src/infrastructure/hook-validator.ts
+136 -36
View File
@@ -5,7 +5,7 @@
*/ */
import type { FastifyInstance, FastifyRequest, FastifyReply } from 'fastify' import type { FastifyInstance, FastifyRequest, FastifyReply } from 'fastify'
import type { FormulaNode } from '../domain/formula.js' import type { FormulaNode } from '../domain/formula.js'
import type { EvalContext, RouteContract } from '../types.js' import type { EvalContext, RouteContract, ObserveSink, ObserveEvent } from '../types.js'
import { parse } from '../formula/parser.js' import { parse } from '../formula/parser.js'
import { evaluateAsync } from '../formula/evaluator.js' import { evaluateAsync } from '../formula/evaluator.js'
import { APOPHIS_INTERNAL_OPERATION_HEADER, createOperationResolver, prefetchPreviousOperations } from '../formula/runtime.js' import { APOPHIS_INTERNAL_OPERATION_HEADER, createOperationResolver, prefetchPreviousOperations } from '../formula/runtime.js'
@@ -13,6 +13,10 @@ import { APOPHIS_INTERNAL_OPERATION_HEADER, createOperationResolver, prefetchPre
interface HookOptions { interface HookOptions {
validateRuntime: boolean validateRuntime: boolean
runtimeLevel?: 'warn' | 'error' runtimeLevel?: 'warn' | 'error'
observers?: {
sinks: ObserveSink[]
sampling: number
}
} }
interface RequestWithCookies extends FastifyRequest { interface RequestWithCookies extends FastifyRequest {
cookies?: Record<string, string> cookies?: Record<string, string>
@@ -42,10 +46,16 @@ const getRouteContract = (request: FastifyRequest): RouteContract | undefined =>
(request.routeOptions?.config as RouteConfig | undefined)?.apophisContract (request.routeOptions?.config as RouteConfig | undefined)?.apophisContract
const isInternalOperationRequest = (request: FastifyRequest): boolean => const isInternalOperationRequest = (request: FastifyRequest): boolean =>
request.headers[APOPHIS_INTERNAL_OPERATION_HEADER] === '1' request.headers[APOPHIS_INTERNAL_OPERATION_HEADER] === '1'
const normalizeHeaders = (headers: Record<string, unknown>): Record<string, string> =>
Object.fromEntries(
Object.entries(headers).map(([k, v]) => [k, Array.isArray(v) ? v.join(', ') : String(v)])
)
const buildPreContext = (request: FastifyRequest): EvalContext => ({ const buildPreContext = (request: FastifyRequest): EvalContext => ({
request: { request: {
body: request.body, body: request.body,
headers: request.headers as Record<string, string>, headers: normalizeHeaders(request.headers),
query: request.query as Record<string, unknown>, query: request.query as Record<string, unknown>,
params: request.params as Record<string, string>, params: request.params as Record<string, string>,
cookies: getCookies(request), cookies: getCookies(request),
@@ -59,74 +69,168 @@ const buildPreContext = (request: FastifyRequest): EvalContext => ({
const buildPostContext = (request: FastifyRequest, reply: FastifyReply): EvalContext => ({ const buildPostContext = (request: FastifyRequest, reply: FastifyReply): EvalContext => ({
request: { request: {
body: request.body, body: request.body,
headers: request.headers as Record<string, string>, headers: normalizeHeaders(request.headers),
query: request.query as Record<string, unknown>, query: request.query as Record<string, unknown>,
params: request.params as Record<string, string>, params: request.params as Record<string, string>,
cookies: getCookies(request), cookies: getCookies(request),
}, },
response: { response: {
body: reply[kApophisPayload] ?? null, body: reply[kApophisPayload] ?? null,
headers: reply.getHeaders() as Record<string, string>, headers: normalizeHeaders(reply.getHeaders()),
statusCode: reply.statusCode, statusCode: reply.statusCode,
}, },
}) })
const routeContractStore = new Map<string, RouteContract>() const instanceStores = new WeakMap<object, {
const routeFormulaStore = new Map<string, { requires: ParsedRuntimeFormula[]; ensures: ParsedRuntimeFormula[] }>() contracts: Map<string, RouteContract>
// Fast-path set: routes that actually have contracts to validate formulas: Map<string, { requires: ParsedRuntimeFormula[]; ensures: ParsedRuntimeFormula[] }>
const routesWithContracts = new Set<string>() withContracts: Set<string>
const parseRuntimeFormula = (formula: string, extensionHeaders: string[]): ParsedRuntimeFormula => { }>()
return { formula, ast: parse(formula, extensionHeaders).ast }
const getInstanceStore = (instance: object) => {
const existing = instanceStores.get(instance)
if (existing) return existing
const store = {
contracts: new Map<string, RouteContract>(),
formulas: new Map<string, { requires: ParsedRuntimeFormula[]; ensures: ParsedRuntimeFormula[] }>(),
withContracts: new Set<string>(),
} }
export const storeRouteContract = (routeKey: string, contract: RouteContract, extensionHeaders: string[] = []): void => { instanceStores.set(instance, store)
return store
}
export const storeRouteContract = (instance: object, routeKey: string, contract: RouteContract, extensionHeaders: string[] = []): void => {
const store = getInstanceStore(instance)
const parsed = { const parsed = {
requires: contract.requires.map((formula) => parseRuntimeFormula(formula, extensionHeaders)), requires: contract.requires.map((formula) => parseRuntimeFormula(formula, extensionHeaders)),
ensures: contract.ensures.map((formula) => parseRuntimeFormula(formula, extensionHeaders)), ensures: contract.ensures.map((formula) => parseRuntimeFormula(formula, extensionHeaders)),
} }
routeContractStore.set(routeKey, contract) store.contracts.set(routeKey, contract)
routeFormulaStore.set(routeKey, parsed) store.formulas.set(routeKey, parsed)
// Track routes that actually have contracts for fast-path filtering
if (hasContractAnnotations(contract)) { if (hasContractAnnotations(contract)) {
routesWithContracts.add(routeKey) store.withContracts.add(routeKey)
} else { } else {
routesWithContracts.delete(routeKey) store.withContracts.delete(routeKey)
} }
} }
/** Clear the route contract store (useful for testing) */ export const clearRouteContractStore = (instance: object): void => {
export const clearRouteContractStore = (): void => { const store = instanceStores.get(instance)
routeContractStore.clear() if (store) {
routeFormulaStore.clear() store.contracts.clear()
routesWithContracts.clear() store.formulas.clear()
store.withContracts.clear()
}
}
const parseRuntimeFormula = (formula: string, extensionHeaders: string[]): ParsedRuntimeFormula => {
return { formula, ast: parse(formula, extensionHeaders).ast }
} }
const evaluateParsedFormulas = async ( const evaluateParsedFormulas = async (
context: EvalContext, context: EvalContext,
formulas: ParsedRuntimeFormula[], formulas: ParsedRuntimeFormula[],
contract: RouteContract, contract: RouteContract,
level: 'warn' | 'error' = 'error' level: 'warn' | 'error' = 'error',
observers?: HookOptions['observers'],
): Promise<void> => { ): Promise<void> => {
const shouldSample = observers
? Math.random() < (observers.sampling ?? 1)
: false
for (const formula of formulas) { for (const formula of formulas) {
try { try {
const evalResult = await evaluateAsync(formula.ast, context, contract) const evalResult = await evaluateAsync(formula.ast, context, contract)
if (!evalResult.success) { if (!evalResult.success) {
if (shouldSample) {
emitToSinks(observers!.sinks, {
type: 'contract.error',
route: `${contract.method} ${contract.path}`,
method: contract.method,
statusCode: context.response.statusCode,
durationMs: context.response.responseTime ?? 0,
formula: formula.formula,
error: evalResult.error,
sampled: true,
timestamp: new Date().toISOString(),
})
}
throw new Error(evalResult.error) throw new Error(evalResult.error)
} }
const result = Boolean(evalResult.value) const result = Boolean(evalResult.value)
if (!result) { if (!result) {
if (shouldSample) {
emitToSinks(observers!.sinks, {
type: 'contract.violation',
route: `${contract.method} ${contract.path}`,
method: contract.method,
statusCode: context.response.statusCode,
durationMs: context.response.responseTime ?? 0,
formula: formula.formula,
sampled: true,
timestamp: new Date().toISOString(),
})
}
if (observers) {
return
}
const message = `Contract violation: ${formula.formula}` const message = `Contract violation: ${formula.formula}`
if (level === 'warn') { if (level === 'warn') {
console.warn(`[APOPHIS] ${message}`) console.warn(`[APOPHIS] ${message}`)
} else { } else {
throw new Error(message) throw new Error(message)
} }
} else if (shouldSample) {
emitToSinks(observers!.sinks, {
type: 'contract.pass',
route: `${contract.method} ${contract.path}`,
method: contract.method,
statusCode: context.response.statusCode,
durationMs: context.response.responseTime ?? 0,
formula: formula.formula,
sampled: true,
timestamp: new Date().toISOString(),
})
} }
} catch (err) { } catch (err) {
if (err instanceof Error && err.message.startsWith('Contract violation:')) { if (err instanceof Error && err.message.startsWith('Contract violation:')) {
throw err if (!observers) throw err
return
}
if (observers) {
if (shouldSample) {
emitToSinks(observers.sinks, {
type: 'contract.error',
route: `${contract.method} ${contract.path}`,
method: contract.method,
statusCode: context.response.statusCode,
durationMs: context.response.responseTime ?? 0,
formula: formula.formula,
error: err instanceof Error ? err.message : String(err),
sampled: true,
timestamp: new Date().toISOString(),
})
}
return
}
if (level === 'error') {
throw new Error(`Formula evaluation error: ${err instanceof Error ? err.message : String(err)}`)
} }
console.error(`[APOPHIS] Formula evaluation error: ${err instanceof Error ? err.message : String(err)}`) console.error(`[APOPHIS] Formula evaluation error: ${err instanceof Error ? err.message : String(err)}`)
} }
} }
} }
function emitToSinks(sinks: ObserveSink[], event: ObserveEvent): void {
for (const sink of sinks) {
try {
const result = sink.emit(event)
if (result instanceof Promise) {
result.catch(() => {})
}
} catch {
// Sink failures must not affect the response
}
}
}
const createPreHandler = (fastify: FastifyInstance, opts: HookOptions) => { const createPreHandler = (fastify: FastifyInstance, opts: HookOptions) => {
const store = getInstanceStore(fastify)
return (request: FastifyRequest, _reply: FastifyReply, done: (err?: Error) => void): void => { return (request: FastifyRequest, _reply: FastifyReply, done: (err?: Error) => void): void => {
void (async () => { void (async () => {
if (isInternalOperationRequest(request)) { if (isInternalOperationRequest(request)) {
@@ -139,12 +243,11 @@ const createPreHandler = (fastify: FastifyInstance, opts: HookOptions) => {
return return
} }
const routeKey = `${contract.method} ${contract.path}` const routeKey = `${contract.method} ${contract.path}`
// Fast-path: skip if route has no contracts if (!store.withContracts.has(routeKey)) {
if (!routesWithContracts.has(routeKey)) {
done() done()
return return
} }
const stored = routeFormulaStore.get(routeKey) const stored = store.formulas.get(routeKey)
if (!stored) { if (!stored) {
done() done()
return return
@@ -158,15 +261,15 @@ const createPreHandler = (fastify: FastifyInstance, opts: HookOptions) => {
} }
await prefetchPreviousOperations( await prefetchPreviousOperations(
[ [
...stored.requires.map((formula) => formula.ast), ...stored.requires.map((formula: ParsedRuntimeFormula) => formula.ast),
...stored.ensures.map((formula) => formula.ast), ...stored.ensures.map((formula: ParsedRuntimeFormula) => formula.ast),
], ],
preContext, preContext,
contract contract
) )
request[kApophisPreContext] = preContext request[kApophisPreContext] = preContext
if (stored.requires.length > 0) { if (stored.requires.length > 0) {
await evaluateParsedFormulas(preContext, stored.requires, contract, opts.runtimeLevel) await evaluateParsedFormulas(preContext, stored.requires, contract, opts.runtimeLevel, opts.observers)
} }
done() done()
})().catch((err) => { })().catch((err) => {
@@ -175,6 +278,7 @@ const createPreHandler = (fastify: FastifyInstance, opts: HookOptions) => {
} }
} }
const createOnSend = (fastify: FastifyInstance, opts: HookOptions) => { const createOnSend = (fastify: FastifyInstance, opts: HookOptions) => {
const store = getInstanceStore(fastify)
return (request: FastifyRequest, reply: FastifyReply, _payload: unknown, done: (err?: Error) => void): void => { return (request: FastifyRequest, reply: FastifyReply, _payload: unknown, done: (err?: Error) => void): void => {
if (isInternalOperationRequest(request)) { if (isInternalOperationRequest(request)) {
done() done()
@@ -186,12 +290,11 @@ const createOnSend = (fastify: FastifyInstance, opts: HookOptions) => {
return return
} }
const routeKey = `${contract.method} ${contract.path}` const routeKey = `${contract.method} ${contract.path}`
// Fast-path: skip if route has no contracts if (!store.withContracts.has(routeKey)) {
if (!routesWithContracts.has(routeKey)) {
done() done()
return return
} }
const stored = routeFormulaStore.get(routeKey) const stored = store.formulas.get(routeKey)
if (!stored || stored.ensures.length === 0) { if (!stored || stored.ensures.length === 0) {
done() done()
return return
@@ -205,16 +308,13 @@ const createOnSend = (fastify: FastifyInstance, opts: HookOptions) => {
request[kApophisPreContext] request[kApophisPreContext]
), ),
} }
void evaluateParsedFormulas(context, stored.ensures, contract, opts.runtimeLevel) void evaluateParsedFormulas(context, stored.ensures, contract, opts.runtimeLevel, opts.observers)
.then(() => done()) .then(() => done())
.catch((err) => { .catch((err) => {
done(err instanceof Error ? err : new Error(String(err))) done(err instanceof Error ? err : new Error(String(err)))
}) })
} }
} }
export const validateRouteContracts = (): Map<string, { requires: ParsedRuntimeFormula[]; ensures: ParsedRuntimeFormula[] }> => {
return new Map(routeFormulaStore)
}
export const registerValidationHooks = (fastify: FastifyInstance, opts: HookOptions): void => { export const registerValidationHooks = (fastify: FastifyInstance, opts: HookOptions): void => {
fastify.addHook('preHandler', createPreHandler(fastify, opts)) fastify.addHook('preHandler', createPreHandler(fastify, opts))
fastify.addHook('preSerialization', (_request, reply, payload, done) => { fastify.addHook('preSerialization', (_request, reply, payload, done) => {
src/infrastructure/http-executor.ts
+2 -2
View File
@@ -23,9 +23,9 @@ export interface MultipartPayload {
// Minimal interface for Fastify inject — avoids a direct dependency on fastify types. // Minimal interface for Fastify inject — avoids a direct dependency on fastify types.
import type { EvalContext, RouteContract } from '../types.js' import type { EvalContext, RouteContract } from '../types.js'
export const PROTOTYPE_POLLUTION_KEYS = ['__proto__', 'constructor', 'prototype'] as const const PROTOTYPE_POLLUTION_KEYS = ['__proto__', 'constructor', 'prototype'] as const
export const isPrototypePollutionKey = (key: string): boolean => const isPrototypePollutionKey = (key: string): boolean =>
PROTOTYPE_POLLUTION_KEYS.includes(key as typeof PROTOTYPE_POLLUTION_KEYS[number]) PROTOTYPE_POLLUTION_KEYS.includes(key as typeof PROTOTYPE_POLLUTION_KEYS[number])
export const getErrorMessage = (err: unknown): string => export const getErrorMessage = (err: unknown): string =>
src/infrastructure/outbound-mock-runtime.ts
+24 -9
View File
@@ -12,6 +12,7 @@
*/ */
import { convertSchema } from '../domain/schema-to-arbitrary.js' import { convertSchema } from '../domain/schema-to-arbitrary.js'
import { SeededRng } from '../infrastructure/seeded-rng.js' import { SeededRng } from '../infrastructure/seeded-rng.js'
import { matchesTarget } from './wildcard-match.js'
import type { OutboundCallRecord, ResolvedOutboundContract } from '../types.js' import type { OutboundCallRecord, ResolvedOutboundContract } from '../types.js'
import * as fc from 'fast-check' import * as fc from 'fast-check'
@@ -26,7 +27,7 @@ export interface OutboundMockRuntime {
/** Inject a specific response for the next call to a contract (for property testing) */ /** Inject a specific response for the next call to a contract (for property testing) */
injectResponse(contractName: string, statusCode: number, body: unknown): void injectResponse(contractName: string, statusCode: number, body: unknown): void
} }
interface OutboundMockOptions { export interface OutboundMockOptions {
readonly contracts: ResolvedOutboundContract[] readonly contracts: ResolvedOutboundContract[]
readonly mode: 'example' | 'property' readonly mode: 'example' | 'property'
readonly overrides?: Record<string, { readonly overrides?: Record<string, {
@@ -38,6 +39,14 @@ interface OutboundMockOptions {
readonly seed: number readonly seed: number
/** Route-level behavioral contracts to constrain mock responses */ /** Route-level behavioral contracts to constrain mock responses */
readonly routeEnsures?: readonly string[] readonly routeEnsures?: readonly string[]
/** Runtime identifier for diagnostics */
readonly runtimeId?: string
}
let activeRuntimeId: string | undefined
export function getActiveMockRuntimeId(): string | undefined {
return activeRuntimeId
} }
/** Resource store: contractName → resourceId → resourceBody */ /** Resource store: contractName → resourceId → resourceBody */
type ResourceStore = Map<string, Map<string, unknown>> type ResourceStore = Map<string, Map<string, unknown>>
@@ -174,9 +183,20 @@ export function createOutboundMockRuntime(opts: OutboundMockOptions): OutboundMo
return { statusCode: 200, body: generatedBody } return { statusCode: 200, body: generatedBody }
} }
const install = (): void => { const install = (): void => {
if (activeRuntimeId !== undefined) {
throw new Error(
`OutboundMockRuntime already active (owner: ${activeRuntimeId}). ` +
'Only one outbound mock runtime can be installed at a time. ' +
'Restore the existing runtime first. ' +
'Consider running mock-dependent tests serially.'
)
}
if (originalFetch !== undefined) { if (originalFetch !== undefined) {
throw new Error('OutboundMockRuntime already installed') throw new Error('OutboundMockRuntime already installed')
} }
if (opts.runtimeId) {
activeRuntimeId = opts.runtimeId
}
originalFetch = globalThis.fetch originalFetch = globalThis.fetch
globalThis.fetch = async (input: RequestInfo | URL, init?: RequestInit) => { globalThis.fetch = async (input: RequestInfo | URL, init?: RequestInit) => {
const url = typeof input === 'string' ? input : input instanceof URL ? input.href : input.url const url = typeof input === 'string' ? input : input instanceof URL ? input.href : input.url
@@ -259,6 +279,9 @@ export function createOutboundMockRuntime(opts: OutboundMockOptions): OutboundMo
globalThis.fetch = originalFetch globalThis.fetch = originalFetch
originalFetch = undefined originalFetch = undefined
} }
if (opts.runtimeId && activeRuntimeId === opts.runtimeId) {
activeRuntimeId = undefined
}
} }
const getCalls = (name?: string): ReadonlyArray<OutboundCallRecord> => { const getCalls = (name?: string): ReadonlyArray<OutboundCallRecord> => {
if (name === undefined) return calls if (name === undefined) return calls
@@ -276,11 +299,3 @@ export function createOutboundMockRuntime(opts: OutboundMockOptions): OutboundMo
} }
return { install, restore, getCalls, getResource, clear, injectResponse } return { install, restore, getCalls, getResource, clear, injectResponse }
} }
function matchesTarget(url: string, target: string): boolean {
if (target === url) return true
if (target.includes('*')) {
const regex = new RegExp('^' + target.replace(/\*/g, '.*') + '$')
return regex.test(url)
}
return url.includes(target)
}
src/infrastructure/production-safety.ts
+12 -17
View File
@@ -9,6 +9,7 @@
*/ */
import { AsyncLocalStorage } from 'node:async_hooks' import { AsyncLocalStorage } from 'node:async_hooks'
// ============================================================================ // ============================================================================
import type { ApophisExtension } from '../extension/types.js'
import type { ApophisOptions } from '../types.js' import type { ApophisOptions } from '../types.js'
// Environment Detection // Environment Detection
// ============================================================================ // ============================================================================
@@ -40,7 +41,12 @@ export function validateProductionSafety(opts: ApophisOptions): void {
unsafeOptions.push('pluginContracts') unsafeOptions.push('pluginContracts')
} }
if (opts.extensions && opts.extensions.length > 0) { if (opts.extensions && opts.extensions.length > 0) {
unsafeOptions.push('extensions') const exts = opts.extensions as ReadonlyArray<ApophisExtension>
const unsafeExtensions = exts.filter(e => e.productionSafe !== true)
if (unsafeExtensions.length > 0) {
const names = unsafeExtensions.map(e => e.name).join(', ')
unsafeOptions.push(`extensions (unsafe: ${names})`)
}
} }
if (unsafeOptions.length > 0) { if (unsafeOptions.length > 0) {
throw new Error( throw new Error(
@@ -55,10 +61,10 @@ export function validateProductionSafety(opts: ApophisOptions): void {
* Used by individual features (chaos, outbound mocks, etc.) at runtime. * Used by individual features (chaos, outbound mocks, etc.) at runtime.
*/ */
export function assertTestEnv(feature: string): void { export function assertTestEnv(feature: string): void {
if (!isTest()) { if (isProduction()) {
throw new Error( throw new Error(
`${feature} is only available in test environment. ` + `${feature} is not available in production. ` +
`Set NODE_ENV=test to enable quality features.` `Quality features require a non-production environment (NODE_ENV != production).`
) )
} }
} }
@@ -166,20 +172,9 @@ export async function createPassthroughAgent(): Promise<MockAgentType> {
return agent return agent
} }
// ============================================================================ // ============================================================================
// URL-Aware Matching // URL-Aware Matching (re-export from shared utility)
// ============================================================================ // ============================================================================
/** export { matchesTarget } from './wildcard-match.js'
* Check if a URL matches a target pattern.
* Supports exact match, wildcard prefix, and substring.
*/
export function matchesTarget(url: string, target: string): boolean {
if (target === url) return true
if (target.includes('*')) {
const regex = new RegExp('^' + target.replace(/\*/g, '.*') + '$')
return regex.test(url)
}
return url.includes(target)
}
// ============================================================================ // ============================================================================
// Cleanup Helpers // Cleanup Helpers
// ============================================================================ // ============================================================================
src/infrastructure/regex-guard.ts
+1 -1
View File
@@ -6,7 +6,7 @@
import safeRegex from 'safe-regex' import safeRegex from 'safe-regex'
export interface RegexValidationResult { interface RegexValidationResult {
readonly safe: boolean readonly safe: boolean
readonly reason?: string readonly reason?: string
readonly severity?: 'safe' | 'linear' | 'polynomial' | 'exponential' readonly severity?: 'safe' | 'linear' | 'polynomial' | 'exponential'
src/infrastructure/wildcard-match.ts
+22
View File
@@ -0,0 +1,22 @@
function escapeRegex(str: string): string {
return str.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
}
export function matchesTarget(url: string, target: string): boolean {
if (target === url) return true
if (target.includes('*')) {
const escaped = escapeRegex(target)
const regex = new RegExp('^' + escaped.replace(/\\\*/g, '.*') + '$')
return regex.test(url)
}
return url.includes(target)
}
export function matchesWildcardPattern(value: string, pattern: string): boolean {
if (!pattern.includes('*') && !pattern.includes('?')) {
return value.toLowerCase() === pattern.toLowerCase()
}
const escaped = escapeRegex(pattern)
const regexPattern = '^' + escaped.replace(/\\\*/g, '.*').replace(/\\\?/g, '.') + '$'
return new RegExp(regexPattern, 'i').test(value)
}
src/plugin/builders.ts
+40 -8
View File
@@ -11,11 +11,13 @@ import type { ScenarioConfig, ScenarioResult, ScopeRegistry, ApophisOptions, Tes
import type { CleanupManager, TrackedResource } from '../infrastructure/cleanup-manager.js' import type { CleanupManager, TrackedResource } from '../infrastructure/cleanup-manager.js'
import type { PluginContractRegistry } from '../domain/plugin-contracts.js' import type { PluginContractRegistry } from '../domain/plugin-contracts.js'
import type { OutboundContractRegistry } from '../domain/outbound-contracts.js' import type { OutboundContractRegistry } from '../domain/outbound-contracts.js'
import { runScenario } from '../test/scenario-runner.js' import { runScenario } from '../quality/scenario-runner.js'
import { runPetitTests } from '../test/petit-runner.js'
import { runStatefulTests } from '../test/stateful-runner.js' import { runPetitTests } from '../quality/petit-runner.js'
import { runStatefulTests } from '../quality/stateful-runner.js'
import { assertNonProduction } from '../infrastructure/production-safety.js' import { assertNonProduction } from '../infrastructure/production-safety.js'
import { discoverRoutes } from '../domain/discovery.js' import { discoverRoutes, discoverRouteDetails } from '../domain/discovery.js'
import { buildRequest, extractPathParams } from '../domain/request-builder.js' import { buildRequest, extractPathParams } from '../domain/request-builder.js'
import { executeHttp } from '../infrastructure/http-executor.js' import { executeHttp } from '../infrastructure/http-executor.js'
import { validatePostconditionsAsync } from '../domain/contract-validation.js' import { validatePostconditionsAsync } from '../domain/contract-validation.js'
@@ -93,18 +95,35 @@ export const buildContract = (
pluginContractRegistry: PluginContractRegistry, pluginContractRegistry: PluginContractRegistry,
outboundContractRegistry: OutboundContractRegistry outboundContractRegistry: OutboundContractRegistry
) => async (opts: TestConfig = {}): Promise<TestSuite> => { ) => async (opts: TestConfig = {}): Promise<TestSuite> => {
assertNonProduction('contract')
const config = normalizeTestConfig(opts) const config = normalizeTestConfig(opts)
const injectInstance = fastify as unknown as import('../types.js').FastifyInjectInstance const injectInstance = fastify as unknown as import('../types.js').FastifyInjectInstance
const suite = await runPetitTests(injectInstance, config, scope, extensionRegistry, pluginContractRegistry, outboundContractRegistry) const suite = await runPetitTests(injectInstance, config, scope, extensionRegistry, pluginContractRegistry, outboundContractRegistry)
// Loud failure on empty discovery
if (suite.tests.length === 0) { if (suite.tests.length === 0) {
const routes = discoverRoutes(fastify as unknown as { routes?: Array<{ method: string; url: string; schema?: Record<string, unknown> } > }) const discovery = discoverRouteDetails(fastify as unknown as { routes?: Array<{ method: string; url: string; schema?: Record<string, unknown> } > })
if (routes.length === 0) { if (discovery.routes.length === 0) {
throw new Error( throw new Error(
'No routes discovered. Did you register APOPHIS before defining routes? ' + 'No routes discovered. Did you register APOPHIS before defining routes? ' +
'APOPHIS must be registered via `await fastify.register(apophis)` before any routes are defined.' 'APOPHIS must be registered via `await fastify.register(apophis)` before any routes are defined.'
) )
} }
if (!discovery.hasSchemaMetadata) {
throw new Error(
'Routes were discovered but without schema metadata. ' +
'Behavioral contracts (x-ensures, x-requires, x-outbound, x-variants, x-timeout) are unavailable. ' +
'Register APOPHIS or install route discovery before defining routes, ' +
'or use createFastify() from @apophis/fastify.\n\n' +
discovery.warnings.join('\n')
)
}
const anyHaveContracts = discovery.routes.some(r => r.requires.length > 0 || r.ensures.length > 0)
if (!anyHaveContracts) {
throw new Error(
'No behavioral contracts found. ' +
`${discovery.routes.length} route(s) discovered but none have x-ensures or x-requires annotations. ` +
'Add behavioral contracts to route schemas. See docs/getting-started.md for examples.'
)
}
} }
return suite return suite
} }
@@ -131,6 +150,7 @@ export const buildStateful = (
pluginContractRegistry: PluginContractRegistry, pluginContractRegistry: PluginContractRegistry,
outboundContractRegistry: OutboundContractRegistry outboundContractRegistry: OutboundContractRegistry
) => async (opts: TestConfig = {}): Promise<TestSuite> => { ) => async (opts: TestConfig = {}): Promise<TestSuite> => {
assertNonProduction('stateful')
const config = normalizeTestConfig(opts) const config = normalizeTestConfig(opts)
const injectInstance = fastify as unknown as import('../types.js').FastifyInjectInstance const injectInstance = fastify as unknown as import('../types.js').FastifyInjectInstance
return runStatefulTests(injectInstance, config, cleanupManager, scope, extensionRegistry, pluginContractRegistry, outboundContractRegistry) return runStatefulTests(injectInstance, config, cleanupManager, scope, extensionRegistry, pluginContractRegistry, outboundContractRegistry)
@@ -239,8 +259,16 @@ export const registerSwagger = async (fastify: FastifyInstance, opts: ApophisOpt
if ((fastify as unknown as Record<string, unknown>).swagger !== undefined) { if ((fastify as unknown as Record<string, unknown>).swagger !== undefined) {
return return
} }
try {
const swagger = await import('@fastify/swagger') const swagger = await import('@fastify/swagger')
await fastify.register(swagger.default as unknown as Parameters<typeof fastify.register>[0], opts.swagger ?? {}) await fastify.register(swagger.default as unknown as Parameters<typeof fastify.register>[0], opts.swagger ?? {})
} catch (err) {
const message = err instanceof Error ? err.message : String(err)
throw new Error(
`APOPHIS requires @fastify/swagger. Failed to load: ${message}\n` +
'Install with: npm install @fastify/swagger'
)
}
} }
/** /**
@@ -250,7 +278,11 @@ export const registerSwagger = async (fastify: FastifyInstance, opts: ApophisOpt
*/ */
export const buildSpec = (fastify: FastifyInstance) => (): Record<string, unknown> => { export const buildSpec = (fastify: FastifyInstance) => (): Record<string, unknown> => {
const routes = discoverRoutes(fastify as unknown as { routes?: Array<{ method: string; url: string; schema?: Record<string, unknown> } > }) const routes = discoverRoutes(fastify as unknown as { routes?: Array<{ method: string; url: string; schema?: Record<string, unknown> } > })
const spec = (fastify as unknown as { swagger: () => Record<string, unknown> }).swagger() const f = fastify as unknown as Record<string, unknown>
if (typeof f.swagger !== 'function') {
throw new Error('@fastify/swagger is not registered. APOPHIS requires @fastify/swagger. Run: npm install @fastify/swagger')
}
const spec = (f.swagger as () => Record<string, unknown>)()
return { return {
...spec, ...spec,
src/plugin/index.ts
+20 -5
View File
@@ -30,6 +30,8 @@ import {
import type { ApophisDecorations, ApophisOptions, OutboundCallRecord, OutboundContractSpec, TestConfig } from '../types.js' import type { ApophisDecorations, ApophisOptions, OutboundCallRecord, OutboundContractSpec, TestConfig } from '../types.js'
export const apophisPlugin = async (fastify: FastifyInstance, opts: ApophisOptions): Promise<void> => { export const apophisPlugin = async (fastify: FastifyInstance, opts: ApophisOptions): Promise<void> => {
// Guard against double registration
if (fastify.hasDecorator('apophis')) return
// Production safety: hard-fail if test-only options are present in production // Production safety: hard-fail if test-only options are present in production
validateProductionSafety(opts) validateProductionSafety(opts)
await registerSwagger(fastify, opts) await registerSwagger(fastify, opts)
@@ -47,14 +49,16 @@ export const apophisPlugin = async (fastify: FastifyInstance, opts: ApophisOptio
let activeMockRuntime: OutboundMockRuntime | undefined let activeMockRuntime: OutboundMockRuntime | undefined
// Capture routes as they're registered via Fastify's onRoute hook // Capture routes as they're registered via Fastify's onRoute hook
fastify.addHook('onRoute', (routeOptions) => { fastify.addHook('onRoute', (routeOptions) => {
const method = Array.isArray(routeOptions.method) const methods = Array.isArray(routeOptions.method)
? routeOptions.method.join(',') ? routeOptions.method
: routeOptions.method : [routeOptions.method]
const schema = routeOptions.schema as Record<string, unknown> | undefined const schema = routeOptions.schema as Record<string, unknown> | undefined
const prefix = (routeOptions as unknown as Record<string, unknown>).prefix as string | undefined const prefix = (routeOptions as unknown as Record<string, unknown>).prefix as string | undefined
const url = prefix && !routeOptions.url.startsWith(prefix) const url = prefix && !routeOptions.url.startsWith(prefix)
? `${prefix}${routeOptions.url}` ? `${prefix}${routeOptions.url}`
: routeOptions.url : routeOptions.url
for (const method of methods) {
if (!method) continue
captureRoute(fastify, { captureRoute(fastify, {
method, method,
url, url,
@@ -69,7 +73,8 @@ export const apophisPlugin = async (fastify: FastifyInstance, opts: ApophisOptio
routeOptions.config = config as typeof routeOptions.config routeOptions.config = config as typeof routeOptions.config
// Store for hook validator lookup (Fastify doesn't expose routes after ready) // Store for hook validator lookup (Fastify doesn't expose routes after ready)
const routeKey = `${contract.method} ${contract.path}` const routeKey = `${contract.method} ${contract.path}`
storeRouteContract(routeKey, contract, extensionRegistry.getExtensionHeaders()) storeRouteContract(fastify, routeKey, contract, extensionRegistry.getExtensionHeaders())
}
} }
}) })
// Initialize scope registry with explicit config or empty // Initialize scope registry with explicit config or empty
@@ -145,6 +150,16 @@ export const apophisPlugin = async (fastify: FastifyInstance, opts: ApophisOptio
// Runtime validation: never register hooks in production // Runtime validation: never register hooks in production
const isProd = process.env.NODE_ENV === 'production' || process.env.NODE_ENV === 'prod' const isProd = process.env.NODE_ENV === 'production' || process.env.NODE_ENV === 'prod'
if (opts.runtime && opts.runtime !== 'off' && !isProd) { if (opts.runtime && opts.runtime !== 'off' && !isProd) {
registerValidationHooks(fastify, { validateRuntime: true, runtimeLevel: opts.runtime }) const hookOpts: { validateRuntime: boolean; runtimeLevel?: 'warn' | 'error'; observers?: { sinks: import('../types.js').ObserveSink[]; sampling: number } } = {
validateRuntime: true,
runtimeLevel: opts.runtime,
}
if (opts.observe?.enabled && opts.observe.sinks && opts.observe.sinks.length > 0) {
hookOpts.observers = {
sinks: opts.observe.sinks,
sampling: opts.observe.sampling ?? 1,
}
}
registerValidationHooks(fastify, hookOpts as Parameters<typeof registerValidationHooks>[1])
} }
} }
src/quality/chaos-v3.ts
+1 -40
View File
@@ -412,43 +412,4 @@ export function formatChaosEvents(events: ReadonlyArray<ChaosEvent>): string {
export function hasAppliedChaos(events: ReadonlyArray<ChaosEvent>): boolean { export function hasAppliedChaos(events: ReadonlyArray<ChaosEvent>): boolean {
return events.some((e) => e.type !== 'none') return events.some((e) => e.type !== 'none')
} }
// ============================================================================
// Legacy compatibility: Convert old ChaosConfig to chaos events
// ============================================================================
/**
* Convert legacy ChaosConfig into a deterministic set of chaos events.
* Used for backward compatibility during migration.
*/
export function legacyConfigToEvents(config: ChaosConfig): ChaosEvent[] {
const events: ChaosEvent[] = []
if (config.delay) {
events.push({
type: 'inbound-delay',
target: 'inbound',
delayMs: config.delay.minMs,
})
}
if (config.error) {
events.push({
type: 'inbound-error',
target: 'inbound',
statusCode: config.error.statusCode,
body: config.error.body,
})
}
if (config.dropout) {
events.push({
type: 'inbound-dropout',
target: 'inbound',
statusCode: config.dropout.statusCode ?? 504,
})
}
if (config.corruption) {
events.push({
type: 'inbound-corruption',
target: 'inbound',
corruptionStrategy: 'truncate',
})
}
return events
}
src/test/failure-analyzer.ts → src/quality/failure-analyzer.ts
View File
src/test/formatters.ts → src/quality/formatters.ts
View File
src/quality/mutation.ts
+1 -1
View File
@@ -11,7 +11,7 @@
* console.log(`Mutation score: ${report.score}%`) * console.log(`Mutation score: ${report.score}%`)
*/ */
import type { FastifyInstance } from 'fastify' import type { FastifyInstance } from 'fastify'
import { runPetitTests } from '../test/petit-runner.js' import { runPetitTests } from './petit-runner.js'
import { discoverRoutes } from '../domain/discovery.js' import { discoverRoutes } from '../domain/discovery.js'
import type { FastifyInjectInstance, RouteContract, TestConfig, TestSuite } from '../types.js' import type { FastifyInjectInstance, RouteContract, TestConfig, TestSuite } from '../types.js'
src/test/petit-command-step.ts → src/quality/petit-command-step.ts
View File
src/test/petit-formula-utils.ts → src/quality/petit-formula-utils.ts
View File
src/test/petit-runner.ts → src/quality/petit-runner.ts
View File

Some files were not shown because too many files have changed in this diff Show More