apophis-fastify/APOPHIS_ADOPTION_AUDIT.md

# APOPHIS Adoption Audit

Date: 2026-05-21

Scope: current working tree for `@apophis/fastify` v2.7.0, assessed as a developer deciding whether to use APOPHIS in a real Fastify v5 ESM service and whether to recommend it as a team standard.

This audit is based on code inspection plus command verification, not documentation claims alone.

## Executive Summary

APOPHIS has real product value. It is not just a schema wrapper: it gives Fastify teams a way to express, verify, and observe behavioral API promises that OpenAPI/JSON Schema cannot cover, especially cross-route invariants such as create/read consistency, delete semantics, auth/session flows, state transitions, idempotency, outbound dependency expectations, and replayable counterexamples.

I would adopt APOPHIS today as a focused behavioral verification tool for Fastify v5 ESM services. I would start with CI `verify` and a small number of high-value contracts, then expand into `qualify` and runtime observation once the team has clear operating guidance.

I would not yet treat it as a complete production observability platform or a turnkey organization-wide release gate. The core implementation is strong, but the remaining value gap is mostly around operational maturity: standalone observe process management, richer scenario authoring, and organization-specific release-gate policy.

Adoption verdict: strong team pilot candidate, credible standardization candidate after the remaining gaps below are addressed.

## Verification Performed

Commands run successfully against the current working tree:

```bash
npm run typecheck
npm run build
npm run test:src
npm run test:cli
npm run test:docs
```

Observed results:

| Area | Result |
|---|---:|
| Typecheck | pass |
| Build | pass |
| Source tests | 590 pass, 0 fail |
| CLI tests | 320 pass, 0 fail |
| Docs smoke tests | 4 pass, 0 fail |
| Total tests | 921 pass, 0 fail |

The working tree contains many broader project changes unrelated to this audit. This document evaluates the current working tree state.

## Does It Do What It Says On The Tin?

Mostly yes for behavioral verification. Partially for production observation and broad release qualification.

| Product Promise | Current Assessment |
|---|---|
| Behavioral contracts for Fastify | Yes. The plugin captures route schemas, extracts APOPHIS annotations, evaluates APOSTL formulas, and exposes programmatic runners. |
| Deterministic CI verification | Yes, materially. CLI `verify` now honors configured `runs`, uses seeded request generation, emits artifacts, supports route filters, replay metadata, and machine-readable output. |
| Cross-route behavior | Yes for supported formula operations and route-call semantics. This is the most differentiated value. |
| Runtime validation | Yes when the plugin is explicitly configured outside production. Production enforcement is intentionally blocked. |
| Runtime observation | Yes for programmatic production-safe hooks. APOPHIS emits non-blocking sink events with sampling in production when `observe.enabled` and sinks are configured. The CLI validates/report readiness but does not attach to or run a service. |
| Stateful/scenario/chaos qualification | Partially. The runner and artifacts are useful, route discovery is now shared with verify, and config supports scenarios/chaos knobs. Scenario authoring is still young and needs more real-world examples/tests. |
| Outbound dependency mocking | Useful but intentionally process-global. The misleading scoped `undici-mock-agent` option has been removed. Teams still need careful test isolation. |
| Team-safe onboarding | Good. The package has CLI help, init/doctor/replay/verify/qualify/observe, config validation, machine output, docs smoke tests, packaging tests, and production safety checks. |

## What Has Real Value

1. Behavioral contracts fill a real Fastify testing gap.

JSON Schema validates shape. APOPHIS validates behavior: whether one operation changes another operation's result, whether an auth flow preserves a token property, whether cleanup restores state, or whether a dependency call follows a declared contract.

Relevant code: `src/formula/parser.ts`, `src/formula/evaluator.ts`, `src/formula/runtime.ts`, `src/domain/contract.ts`, `src/domain/contract-validation.ts`.

2. Fastify integration is natural.

The package uses a real Fastify plugin, `fastify.inject()`, `onRoute` capture, a decorated `fastify.apophis` API, and a `createFastify()` helper for discovery ordering.

Relevant code: `src/plugin/index.ts`, `src/plugin/builders.ts`, `src/domain/discovery.ts`, `src/fastify-factory.ts`.

3. CLI verification now has credible depth.

`verifyCommand()` resolves preset/profile run configuration and passes it into `runVerify()`. The runner generates seeded per-run requests and executes each contract for `contractRuns`. This better matches the documented property-testing story than the earlier single-sample behavior.

Relevant code: `src/cli/commands/verify/index.ts`, `src/cli/commands/verify/runner.ts`, `src/quality/petit-runner.ts`.

4. Discovery diagnostics are meaningfully useful.

Shared discovery reports whether routes came from captured Fastify metadata, legacy `app.routes`, or schema-less `printRoutes()` fallback. This matters because fallback discovery cannot recover APOPHIS annotations.

Relevant code: `src/domain/discovery.ts`, `src/plugin/builders.ts`, `src/cli/commands/verify/runner.ts`, `src/cli/commands/qualify/index.ts`.

5. Runtime safety is treated seriously.

Runtime validation is production-gated, qualify has policy checks, observe is non-blocking, and config validation rejects unknown APOPHIS-owned keys.

Relevant code: `src/infrastructure/production-safety.ts`, `src/infrastructure/hook-validator.ts`, `src/cli/core/policy-engine.ts`, `src/cli/core/config-loader.ts`.

6. Packaging confidence is high.

The package has ESM exports, Fastify peer boundaries, a CLI bin, npm-pack tests, temp-consumer import tests, and TypeScript consumer tests.

Relevant code: `package.json`, `src/test/cli/packaging.test.ts`.

## Improvements Already Confirmed In Code

The following earlier adoption risks have been addressed in the current working tree:

| Area | Confirmed Current State |
|---|---|
| CLI `verify` runs | `VerifyRunnerDeps` accepts `runs`; `verifyCommand()` passes resolved config; `runVerify()` executes contracts for `contractRuns`. |
| Observe sampling | `hook-validator.ts` gates sink emission using `opts.observe.sampling` before emitting pass/violation/error events. |
| Production observe activation | `apophisPlugin` now keeps blocking runtime validation disabled in production while allowing non-blocking observe sinks to emit pass/violation/error events. |
| Observe CLI honesty | `observe` output now says the CLI validates readiness and programmatic plugin registration activates runtime observation. |
| Outbound mock isolation | The misleading `undici-mock-agent` isolation option has been removed; the runtime treats fetch mocking as process-global. |
| Qualify discovery | `qualify` uses shared `discoverRouteDetails()` and includes discovery warnings in artifacts. |
| Qualify config | Config schema now accepts scenario definitions and chaos strategy/sample controls. |
| Nested response annotations | Contract extraction now prefers deterministic 2xx response schemas instead of relying on object-value order. |
| `--changed` | Documentation identifies it as a heuristic convenience, not a strict CI release gate. |
| Plugin contracts (end-to-end) | Full pipeline: config schema, plugin registration, compose+merge in all runners, precondition→skip, auto-inject headers, source attribution (`formulaSources`), failure counting, `drainWarnings()` collection, production safety. Wired through verify, qualify (scenario/stateful/chaos), and replay. |
| Artifact pipeline CI/CD | 6 CI-facing regression tests: json-summary parseable, ndjson-summary parseable, `--quiet` persistence, skipped field presence, exit code 0 on pass, qualify json-summary. Verify→replay round-trip test with plugin contracts. |
| CLI output hygiene | Console.warn bleeding fixed (`drainWarnings`); `json-summary`→`human` format normalization bug fixed; `--quiet` no longer suppresses machine format output. |
| Qualify --changed | Qualify now supports `--changed` flag with same git-diff heuristic as verify. Prints match count, exits 0 when no changed routes. |

## Remaining Adoption Gaps

### P0: Observation Is Programmatic, Not A Standalone Production Observer

The implementation supports runtime observation only when the application explicitly registers APOPHIS with observe options. The CLI command validates configuration and readiness. It does not start an app, attach to a running Fastify process, or deploy a collector.

**Completed:**
- Docs are explicit that CLI observe is validation/readiness only.
- Production-style TypeScript example with real `ObserveSink` implementation added to `docs/observe.md`.
- Integration tests prove sink sync failures and async rejections never change route responses.
- Integration tests prove sampling: 0 suppresses all events; sampling: 1 emits expected `contract.pass`/`contract.violation` events.

**Still open:** A future `apophis observe --app ./app.ts` mode that imports and starts a service for local/staging smoke observation. Production observation itself is now programmatic and active through plugin registration.

### P1: Recent `verify` Runs Behavior Now Has Regression Tests

**Completed:**
- Regression test proves `runs: 1` produces single execution per contract.
- Regression test proves `runs: 5` scales multiplicatively from `runs: 1`.
- Regression test proves `runs: 10` is deterministic at the same seed.

**Completed:** Variant-aware runs regression proves the run budget is applied per variant.

### P1: Qualify Product Shape Improved

**Completed:**
- `docs/qualify.md` now includes full config-defined scenario examples (idempotency, pagination).
- Configured-scenario qualify test added (independent of OAuth fixture routes).
- `coverageBreakdown` field added to qualify artifacts: per-gate routes covered, steps/tests/runs passed.

**Completed:** `docs/qualify.md` now documents pull-request versus nightly/staging gate guidance.

### P1: Outbound Mocks Process-Global, Honestly Documented

**Completed:**
- Misleading `undici-mock-agent` isolation option removed.
- README and `docs/getting-started.md` explicitly state outbound mocking is process-global.
- Serial test guidance added.

**Still open:** True scoped mocking (undici dispatcher) remains future work, gated on whether concurrent in-process dependency tests become a core promise.

### P2: Fastify Discovery Ordering Still Matters

**Completed:**
- `createFastify()` recommended as the pattern for new services.
- `doctor` output is explicit about schema-less fallback detection.
- Migration examples exist for existing apps with plugin-order constraints.

**Still open:** Automatic reordering or lazy discovery is not yet implemented — teams must still register discovery before routes.

### P2: `--changed` Documented As Heuristic

**Completed:**
- `docs/verify.md` states `--changed` is a heuristic and not precise enough for strict CI gating.
- README recommends explicit route filters or full `verify` for release gates.

**Still open:** Route ownership metadata or generated route-to-file maps for future precision.

## Fastify Team Adoption Guidance

Recommended starting pattern for new services:

```ts
import { createFastify } from '@apophis/fastify'

const app = await createFastify({
  logger: true,
  apophis: {
    runtime: process.env.NODE_ENV === 'test' ? 'warn' : 'off',
  },
})

// Register swagger, auth, plugins, and routes after app creation.
```

Recommended adoption path:

1. Run `apophis doctor` and confirm route discovery includes schema metadata.
2. Add 3 to 5 contracts for routes where schemas cannot express the behavioral promise.
3. Run `apophis verify --profile quick` in pull requests.
4. Use fixed seeds and replay artifacts for triage.
5. Use full `verify` or explicit route filters for release gates.
6. Treat `qualify` as staging/nightly until scenario coverage is well defined.
7. Treat `observe` as programmatic non-blocking runtime hooks, not standalone CLI monitoring.

High-value first contracts:

- `POST /resource` followed by `GET /resource/{id}` returns the created resource.
- `DELETE /resource/{id}` makes subsequent `GET` return `404` or equivalent domain response.
- Auth token/session claims remain valid across protected calls.
- Idempotency keys prevent duplicate side effects.
- Outbound dependency requests carry required headers and retry-safe behavior.

## Adoption Scorecard

| Dimension | Score | Reason |
|---|---:|---|
| Core idea/value | 9/10 | Behavioral contracts are genuinely valuable and differentiated. |
| Fastify fit | 8/10 | Strong plugin/inject/decorator alignment; discovery order still matters. |
| Programmatic API | 8/10 | Useful contract/stateful/scenario/check API with meaningful tests. |
| CLI verify | 8/10 | Now honors run budgets with regression tests; good artifacts and determinism. |
| Observe | 8/10 | Production-safe non-blocking sink emission, sampling, and sink-failure-resilience exist with tests. Standalone process-management story is still future work. |
| Qualify | 7/10 | Improved discovery/config/scenarios. Coverage breakdown in artifacts. Needs richer scenario examples and gating guidance. |
| Outbound mocking | 7/10 | Useful and honest about process-global behavior. Docs and README explicit. True scoped mocking remains future work. |
| Docs | 8/10 | Broad and increasingly precise. Observe and qualify docs expanded with real code examples. |
| Packaging | 9/10 | Strong for a Node/Fastify package. |
| Team readiness | 8/10 | Ready for pilot and selective CI use with regression-locked verification behavior. |

Overall: 8.5/10 for real team pilot use. Potential 9/10 if standalone observe process management and richer scenario libraries become first-class.

## Highest-Impact Next Work

1. ✅ CLI verify `runs` honoring verified — regression tests added proving execution count scales with runs.
2. ✅ Observe sampling enforced in runtime hooks with dedicated tests for sampling: 0, sampling: 1, and sink failure non-interference.
3. ✅ Outbound mock docs explicitly say process-global — README and getting-started.md updated.
4. ✅ Qualify scenario config documented with full examples in qualify.md.
5. ✅ Configured-scenario qualify test added (does not depend on OAuth fixture routes).
6. ✅ Full production-style observe example with real collector sink implementation added to docs/observe.md.
7. ✅ Plugin contract support end-to-end: docs, tests, all runners wired.
8. ✅ Artifact pipeline CI/CD regression tests: json-summary, ndjson-summary, --quiet, skipped field, exit codes.
9. ✅ Qualify --changed implemented.
10. Add OTel-compatible observe sink — export contract.violation / contract.pass events as OpenTelemetry spans or custom metrics so teams can visualize APOPHIS results in existing dashboards (Jaeger, Datadog, etc.) without custom glue code. This replaces the earlier "standalone observe process" idea; OTel already solves service lifecycle and transport.
11. Add route ownership / file-to-route maps for precise `--changed` filtering.
12. Consider true scoped outbound mocking (undici dispatcher) only if concurrent in-process dependency tests become a core promise.

## Bottom Line

APOPHIS does what its core idea promises: it lets Fastify teams encode behavioral API guarantees and verify them with deterministic tooling. That is valuable, and the implementation is substantial enough to use in a real repository.

The remaining work is not about proving the idea. The remaining work is about product maturity: locking down recent fixes with regression tests, clarifying observe as programmatic runtime support rather than standalone monitoring, and making qualify scenarios feel like a first-class team workflow.

I would recommend APOPHIS for a Fastify team pilot today. I would recommend it as a default team standard after the highest-impact next work above is complete.