apophis-fastify/APOPHIS_ADOPTION_AUDIT.md

# APOPHIS Adoption Audit

Date: 2026-05-21

Scope: current working tree for `@apophis/fastify` v2.7.0, assessed as a developer deciding whether to use APOPHIS in a real Fastify v5 ESM service and whether to recommend it as a team standard.

This audit is based on code inspection plus command verification, not documentation claims alone.

## Executive Summary

APOPHIS has real product value. It is not just a schema wrapper: it gives Fastify teams a way to express and verify behavioral API promises that OpenAPI/JSON Schema cannot cover, especially cross-route invariants such as create/read consistency, delete semantics, auth/session flows, state transitions, idempotency, outbound dependency expectations, and replayable counterexamples.

I would adopt APOPHIS today as a focused behavioral verification tool for Fastify v5 ESM services. I would start with CI `verify` and a small number of high-value contracts, then expand into `qualify` and runtime observation once the team has clear operating guidance.

I would not yet treat it as a complete production observability platform or a turnkey organization-wide release gate. The core implementation is strong, but the remaining value gap is mostly around operational maturity: standalone observe activation, deeper tests around recent CLI behavior, richer scenario authoring, and clearer release-gate recommendations.

Adoption verdict: strong team pilot candidate, credible standardization candidate after the remaining gaps below are addressed.

## Verification Performed

Commands run successfully against the current working tree:

```bash
npm run typecheck
npm run build
npm run test:src
npm run test:cli
npm run test:docs
```

Observed results:

| Area | Result |
|---|---:|
| Typecheck | pass |
| Build | pass |
| Source tests | 587 pass, 0 fail |
| CLI tests | 311 pass, 0 fail |
| Docs smoke tests | 4 pass, 0 fail |
| Total tests | 902 pass, 0 fail |

The working tree contains many broader project changes unrelated to this audit. This document evaluates the current working tree state.

## Does It Do What It Says On The Tin?

Mostly yes for behavioral verification. Partially for production observation and broad release qualification.

| Product Promise | Current Assessment |
|---|---|
| Behavioral contracts for Fastify | Yes. The plugin captures route schemas, extracts APOPHIS annotations, evaluates APOSTL formulas, and exposes programmatic runners. |
| Deterministic CI verification | Yes, materially. CLI `verify` now honors configured `runs`, uses seeded request generation, emits artifacts, supports route filters, replay metadata, and machine-readable output. |
| Cross-route behavior | Yes for supported formula operations and route-call semantics. This is the most differentiated value. |
| Runtime validation | Yes when the plugin is explicitly configured outside production. Production enforcement is intentionally blocked. |
| Runtime observation | Partially. Programmatic plugin observation exists and emits non-blocking sink events with sampling. The CLI validates/report readiness but does not attach to or run a service. |
| Stateful/scenario/chaos qualification | Partially. The runner and artifacts are useful, route discovery is now shared with verify, and config supports scenarios/chaos knobs. Scenario authoring is still young and needs more real-world examples/tests. |
| Outbound dependency mocking | Useful but intentionally process-global. The misleading scoped `undici-mock-agent` option has been removed. Teams still need careful test isolation. |
| Team-safe onboarding | Good. The package has CLI help, init/doctor/replay/verify/qualify/observe, config validation, machine output, docs smoke tests, packaging tests, and production safety checks. |

## What Has Real Value

1. Behavioral contracts fill a real Fastify testing gap.

JSON Schema validates shape. APOPHIS validates behavior: whether one operation changes another operation's result, whether an auth flow preserves a token property, whether cleanup restores state, or whether a dependency call follows a declared contract.

Relevant code: `src/formula/parser.ts`, `src/formula/evaluator.ts`, `src/formula/runtime.ts`, `src/domain/contract.ts`, `src/domain/contract-validation.ts`.

2. Fastify integration is natural.

The package uses a real Fastify plugin, `fastify.inject()`, `onRoute` capture, a decorated `fastify.apophis` API, and a `createFastify()` helper for discovery ordering.

Relevant code: `src/plugin/index.ts`, `src/plugin/builders.ts`, `src/domain/discovery.ts`, `src/fastify-factory.ts`.

3. CLI verification now has credible depth.

`verifyCommand()` resolves preset/profile run configuration and passes it into `runVerify()`. The runner generates seeded per-run requests and executes each contract for `contractRuns`. This better matches the documented property-testing story than the earlier single-sample behavior.

Relevant code: `src/cli/commands/verify/index.ts`, `src/cli/commands/verify/runner.ts`, `src/quality/petit-runner.ts`.

4. Discovery diagnostics are meaningfully useful.

Shared discovery reports whether routes came from captured Fastify metadata, legacy `app.routes`, or schema-less `printRoutes()` fallback. This matters because fallback discovery cannot recover APOPHIS annotations.

Relevant code: `src/domain/discovery.ts`, `src/plugin/builders.ts`, `src/cli/commands/verify/runner.ts`, `src/cli/commands/qualify/index.ts`.

5. Runtime safety is treated seriously.

Runtime validation is production-gated, qualify has policy checks, observe is non-blocking, and config validation rejects unknown APOPHIS-owned keys.

Relevant code: `src/infrastructure/production-safety.ts`, `src/infrastructure/hook-validator.ts`, `src/cli/core/policy-engine.ts`, `src/cli/core/config-loader.ts`.

6. Packaging confidence is high.

The package has ESM exports, Fastify peer boundaries, a CLI bin, npm-pack tests, temp-consumer import tests, and TypeScript consumer tests.

Relevant code: `package.json`, `src/test/cli/packaging.test.ts`.

## Improvements Already Confirmed In Code

The following earlier adoption risks have been addressed in the current working tree:

| Area | Confirmed Current State |
|---|---|
| CLI `verify` runs | `VerifyRunnerDeps` accepts `runs`; `verifyCommand()` passes resolved config; `runVerify()` executes contracts for `contractRuns`. |
| Observe sampling | `hook-validator.ts` gates sink emission using `opts.observe.sampling` before emitting pass/violation/error events. |
| Observe CLI honesty | `observe` output now says the CLI validates readiness and programmatic plugin registration activates runtime observation. |
| Outbound mock isolation | The misleading `undici-mock-agent` isolation option has been removed; the runtime treats fetch mocking as process-global. |
| Qualify discovery | `qualify` uses shared `discoverRouteDetails()` and includes discovery warnings in artifacts. |
| Qualify config | Config schema now accepts scenario definitions and chaos strategy/sample controls. |
| Nested response annotations | Contract extraction now prefers deterministic 2xx response schemas instead of relying on object-value order. |
| `--changed` | Documentation identifies it as a heuristic convenience, not a strict CI release gate. |

## Remaining Adoption Gaps

### P0: Observation Is Programmatic, Not A Standalone Production Observer

The implementation supports runtime observation only when the application explicitly registers APOPHIS with observe options. The CLI command validates configuration and readiness. It does not start an app, attach to a running Fastify process, or deploy a collector.

**Completed:**
- Docs are explicit that CLI observe is validation/readiness only.
- Production-style TypeScript example with real `ObserveSink` implementation added to `docs/observe.md`.
- Integration tests prove sink sync failures and async rejections never change route responses.
- Integration tests prove sampling: 0 suppresses all events; sampling: 1 emits expected `contract.pass`/`contract.violation` events.

**Still open:** A future `apophis observe --app ./app.ts` mode that activates a running service observer.

### P1: Recent `verify` Runs Behavior Now Has Regression Tests

**Completed:**
- Regression test proves `runs: 1` produces single execution per contract.
- Regression test proves `runs: 5` scales multiplicatively from `runs: 1`.
- Regression test proves `runs: 10` is deterministic at the same seed.

**Still open:** Variant-aware runs test (verifying run budget is per-variant or shared).

### P1: Qualify Product Shape Improved

**Completed:**
- `docs/qualify.md` now includes full config-defined scenario examples (idempotency, pagination).
- Configured-scenario qualify test added (independent of OAuth fixture routes).
- `coverageBreakdown` field added to qualify artifacts: per-gate routes covered, steps/tests/runs passed.

**Still open:** Clear guidance for nightly/staging use versus pull-request gating in qualify docs.

### P1: Outbound Mocks Process-Global, Honestly Documented

**Completed:**
- Misleading `undici-mock-agent` isolation option removed.
- README and `docs/getting-started.md` explicitly state outbound mocking is process-global.
- Serial test guidance added.

**Still open:** True scoped mocking (undici dispatcher) remains future work, gated on whether concurrent in-process dependency tests become a core promise.

### P2: Fastify Discovery Ordering Still Matters

**Completed:**
- `createFastify()` recommended as the pattern for new services.
- `doctor` output is explicit about schema-less fallback detection.
- Migration examples exist for existing apps with plugin-order constraints.

**Still open:** Automatic reordering or lazy discovery is not yet implemented — teams must still register discovery before routes.

### P2: `--changed` Documented As Heuristic

**Completed:**
- `docs/verify.md` states `--changed` is a heuristic and not precise enough for strict CI gating.
- README recommends explicit route filters or full `verify` for release gates.

**Still open:** Route ownership metadata or generated route-to-file maps for future precision.

## Fastify Team Adoption Guidance

Recommended starting pattern for new services:

```ts
import { createFastify } from '@apophis/fastify'

const app = await createFastify({
  logger: true,
  apophis: {
    runtime: process.env.NODE_ENV === 'test' ? 'warn' : 'off',
  },
})

// Register swagger, auth, plugins, and routes after app creation.
```

Recommended adoption path:

1. Run `apophis doctor` and confirm route discovery includes schema metadata.
2. Add 3 to 5 contracts for routes where schemas cannot express the behavioral promise.
3. Run `apophis verify --profile quick` in pull requests.
4. Use fixed seeds and replay artifacts for triage.
5. Use full `verify` or explicit route filters for release gates.
6. Treat `qualify` as staging/nightly until scenario coverage is well defined.
7. Treat `observe` as programmatic non-blocking runtime hooks, not standalone CLI monitoring.

High-value first contracts:

- `POST /resource` followed by `GET /resource/{id}` returns the created resource.
- `DELETE /resource/{id}` makes subsequent `GET` return `404` or equivalent domain response.
- Auth token/session claims remain valid across protected calls.
- Idempotency keys prevent duplicate side effects.
- Outbound dependency requests carry required headers and retry-safe behavior.

## Adoption Scorecard

| Dimension | Score | Reason |
|---|---:|---|
| Core idea/value | 9/10 | Behavioral contracts are genuinely valuable and differentiated. |
| Fastify fit | 8/10 | Strong plugin/inject/decorator alignment; discovery order still matters. |
| Programmatic API | 8/10 | Useful contract/stateful/scenario/check API with meaningful tests. |
| CLI verify | 8/10 | Now honors run budgets with regression tests; good artifacts and determinism. |
| Observe | 7/10 | Runtime sink primitives, sampling, and sink-failure-resilience exist with tests. Production-style docs added. Standalone operational story not complete. |
| Qualify | 7/10 | Improved discovery/config/scenarios. Coverage breakdown in artifacts. Needs richer scenario examples and gating guidance. |
| Outbound mocking | 7/10 | Useful and honest about process-global behavior. Docs and README explicit. True scoped mocking remains future work. |
| Docs | 8/10 | Broad and increasingly precise. Observe and qualify docs expanded with real code examples. |
| Packaging | 9/10 | Strong for a Node/Fastify package. |
| Team readiness | 8/10 | Ready for pilot and selective CI use with regression-locked verification behavior. |

Overall: 8/10 for real team pilot use. Potential 9/10 if observe gains a clearer production story and qualify gets first-class CI workflow guidance.

## Highest-Impact Next Work

1. ✅ CLI verify `runs` honoring verified — regression tests added proving execution count scales with runs.
2. ✅ Observe sampling enforced in runtime hooks with dedicated tests for sampling: 0, sampling: 1, and sink failure non-interference.
3. ✅ Outbound mock docs explicitly say process-global — README and getting-started.md updated.
4. ✅ Qualify scenario config documented with full examples in qualify.md.
5. ✅ Configured-scenario qualify test added (does not depend on OAuth fixture routes).
6. Add full production-style observe example with a real collector sink implementation.
7. Improve qualify artifact coverage summaries to distinguish route-contract, scenario, stateful, and chaos coverage more clearly.
8. Consider true scoped outbound mocking (undici dispatcher) only if concurrent in-process dependency tests become a core promise.

## Bottom Line

APOPHIS does what its core idea promises: it lets Fastify teams encode behavioral API guarantees and verify them with deterministic tooling. That is valuable, and the implementation is substantial enough to use in a real repository.

The remaining work is not about proving the idea. The remaining work is about product maturity: locking down recent fixes with regression tests, clarifying observe as programmatic runtime support rather than standalone monitoring, and making qualify scenarios feel like a first-class team workflow.

I would recommend APOPHIS for a Fastify team pilot today. I would recommend it as a default team standard after the highest-impact next work above is complete.