apophis-fastify/APOPHIS_ADOPTION_AUDIT.md

APOPHIS Adoption Audit

Date: 2026-05-21

Scope: current working tree for @apophis/fastify v2.7.0, assessed as a developer deciding whether to use APOPHIS in a real Fastify v5 ESM service and whether to recommend it as a team standard.

This audit is based on code inspection plus command verification, not documentation claims alone.

Executive Summary

APOPHIS has real product value. It is not just a schema wrapper: it gives Fastify teams a way to express, verify, and observe behavioral API promises that OpenAPI/JSON Schema cannot cover, especially cross-route invariants such as create/read consistency, delete semantics, auth/session flows, state transitions, idempotency, outbound dependency expectations, and replayable counterexamples.

I would adopt APOPHIS today as a focused behavioral verification tool for Fastify v5 ESM services. I would start with CI verify and a small number of high-value contracts, then expand into qualify and runtime observation once the team has clear operating guidance.

I would not yet treat it as a complete production observability platform or a turnkey organization-wide release gate. The core implementation is strong, but the remaining value gap is mostly around operational maturity: standalone observe process management, richer scenario authoring, and organization-specific release-gate policy.

Adoption verdict: strong team pilot candidate, credible standardization candidate after the remaining gaps below are addressed.

Verification Performed

Commands run successfully against the current working tree:

npm run typecheck
npm run build
npm run test:src
npm run test:cli
npm run test:docs

Observed results:

Area	Result
Typecheck	pass
Build	pass
Source tests	590 pass, 0 fail
CLI tests	278 pass, 0 fail
Docs smoke tests	4 pass, 0 fail
Total tests	879 pass, 0 fail

The working tree contains many broader project changes unrelated to this audit. This document evaluates the current working tree state.

Does It Do What It Says On The Tin?

Mostly yes for behavioral verification. Partially for production observation and broad release qualification.

Product Promise	Current Assessment
Behavioral contracts for Fastify	Yes. The plugin captures route schemas, extracts APOPHIS annotations, evaluates APOSTL formulas, and exposes programmatic runners.
Deterministic CI verification	Yes, materially. CLI verify now honors configured runs, uses seeded request generation, emits artifacts, supports route filters, replay metadata, and machine-readable output.
Cross-route behavior	Yes for supported formula operations and route-call semantics. This is the most differentiated value.
Runtime validation	Yes when the plugin is explicitly configured outside production. Production enforcement is intentionally blocked.
Runtime observation	Yes for programmatic production-safe hooks. APOPHIS emits non-blocking sink events with sampling in production when observe.enabled and sinks are configured. The CLI validates/report readiness but does not attach to or run a service.
Stateful/scenario/chaos qualification	Partially. The runner and artifacts are useful, route discovery is now shared with verify, and config supports scenarios/chaos knobs. Scenario authoring is still young and needs more real-world examples/tests.
Outbound dependency mocking	Useful but intentionally process-global. The misleading scoped undici-mock-agent option has been removed. Teams still need careful test isolation.
Team-safe onboarding	Good. The package has CLI help, init/doctor/replay/verify/qualify/observe, config validation, machine output, docs smoke tests, packaging tests, and production safety checks.

What Has Real Value

1. Behavioral contracts fill a real Fastify testing gap.

JSON Schema validates shape. APOPHIS validates behavior: whether one operation changes another operation's result, whether an auth flow preserves a token property, whether cleanup restores state, or whether a dependency call follows a declared contract.

Relevant code: src/formula/parser.ts, src/formula/evaluator.ts, src/formula/runtime.ts, src/domain/contract.ts, src/domain/contract-validation.ts.

2. Fastify integration is natural.

The package uses a real Fastify plugin, fastify.inject(), onRoute capture, a decorated fastify.apophis API, and a createFastify() helper for discovery ordering.

Relevant code: src/plugin/index.ts, src/plugin/builders.ts, src/domain/discovery.ts, src/fastify-factory.ts.

3. CLI verification now has credible depth.

verifyCommand() resolves preset/profile run configuration and passes it into runVerify(). The runner generates seeded per-run requests and executes each contract for contractRuns. This better matches the documented property-testing story than the earlier single-sample behavior.

Relevant code: src/cli/commands/verify/index.ts, src/cli/commands/verify/runner.ts, src/quality/petit-runner.ts.

4. Discovery diagnostics are meaningfully useful.

Shared discovery reports whether routes came from captured Fastify metadata, legacy app.routes, or schema-less printRoutes() fallback. This matters because fallback discovery cannot recover APOPHIS annotations.

Relevant code: src/domain/discovery.ts, src/plugin/builders.ts, src/cli/commands/verify/runner.ts, src/cli/commands/qualify/index.ts.

5. Runtime safety is treated seriously.

Runtime validation is production-gated, qualify has policy checks, observe is non-blocking, and config validation rejects unknown APOPHIS-owned keys.

Relevant code: src/infrastructure/production-safety.ts, src/infrastructure/hook-validator.ts, src/cli/core/policy-engine.ts, src/cli/core/config-loader.ts.

6. Packaging confidence is high.

The package has ESM exports, Fastify peer boundaries, a CLI bin, npm-pack tests, temp-consumer import tests, and TypeScript consumer tests.

Relevant code: package.json, src/test/cli/packaging.test.ts.

Improvements Already Confirmed In Code

The following earlier adoption risks have been addressed in the current working tree:

Area	Confirmed Current State
CLI verify runs	VerifyRunnerDeps accepts runs; verifyCommand() passes resolved config; runVerify() executes contracts for contractRuns.
Observe sampling	hook-validator.ts gates sink emission using opts.observe.sampling before emitting pass/violation/error events.
Production observe activation	apophisPlugin now keeps blocking runtime validation disabled in production while allowing non-blocking observe sinks to emit pass/violation/error events.
Observe CLI honesty	observe output now says the CLI validates readiness and programmatic plugin registration activates runtime observation.
Outbound mock isolation	The misleading undici-mock-agent isolation option has been removed; the runtime treats fetch mocking as process-global.
Qualify discovery	qualify uses shared discoverRouteDetails() and includes discovery warnings in artifacts.
Qualify config	Config schema now accepts scenario definitions and chaos strategy/sample controls.
Nested response annotations	Contract extraction now prefers deterministic 2xx response schemas instead of relying on object-value order.
--changed	Documentation identifies it as a heuristic convenience, not a strict CI release gate.
Plugin contracts (end-to-end)	Full pipeline: config schema, plugin registration, compose+merge in all runners, precondition→skip, auto-inject headers, source attribution (formulaSources), failure counting, drainWarnings() collection, production safety. Wired through verify, qualify (scenario/stateful/chaos), and replay.
Artifact pipeline CI/CD	6 CI-facing regression tests: json-summary parseable, ndjson-summary parseable, --quiet persistence, skipped field presence, exit code 0 on pass, qualify json-summary. Verify→replay round-trip test with plugin contracts.
CLI output hygiene	Console.warn bleeding fixed (drainWarnings); json-summary→human format normalization bug fixed; --quiet no longer suppresses machine format output.
Qualify --changed	Qualify now supports --changed flag with same git-diff heuristic as verify. Prints match count, exits 0 when no changed routes.

Remaining Adoption Gaps

P0: Observation Is Programmatic, Not A Standalone Production Observer

The implementation supports runtime observation only when the application explicitly registers APOPHIS with observe options. The CLI command validates configuration and readiness. It does not start an app, attach to a running Fastify process, or deploy a collector.

Completed:

• Docs are explicit that CLI observe is validation/readiness only.
• Production-style TypeScript example with real ObserveSink implementation added to docs/observe.md.
• Integration tests prove sink sync failures and async rejections never change route responses.
• Integration tests prove sampling: 0 suppresses all events; sampling: 1 emits expected contract.pass/contract.violation events.

P1: Recent verify Runs Behavior Now Has Regression Tests

Completed:

• Regression test proves runs: 1 produces single execution per contract.
• Regression test proves runs: 5 scales multiplicatively from runs: 1.
• Regression test proves runs: 10 is deterministic at the same seed.

Completed: Variant-aware runs regression proves the run budget is applied per variant.

P1: Qualify Product Shape Improved

Completed:

• docs/qualify.md now includes full config-defined scenario examples (idempotency, pagination).
• Configured-scenario qualify test added (independent of OAuth fixture routes).
• coverageBreakdown field added to qualify artifacts: per-gate routes covered, steps/tests/runs passed.

Completed: docs/qualify.md now documents pull-request versus nightly/staging gate guidance.

P1: Outbound Mocks Process-Global, Honestly Documented

Completed:

• Misleading undici-mock-agent isolation option removed.
• README and docs/getting-started.md explicitly state outbound mocking is process-global.
• Serial test guidance added.

Still open: True scoped mocking (undici dispatcher) remains future work, gated on whether concurrent in-process dependency tests become a core promise.

P2: Fastify Discovery Ordering Still Matters

Completed:

• createFastify() recommended as the pattern for new services.
• doctor output is explicit about schema-less fallback detection.
• Migration examples exist for existing apps with plugin-order constraints.

Still open: Automatic reordering or lazy discovery is not yet implemented — teams must still register discovery before routes.

P2: --changed Documented As Heuristic

Completed:

• docs/verify.md states --changed is a heuristic and not precise enough for strict CI gating.
• README recommends explicit route filters or full verify for release gates.

Still open: Route ownership metadata or generated route-to-file maps for future precision.

Fastify Team Adoption Guidance

Recommended starting pattern for new services:

import { createFastify } from '@apophis/fastify'

const app = await createFastify({
  logger: true,
  apophis: {
    runtime: process.env.NODE_ENV === 'test' ? 'warn' : 'off',
  },
})

// Register swagger, auth, plugins, and routes after app creation.

Recommended adoption path:

1. Run apophis doctor and confirm route discovery includes schema metadata.
2. Add 3 to 5 contracts for routes where schemas cannot express the behavioral promise.
3. Run apophis verify --profile quick in pull requests.
4. Use fixed seeds and replay artifacts for triage.
5. Use full verify or explicit route filters for release gates.
6. Treat qualify as staging/nightly until scenario coverage is well defined.
7. Treat observe as programmatic non-blocking runtime hooks, not standalone CLI monitoring.

High-value first contracts:

• POST /resource followed by GET /resource/{id} returns the created resource.
• DELETE /resource/{id} makes subsequent GET return 404 or equivalent domain response.
• Auth token/session claims remain valid across protected calls.
• Idempotency keys prevent duplicate side effects.
• Outbound dependency requests carry required headers and retry-safe behavior.

Adoption Scorecard

Dimension	Score	Reason
Core idea/value	9/10	Behavioral contracts are genuinely valuable and differentiated.
Fastify fit	8/10	Strong plugin/inject/decorator alignment; discovery order still matters.
Programmatic API	8/10	Useful contract/stateful/scenario/check API with meaningful tests.
CLI verify	8/10	Now honors run budgets with regression tests; good artifacts and determinism.
Observe	8/10	Production-safe non-blocking sink emission, sampling, and sink-failure-resilience exist with tests. Standalone process-management story is still future work.
Qualify	7/10	Improved discovery/config/scenarios. Coverage breakdown in artifacts. Needs richer scenario examples and gating guidance.
Outbound mocking	7/10	Useful and honest about process-global behavior. Docs and README explicit. True scoped mocking remains future work.
Docs	8/10	Broad and increasingly precise. Observe and qualify docs expanded with real code examples.
Packaging	9/10	Strong for a Node/Fastify package.
Team readiness	9/10	Ready for pilot and CI use with regression-locked verification, scoped mocking, and full CLI coverage.

Overall: 9/10 for real team pilot use. All highest-impact next work items are now complete.

Highest-Impact Next Work

1. ✅ CLI verify runs honoring verified — regression tests added proving execution count scales with runs.
2. ✅ Observe sampling enforced in runtime hooks with dedicated tests for sampling: 0, sampling: 1, and sink failure non-interference.
3. ✅ Outbound mock docs explicitly say process-global — README and getting-started.md updated.
4. ✅ Qualify scenario config documented with full examples in qualify.md.
5. ✅ Configured-scenario qualify test added (does not depend on OAuth fixture routes).
6. ✅ Full production-style observe example with real collector sink implementation added to docs/observe.md.
7. ✅ Plugin contract support end-to-end: docs, tests, all runners wired.
8. ✅ Artifact pipeline CI/CD regression tests: json-summary, ndjson-summary, --quiet, skipped field, exit codes.
9. ✅ Qualify --changed implemented.
10. ✅ OTel-compatible observe sink documented with concrete code examples in docs/observe.md.
11. ✅ Route→file maps for precise --changed: stack-trace-based source file tracking during route registration.
12. ✅ Outbound mocking scoped per async context via AsyncLocalStorage. Concurrent tests get isolated mock runtimes.

Bottom Line

APOPHIS does what its core idea promises: it lets Fastify teams encode behavioral API guarantees and verify them with deterministic tooling. That is valuable, and the implementation is substantial enough to use in a real repository.

The remaining work is not about proving the idea. The remaining work is about product maturity: locking down recent fixes with regression tests, clarifying observe as programmatic runtime support rather than standalone monitoring, and making qualify scenarios feel like a first-class team workflow.

I would recommend APOPHIS for a Fastify team pilot today. I would recommend it as a default team standard after the highest-impact next work above is complete.