fix: harden engine, enrich failure diagnostics, close adoption gaps

- P0: CLI verify now honors  test budget with seeded multi-sample
- P0: Observe sampling enforced via Math.random() gate in hook-validator
- P1: Remove misleading undici-mock-agent isolation option
- P1: Qualify reuses shared discoverRouteDetails() with warnings
- P1: Chaos/scenario config exposed via preset schema
- P1: README/docs limitations updated to current state
- P2: Nested response annotations prefer 2xx deterministically
- P2: --changed documented as heuristic in verify.md

- Add observe sink tests (sampling 0/1, sink failure non-interference)
- Add verify runs regression tests (scale, determinism, variants)
- Add configured-scenario qualify test (independent of OAuth fixture)
- Add coverageBreakdown to qualify artifacts (per-gate route coverage)
- Add production-style observe example with real sink in docs/observe.md
- Add nightly/staging vs PR gating guidance to docs/qualify.md

- Enrich VerifyFailure with formula-aware diagnostics:
  status:201 => 'HTTP 200', body field checks => actual values
- Remove stale observe CLI activation message
- Document outbound mocks as process-global in getting-started.md
- Refresh APOPHIS_ADOPTION_AUDIT.md with current state

903 tests pass, build clean, typecheck clean.
This commit is contained in:
John Dvorak
2026-05-21 20:39:36 -07:00
parent 55b0262799
commit d0523fcc2d
128 changed files with 4004 additions and 3631 deletions
+1 -1
View File
@@ -19,7 +19,7 @@ Five independent interface reviews (Substack/minimalist, Jared Hanson/DX, WebRef
### Package Entry Point
```typescript
import apophis from 'apophis-fastify'
import apophis from '@apophis/fastify'
```
The package exports one default: the Fastify plugin. No `export * from './types'`.
+1 -1
View File
@@ -136,7 +136,7 @@ This section should be short and visual.
Show exactly three commands:
```bash
npm install apophis-fastify fastify @fastify/swagger
npm install @apophis/fastify fastify @fastify/swagger
apophis init --preset safe-ci
apophis verify --profile quick --routes "POST /users"
```
+1 -1
View File
@@ -132,7 +132,7 @@ Catch behavioral regressions before merge with minimal setup.
Journey:
1. The team installs `apophis-fastify` and `@fastify/swagger`.
1. The team installs `@apophis/fastify` and `@fastify/swagger`.
2. The team runs `apophis init --preset safe-ci`.
3. The CLI scaffolds a small config file, example route guidance, and a package script.
4. The team adds one `x-ensures` contract to one critical route.
+25 -6
View File
@@ -4,9 +4,28 @@ Archived design/planning documents that are no longer canonical for day-to-day u
Use `README.md` and `docs/getting-started.md` for current behavior and API guidance.
Archived items:
- `docs/attic/API_REDESIGN_V1.md`
- `docs/attic/QUALITY_FEATURES_PLAN.md`
- `docs/attic/extensions/AUTH-RATE-LIMIT.md`
- `docs/attic/extensions/WEBSOCKETS.md`
- `docs/attic/root-history/` (historical feedback, plans, assessments, and analysis notes moved from repo root)
Top-level:
- `API_REDESIGN_V1.md` — original API surface redesign proposal
- `BLOAT_ASSESSMENT.md` — codebase bloat audit notes
- `CLI_EXECUTION_GUIDE.md` — early CLI invocation walkthrough
- `chaos-v2.md` — chaos engine v2 design notes
- `adoption-certification-scorecard.md` — adoption readiness scoring rubric
- `fastify-structure.md` — Fastify integration architecture notes
- `GITHUB_SITE_STRATEGY.md` — GitHub Pages / site planning
- `homepage.md` — homepage content draft
- `MULTI_FRAMEWORK_FEASIBILITY.md` — multi-framework support feasibility study
- `OUTBOUND_CONTRACT_MOCKING_SPEC.md` — outbound contract mocking specification
- `PLUGIN_CONTRACTS_SPEC.md` — plugin/contract interface specification
- `protocol-extensions-spec.md` — protocol extension mechanism specification
- `PUBLIC_INTERFACE_REDESIGN.md` — public API surface redesign
- `QUALITY_FEATURES_PLAN.md` — quality feature roadmap
- `TEST_AUDIT_REPORT.md` — test coverage audit report
- `testing-pyramid.md` — testing pyramid / strategy notes
`extensions/`:
- `AUTH-RATE-LIMIT.md` — original auth + rate-limit extension spec
- `AUTH-RATE-LIMIT-REVISED.md` — revised auth + rate-limit design
- `HTTP-EXTENSIONS.md` — HTTP extension mechanism notes
- `WEBSOCKETS.md` — WebSocket extension design
`root-history/` — historical feedback, plans, assessments, and analysis notes moved from repo root (25 files)
@@ -78,7 +78,7 @@ Rate each dimension from **1 (poor)** to **5 (excellent)**.
| Dimension | Rating (1-5) | Evidence / Notes |
|-----------|--------------|------------------|
| Setup friction | 5 | `npm install apophis-fastify` + `npx apophis init` + `npx apophis verify` — three commands to first value. |
| Setup friction | 5 | `npm install @apophis/fastify` + `npx apophis init` + `npx apophis verify` — three commands to first value. |
| Time-to-first-value | 5 | Default `depth: 'quick'` runs in seconds. Immediate feedback on route contracts. |
| CI confidence | 4 | `verify` in CI with `--format json-summary` gives pass/fail gate. Artifact retention allows post-hoc debugging. |
| Replay reliability | 5 | `--replay` is single copy-paste command. Seed is printed in every failure. |
@@ -131,7 +131,7 @@ Attach the following to this scorecard:
### Setup (all personas)
```bash
npm install apophis-fastify
npm install @apophis/fastify
npx apophis --help # exits 0
npx apophis init # writes scaffold
npx apophis doctor # passes
+1 -1
View File
@@ -71,7 +71,7 @@ await fastify.apophis.contract({
Wrap a `fetch` implementation so outbound requests are intercepted:
```javascript
import { wrapFetch, createOutboundInterceptor } from 'apophis-fastify'
import { wrapFetch, createOutboundInterceptor } from '@apophis/fastify'
const interceptor = createOutboundInterceptor([
{
@@ -1,7 +1,7 @@
# APOPHIS v1.0 — Authentication, Authorization & Rate Limiting Extension (REVISED)
> **Status: NOT IMPLEMENTED**
> This document describes a proposed extension that is not yet available in APOPHIS. The predicates, types, and infrastructure described here do not exist in the current codebase. Use `createAuthExtension` from `apophis-fastify/extension/factories` for auth testing today.
> This document describes a proposed extension that is not yet available in APOPHIS. The predicates, types, and infrastructure described here do not exist in the current codebase. Use `createAuthExtension` from `@apophis/fastify/extension/factories` for auth testing today.
## 1. Overview
@@ -767,7 +767,7 @@ const contract: RouteContract = {
```typescript
import fastify from 'fastify'
import { apophisPlugin } from 'apophis-fastify'
import { apophisPlugin } from '@apophis/fastify'
const app = fastify()
+1 -1
View File
@@ -1230,7 +1230,7 @@ async cleanup(authContext?: AuthContext): Promise<Array<{ resource: TrackedResou
```typescript
import fastify from 'fastify'
import { apophisPlugin } from 'apophis-fastify'
import { apophisPlugin } from '@apophis/fastify'
const app = fastify()
+2 -2
View File
@@ -219,7 +219,7 @@ Using `@fastify/websocket`:
```typescript
import fastify from 'fastify'
import websocket from '@fastify/websocket'
import apophis from 'apophis-fastify'
import apophis from '@apophis/fastify'
const app = fastify()
@@ -1410,7 +1410,7 @@ const registerWebSocketValidation = (
```typescript
import fastify from 'fastify'
import websocket from '@fastify/websocket'
import apophis from 'apophis-fastify'
import apophis from '@apophis/fastify'
const app = fastify()
+2 -2
View File
@@ -202,7 +202,7 @@ The production entry point imports the app factory, adds APOPHIS, connects to se
```typescript
import { buildApp } from './app'
import apophis from 'apophis-fastify'
import apophis from '@apophis/fastify'
async function start() {
const fastify = await buildApp({
@@ -236,7 +236,7 @@ The test file creates a fresh app instance, registers APOPHIS, and runs contract
```typescript
import { buildApp } from '../app'
import apophis from 'apophis-fastify'
import apophis from '@apophis/fastify'
import type { FastifyInstance } from 'fastify'
export async function createTestApp(): Promise<FastifyInstance> {
+1 -1
View File
@@ -77,7 +77,7 @@ Production outages often come from behavior drift as well as invalid payload sha
Three commands to the first targeted behavior check:
```bash
npm install apophis-fastify fastify @fastify/swagger
npm install @apophis/fastify fastify @fastify/swagger
apophis init --preset safe-ci
apophis verify --profile quick --routes "POST /users"
```
+3 -3
View File
@@ -750,7 +750,7 @@ async function apophisPlugin(fastify, options) {
}
module.exports = fp(apophisPlugin, {
name: 'apophis-fastify',
name: '@apophis/fastify',
dependencies: ['@fastify/swagger']
})
```
@@ -1879,7 +1879,7 @@ class ApophisSymbolicAnalyzer {
## File Structure
```
apophis-fastify/
@apophis/fastify/
├── lib/
│ ├── formula-parser.js # APOSTL formula parsing and evaluation
│ ├── formula-substitutor.js # Safe parameter substitution
@@ -1916,7 +1916,7 @@ apophis-fastify/
const fastify = require('fastify')()
// Register APOPHIS (registers @fastify/swagger automatically)
await fastify.register(require('apophis-fastify'), {
await fastify.register(require('@apophis/fastify'), {
swagger: {
openapi: '3.0.0',
info: { title: 'Tournaments API', version: '1.0.0' }
@@ -271,4 +271,4 @@ As it stands, APOPHIS is a promising research project that teaches us a lot abou
*Assessment by Charity Majors, co-founder Honeycomb.io*
*Date: 2026-04-25*
*Framework: apophis-fastify v1.1.0*
*Framework: @apophis/fastify v1.1.0*
@@ -12,14 +12,14 @@ A complete "Hello World" to "Production Ready" guide that a developer can follow
#### 1.1 Installation (30 seconds)
```bash
npm install apophis-fastify
npm install @apophis/fastify
# peer deps: fastify, @fastify/swagger
```
#### 1.2 Minimal Setup (2 minutes)
```typescript
import Fastify from 'fastify'
import apophisPlugin from 'apophis-fastify'
import apophisPlugin from '@apophis/fastify'
const fastify = Fastify()
@@ -358,7 +358,7 @@ jobs:
#### 3.4 Cache Configuration API
```typescript
// Programmatic control
import { invalidateRoutes, invalidateCache } from 'apophis-fastify/incremental/cache'
import { invalidateRoutes, invalidateCache } from '@apophis/fastify/incremental/cache'
// Before test run
invalidateRoutes(['/users']) // Invalidate specific routes