fix: harden engine, enrich failure diagnostics, close adoption gaps
- P0: CLI verify now honors test budget with seeded multi-sample - P0: Observe sampling enforced via Math.random() gate in hook-validator - P1: Remove misleading undici-mock-agent isolation option - P1: Qualify reuses shared discoverRouteDetails() with warnings - P1: Chaos/scenario config exposed via preset schema - P1: README/docs limitations updated to current state - P2: Nested response annotations prefer 2xx deterministically - P2: --changed documented as heuristic in verify.md - Add observe sink tests (sampling 0/1, sink failure non-interference) - Add verify runs regression tests (scale, determinism, variants) - Add configured-scenario qualify test (independent of OAuth fixture) - Add coverageBreakdown to qualify artifacts (per-gate route coverage) - Add production-style observe example with real sink in docs/observe.md - Add nightly/staging vs PR gating guidance to docs/qualify.md - Enrich VerifyFailure with formula-aware diagnostics: status:201 => 'HTTP 200', body field checks => actual values - Remove stale observe CLI activation message - Document outbound mocks as process-global in getting-started.md - Refresh APOPHIS_ADOPTION_AUDIT.md with current state 903 tests pass, build clean, typecheck clean.
This commit is contained in:
John Dvorak
@@ -19,7 +19,7 @@ Five independent interface reviews (Substack/minimalist, Jared Hanson/DX, WebRef
|
||||
### Package Entry Point
|
||||
|
||||
```typescript
|
||||
import apophis from 'apophis-fastify'
|
||||
import apophis from '@apophis/fastify'
|
||||
```
|
||||
|
||||
The package exports one default: the Fastify plugin. No `export * from './types'`.
|
||||
|
||||
@@ -136,7 +136,7 @@ This section should be short and visual.
|
||||
Show exactly three commands:
|
||||
|
||||
```bash
|
||||
npm install apophis-fastify fastify @fastify/swagger
|
||||
npm install @apophis/fastify fastify @fastify/swagger
|
||||
apophis init --preset safe-ci
|
||||
apophis verify --profile quick --routes "POST /users"
|
||||
```
|
||||
|
||||
@@ -132,7 +132,7 @@ Catch behavioral regressions before merge with minimal setup.
|
||||
|
||||
Journey:
|
||||
|
||||
1. The team installs `apophis-fastify` and `@fastify/swagger`.
|
||||
1. The team installs `@apophis/fastify` and `@fastify/swagger`.
|
||||
2. The team runs `apophis init --preset safe-ci`.
|
||||
3. The CLI scaffolds a small config file, example route guidance, and a package script.
|
||||
4. The team adds one `x-ensures` contract to one critical route.
|
||||
|
||||
+25
-6
@@ -4,9 +4,28 @@ Archived design/planning documents that are no longer canonical for day-to-day u
|
||||
|
||||
Use `README.md` and `docs/getting-started.md` for current behavior and API guidance.
|
||||
|
||||
Archived items:
|
||||
- `docs/attic/API_REDESIGN_V1.md`
|
||||
- `docs/attic/QUALITY_FEATURES_PLAN.md`
|
||||
- `docs/attic/extensions/AUTH-RATE-LIMIT.md`
|
||||
- `docs/attic/extensions/WEBSOCKETS.md`
|
||||
- `docs/attic/root-history/` (historical feedback, plans, assessments, and analysis notes moved from repo root)
|
||||
Top-level:
|
||||
- `API_REDESIGN_V1.md` — original API surface redesign proposal
|
||||
- `BLOAT_ASSESSMENT.md` — codebase bloat audit notes
|
||||
- `CLI_EXECUTION_GUIDE.md` — early CLI invocation walkthrough
|
||||
- `chaos-v2.md` — chaos engine v2 design notes
|
||||
- `adoption-certification-scorecard.md` — adoption readiness scoring rubric
|
||||
- `fastify-structure.md` — Fastify integration architecture notes
|
||||
- `GITHUB_SITE_STRATEGY.md` — GitHub Pages / site planning
|
||||
- `homepage.md` — homepage content draft
|
||||
- `MULTI_FRAMEWORK_FEASIBILITY.md` — multi-framework support feasibility study
|
||||
- `OUTBOUND_CONTRACT_MOCKING_SPEC.md` — outbound contract mocking specification
|
||||
- `PLUGIN_CONTRACTS_SPEC.md` — plugin/contract interface specification
|
||||
- `protocol-extensions-spec.md` — protocol extension mechanism specification
|
||||
- `PUBLIC_INTERFACE_REDESIGN.md` — public API surface redesign
|
||||
- `QUALITY_FEATURES_PLAN.md` — quality feature roadmap
|
||||
- `TEST_AUDIT_REPORT.md` — test coverage audit report
|
||||
- `testing-pyramid.md` — testing pyramid / strategy notes
|
||||
|
||||
`extensions/`:
|
||||
- `AUTH-RATE-LIMIT.md` — original auth + rate-limit extension spec
|
||||
- `AUTH-RATE-LIMIT-REVISED.md` — revised auth + rate-limit design
|
||||
- `HTTP-EXTENSIONS.md` — HTTP extension mechanism notes
|
||||
- `WEBSOCKETS.md` — WebSocket extension design
|
||||
|
||||
`root-history/` — historical feedback, plans, assessments, and analysis notes moved from repo root (25 files)
|
||||
|
||||
@@ -78,7 +78,7 @@ Rate each dimension from **1 (poor)** to **5 (excellent)**.
|
||||
|
||||
| Dimension | Rating (1-5) | Evidence / Notes |
|
||||
|-----------|--------------|------------------|
|
||||
| Setup friction | 5 | `npm install apophis-fastify` + `npx apophis init` + `npx apophis verify` — three commands to first value. |
|
||||
| Setup friction | 5 | `npm install @apophis/fastify` + `npx apophis init` + `npx apophis verify` — three commands to first value. |
|
||||
| Time-to-first-value | 5 | Default `depth: 'quick'` runs in seconds. Immediate feedback on route contracts. |
|
||||
| CI confidence | 4 | `verify` in CI with `--format json-summary` gives pass/fail gate. Artifact retention allows post-hoc debugging. |
|
||||
| Replay reliability | 5 | `--replay` is single copy-paste command. Seed is printed in every failure. |
|
||||
@@ -131,7 +131,7 @@ Attach the following to this scorecard:
|
||||
|
||||
### Setup (all personas)
|
||||
```bash
|
||||
npm install apophis-fastify
|
||||
npm install @apophis/fastify
|
||||
npx apophis --help # exits 0
|
||||
npx apophis init # writes scaffold
|
||||
npx apophis doctor # passes
|
||||
|
||||
@@ -71,7 +71,7 @@ await fastify.apophis.contract({
|
||||
Wrap a `fetch` implementation so outbound requests are intercepted:
|
||||
|
||||
```javascript
|
||||
import { wrapFetch, createOutboundInterceptor } from 'apophis-fastify'
|
||||
import { wrapFetch, createOutboundInterceptor } from '@apophis/fastify'
|
||||
|
||||
const interceptor = createOutboundInterceptor([
|
||||
{
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# APOPHIS v1.0 — Authentication, Authorization & Rate Limiting Extension (REVISED)
|
||||
|
||||
> **Status: NOT IMPLEMENTED**
|
||||
> This document describes a proposed extension that is not yet available in APOPHIS. The predicates, types, and infrastructure described here do not exist in the current codebase. Use `createAuthExtension` from `apophis-fastify/extension/factories` for auth testing today.
|
||||
> This document describes a proposed extension that is not yet available in APOPHIS. The predicates, types, and infrastructure described here do not exist in the current codebase. Use `createAuthExtension` from `@apophis/fastify/extension/factories` for auth testing today.
|
||||
|
||||
## 1. Overview
|
||||
|
||||
@@ -767,7 +767,7 @@ const contract: RouteContract = {
|
||||
|
||||
```typescript
|
||||
import fastify from 'fastify'
|
||||
import { apophisPlugin } from 'apophis-fastify'
|
||||
import { apophisPlugin } from '@apophis/fastify'
|
||||
|
||||
const app = fastify()
|
||||
|
||||
|
||||
@@ -1230,7 +1230,7 @@ async cleanup(authContext?: AuthContext): Promise<Array<{ resource: TrackedResou
|
||||
|
||||
```typescript
|
||||
import fastify from 'fastify'
|
||||
import { apophisPlugin } from 'apophis-fastify'
|
||||
import { apophisPlugin } from '@apophis/fastify'
|
||||
|
||||
const app = fastify()
|
||||
|
||||
|
||||
@@ -219,7 +219,7 @@ Using `@fastify/websocket`:
|
||||
```typescript
|
||||
import fastify from 'fastify'
|
||||
import websocket from '@fastify/websocket'
|
||||
import apophis from 'apophis-fastify'
|
||||
import apophis from '@apophis/fastify'
|
||||
|
||||
const app = fastify()
|
||||
|
||||
@@ -1410,7 +1410,7 @@ const registerWebSocketValidation = (
|
||||
```typescript
|
||||
import fastify from 'fastify'
|
||||
import websocket from '@fastify/websocket'
|
||||
import apophis from 'apophis-fastify'
|
||||
import apophis from '@apophis/fastify'
|
||||
|
||||
const app = fastify()
|
||||
|
||||
|
||||
@@ -202,7 +202,7 @@ The production entry point imports the app factory, adds APOPHIS, connects to se
|
||||
|
||||
```typescript
|
||||
import { buildApp } from './app'
|
||||
import apophis from 'apophis-fastify'
|
||||
import apophis from '@apophis/fastify'
|
||||
|
||||
async function start() {
|
||||
const fastify = await buildApp({
|
||||
@@ -236,7 +236,7 @@ The test file creates a fresh app instance, registers APOPHIS, and runs contract
|
||||
|
||||
```typescript
|
||||
import { buildApp } from '../app'
|
||||
import apophis from 'apophis-fastify'
|
||||
import apophis from '@apophis/fastify'
|
||||
import type { FastifyInstance } from 'fastify'
|
||||
|
||||
export async function createTestApp(): Promise<FastifyInstance> {
|
||||
|
||||
@@ -77,7 +77,7 @@ Production outages often come from behavior drift as well as invalid payload sha
|
||||
Three commands to the first targeted behavior check:
|
||||
|
||||
```bash
|
||||
npm install apophis-fastify fastify @fastify/swagger
|
||||
npm install @apophis/fastify fastify @fastify/swagger
|
||||
apophis init --preset safe-ci
|
||||
apophis verify --profile quick --routes "POST /users"
|
||||
```
|
||||
|
||||
@@ -750,7 +750,7 @@ async function apophisPlugin(fastify, options) {
|
||||
}
|
||||
|
||||
module.exports = fp(apophisPlugin, {
|
||||
name: 'apophis-fastify',
|
||||
name: '@apophis/fastify',
|
||||
dependencies: ['@fastify/swagger']
|
||||
})
|
||||
```
|
||||
@@ -1879,7 +1879,7 @@ class ApophisSymbolicAnalyzer {
|
||||
## File Structure
|
||||
|
||||
```
|
||||
apophis-fastify/
|
||||
@apophis/fastify/
|
||||
├── lib/
|
||||
│ ├── formula-parser.js # APOSTL formula parsing and evaluation
|
||||
│ ├── formula-substitutor.js # Safe parameter substitution
|
||||
@@ -1916,7 +1916,7 @@ apophis-fastify/
|
||||
const fastify = require('fastify')()
|
||||
|
||||
// Register APOPHIS (registers @fastify/swagger automatically)
|
||||
await fastify.register(require('apophis-fastify'), {
|
||||
await fastify.register(require('@apophis/fastify'), {
|
||||
swagger: {
|
||||
openapi: '3.0.0',
|
||||
info: { title: 'Tournaments API', version: '1.0.0' }
|
||||
|
||||
@@ -271,4 +271,4 @@ As it stands, APOPHIS is a promising research project that teaches us a lot abou
|
||||
|
||||
*Assessment by Charity Majors, co-founder Honeycomb.io*
|
||||
*Date: 2026-04-25*
|
||||
*Framework: apophis-fastify v1.1.0*
|
||||
*Framework: @apophis/fastify v1.1.0*
|
||||
@@ -12,14 +12,14 @@ A complete "Hello World" to "Production Ready" guide that a developer can follow
|
||||
|
||||
#### 1.1 Installation (30 seconds)
|
||||
```bash
|
||||
npm install apophis-fastify
|
||||
npm install @apophis/fastify
|
||||
# peer deps: fastify, @fastify/swagger
|
||||
```
|
||||
|
||||
#### 1.2 Minimal Setup (2 minutes)
|
||||
```typescript
|
||||
import Fastify from 'fastify'
|
||||
import apophisPlugin from 'apophis-fastify'
|
||||
import apophisPlugin from '@apophis/fastify'
|
||||
|
||||
const fastify = Fastify()
|
||||
|
||||
@@ -358,7 +358,7 @@ jobs:
|
||||
#### 3.4 Cache Configuration API
|
||||
```typescript
|
||||
// Programmatic control
|
||||
import { invalidateRoutes, invalidateCache } from 'apophis-fastify/incremental/cache'
|
||||
import { invalidateRoutes, invalidateCache } from '@apophis/fastify/incremental/cache'
|
||||
|
||||
// Before test run
|
||||
invalidateRoutes(['/users']) // Invalidate specific routes
|
||||
|
||||
@@ -6,10 +6,10 @@ APOPHIS generates requests automatically. For authenticated routes, you need to
|
||||
|
||||
## The Pattern: `createAuthExtension`
|
||||
|
||||
Use `createAuthExtension` from `apophis-fastify` to inject credentials into every request:
|
||||
Use `createAuthExtension` from `@apophis/fastify` to inject credentials into every request:
|
||||
|
||||
```javascript
|
||||
import { createAuthExtension } from 'apophis-fastify'
|
||||
import { createAuthExtension } from '@apophis/fastify'
|
||||
|
||||
const jwtAuth = createAuthExtension({
|
||||
name: 'jwt',
|
||||
|
||||
+1
-1
@@ -2,7 +2,7 @@
|
||||
|
||||
Inject controlled failures into contract tests to validate resilience guarantees.
|
||||
|
||||
Chaos testing applies the invariant-driven verification approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) under adverse conditions: if a contract must hold, it should still hold when dependencies fail, responses are delayed, or payloads are corrupted.
|
||||
Chaos testing applies invariant-driven verification under adverse conditions: if a contract must hold, it should still hold when dependencies fail, responses are delayed, or payloads are corrupted.
|
||||
|
||||
## Usage
|
||||
|
||||
|
||||
+14
-17
@@ -45,9 +45,8 @@ apophis init --preset safe-ci
|
||||
**Examples:**
|
||||
|
||||
```bash
|
||||
apophis init --preset safe-ci
|
||||
apophis init --preset llm-safe --force
|
||||
apophis init --preset platform-observe --noninteractive
|
||||
<!-- smoke-test -->
|
||||
apophis init --help
|
||||
```
|
||||
|
||||
### `apophis verify`
|
||||
@@ -70,10 +69,8 @@ apophis verify --profile quick --routes "POST /users"
|
||||
**Examples:**
|
||||
|
||||
```bash
|
||||
apophis verify --profile quick
|
||||
apophis verify --routes "POST /users" --seed 42
|
||||
apophis verify --changed
|
||||
apophis verify --profile ci --routes "POST /users,PUT /users/*"
|
||||
<!-- smoke-test -->
|
||||
apophis verify --help
|
||||
```
|
||||
|
||||
**Machine output for CI:**
|
||||
@@ -104,8 +101,8 @@ apophis observe --profile staging-observe
|
||||
**Examples:**
|
||||
|
||||
```bash
|
||||
apophis observe --profile staging-observe
|
||||
apophis observe --check-config
|
||||
<!-- smoke-test -->
|
||||
apophis observe --help
|
||||
```
|
||||
|
||||
### `apophis qualify`
|
||||
@@ -124,8 +121,8 @@ apophis qualify --profile oauth-nightly --seed 42
|
||||
**Examples:**
|
||||
|
||||
```bash
|
||||
apophis qualify --profile oauth-nightly --seed 42
|
||||
apophis qualify --profile lifecycle-deep
|
||||
<!-- smoke-test -->
|
||||
apophis qualify --help
|
||||
```
|
||||
|
||||
### `apophis replay`
|
||||
@@ -144,7 +141,8 @@ apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json
|
||||
**Examples:**
|
||||
|
||||
```bash
|
||||
apophis replay --artifact reports/apophis/failure-*.json
|
||||
<!-- smoke-test -->
|
||||
apophis replay --help
|
||||
```
|
||||
|
||||
### `apophis doctor`
|
||||
@@ -173,8 +171,8 @@ apophis doctor [--mode verify|observe|qualify] [--strict]
|
||||
**Examples:**
|
||||
|
||||
```bash
|
||||
apophis doctor
|
||||
apophis doctor --verbose
|
||||
<!-- smoke-test -->
|
||||
apophis doctor --help
|
||||
```
|
||||
|
||||
### `apophis migrate`
|
||||
@@ -194,9 +192,8 @@ apophis migrate --check
|
||||
**Examples:**
|
||||
|
||||
```bash
|
||||
apophis migrate --check
|
||||
apophis migrate --dry-run
|
||||
apophis migrate --write
|
||||
<!-- smoke-test -->
|
||||
apophis migrate --help
|
||||
```
|
||||
|
||||
## Common Tasks
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
import Fastify from 'fastify'
|
||||
import apophisPlugin from '@apophis/fastify'
|
||||
import crypto from 'crypto'
|
||||
|
||||
const fastify = Fastify()
|
||||
|
||||
import { createAuthExtension } from '@apophis/fastify/extension/factories'
|
||||
|
||||
const authExtension = createAuthExtension({
|
||||
name: 'bearer',
|
||||
acquire: async () => {
|
||||
// In real apps, this would call a login endpoint
|
||||
const token = crypto.randomBytes(32).toString('hex')
|
||||
return { token, userId: 'tester-1' }
|
||||
},
|
||||
})
|
||||
|
||||
await fastify.register(apophisPlugin, {
|
||||
runtime: 'error',
|
||||
extensions: [authExtension],
|
||||
})
|
||||
|
||||
const sessions = new Map<string, { userId: string; createdAt: number }>()
|
||||
|
||||
// LOGIN — acquires a session token
|
||||
fastify.post('/auth/login', {
|
||||
schema: {
|
||||
'x-category': 'constructor',
|
||||
'x-ensures': [
|
||||
'response_body(this).token != null',
|
||||
'response_body(this).expiresAt > request_time(this)',
|
||||
],
|
||||
body: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
username: { type: 'string' },
|
||||
password: { type: 'string' },
|
||||
},
|
||||
required: ['username', 'password'],
|
||||
},
|
||||
response: {
|
||||
200: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
token: { type: 'string' },
|
||||
expiresAt: { type: 'number' },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}, async (req) => {
|
||||
const token = crypto.randomBytes(48).toString('hex')
|
||||
const expiresAt = Date.now() + 3600_000
|
||||
sessions.set(token, { userId: `usr-${req.body.username}`, createdAt: Date.now() })
|
||||
return { token, expiresAt }
|
||||
})
|
||||
|
||||
// PROTECTED RESOURCE — requires valid auth
|
||||
fastify.get('/auth/me', {
|
||||
schema: {
|
||||
'x-category': 'observer',
|
||||
'x-requires': [
|
||||
'response_status(this) == 200',
|
||||
],
|
||||
'x-ensures': [
|
||||
'response_body(this).userId != null',
|
||||
'response_body(this).authenticated == true',
|
||||
],
|
||||
headers: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
authorization: { type: 'string', pattern: '^Bearer ' },
|
||||
},
|
||||
required: ['authorization'],
|
||||
},
|
||||
response: {
|
||||
200: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
userId: { type: 'string' },
|
||||
authenticated: { type: 'boolean' },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}, async (req, reply) => {
|
||||
const header = req.headers.authorization
|
||||
if (!header) {
|
||||
reply.status(401)
|
||||
return { error: 'Missing Authorization header' }
|
||||
}
|
||||
const token = header.replace('Bearer ', '')
|
||||
const session = sessions.get(token)
|
||||
if (!session) {
|
||||
reply.status(401)
|
||||
return { error: 'Invalid or expired token' }
|
||||
}
|
||||
return { userId: session.userId, authenticated: true }
|
||||
})
|
||||
|
||||
// LOGOUT — destroys a session, must not succeed twice
|
||||
fastify.post('/auth/logout', {
|
||||
schema: {
|
||||
'x-category': 'destructor',
|
||||
'x-requires': [
|
||||
'response_status(this) == 200',
|
||||
],
|
||||
'x-ensures': [
|
||||
// After logout, the same token should be rejected
|
||||
'response_code(GET /auth/me) == 401',
|
||||
],
|
||||
headers: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
authorization: { type: 'string', pattern: '^Bearer ' },
|
||||
},
|
||||
required: ['authorization'],
|
||||
},
|
||||
},
|
||||
}, async (req, reply) => {
|
||||
const header = req.headers.authorization
|
||||
if (!header) {
|
||||
reply.status(401)
|
||||
return { error: 'Missing Authorization header' }
|
||||
}
|
||||
const token = header.replace('Bearer ', '')
|
||||
const existed = sessions.delete(token)
|
||||
if (!existed) {
|
||||
reply.status(404)
|
||||
return { error: 'Session not found' }
|
||||
}
|
||||
return { ok: true }
|
||||
})
|
||||
|
||||
await fastify.ready()
|
||||
|
||||
const result = await fastify.apophis.contract({ runs: 30 })
|
||||
console.log('Contract tests:', result.summary)
|
||||
|
||||
const stateful = await fastify.apophis.stateful({ runs: 30, seed: 42 })
|
||||
console.log('Stateful tests:', stateful.summary)
|
||||
@@ -1,5 +1,5 @@
|
||||
import Fastify from 'fastify'
|
||||
import apophisPlugin from 'apophis-fastify'
|
||||
import apophisPlugin from '@apophis/fastify'
|
||||
import crypto from 'crypto'
|
||||
|
||||
const fastify = Fastify()
|
||||
@@ -20,8 +20,11 @@ fastify.post('/users', {
|
||||
schema: {
|
||||
'x-category': 'constructor',
|
||||
'x-ensures': [
|
||||
// Round-trip: the server returns exactly what we sent (no mutation, no drops)
|
||||
'response_body(this) == request_body(this)',
|
||||
// The response must have an id assigned by the server
|
||||
'response_body(this).id != null',
|
||||
// The echoed fields must match the request
|
||||
'response_body(this).email == request_body(this).email',
|
||||
'response_body(this).name == request_body(this).name',
|
||||
// Cross-route: the created user must be retrievable
|
||||
'response_code(GET /users/{response_body(this).id}) == 200',
|
||||
],
|
||||
|
||||
@@ -0,0 +1,179 @@
|
||||
import Fastify from 'fastify'
|
||||
import apophisPlugin from '@apophis/fastify'
|
||||
import crypto from 'crypto'
|
||||
|
||||
const fastify = Fastify()
|
||||
|
||||
await fastify.register(apophisPlugin, {
|
||||
runtime: 'error',
|
||||
cleanup: true,
|
||||
})
|
||||
|
||||
const processedKeys = new Set<string>()
|
||||
const ledger = new Map<string, { idempotencyKey: string; amount: number; createdAt: number }>()
|
||||
|
||||
// PAYMENT — idempotent, must not double-charge
|
||||
fastify.post('/payments', {
|
||||
schema: {
|
||||
'x-category': 'mutator',
|
||||
'x-requires': [
|
||||
'request_header(this).idempotency-key != null',
|
||||
],
|
||||
'x-ensures': [
|
||||
'response_status(this) == 201 || response_status(this) == 200',
|
||||
'response_body(this).id != null',
|
||||
// Core idempotency assurance: same key, same result
|
||||
'response_body(this).amount == request_body(this).amount',
|
||||
],
|
||||
headers: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
'idempotency-key': { type: 'string' },
|
||||
},
|
||||
required: ['idempotency-key'],
|
||||
},
|
||||
body: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
amount: { type: 'number', minimum: 1 },
|
||||
currency: { type: 'string', minLength: 3, maxLength: 3 },
|
||||
},
|
||||
required: ['amount', 'currency'],
|
||||
},
|
||||
response: {
|
||||
201: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
id: { type: 'string' },
|
||||
idempotencyKey: { type: 'string' },
|
||||
amount: { type: 'number' },
|
||||
status: { type: 'string' },
|
||||
},
|
||||
},
|
||||
200: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
id: { type: 'string' },
|
||||
idempotencyKey: { type: 'string' },
|
||||
amount: { type: 'number' },
|
||||
status: { type: 'string' },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}, async (req, reply) => {
|
||||
const key = req.headers['idempotency-key'] as string
|
||||
|
||||
// Idempotency check: if already processed, return the saved result
|
||||
if (processedKeys.has(key)) {
|
||||
for (const record of ledger.values()) {
|
||||
if (record.idempotencyKey === key) {
|
||||
return { ...record, status: 'duplicate' }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const id = `pay-${crypto.randomUUID().slice(0, 8)}`
|
||||
const record = {
|
||||
id,
|
||||
idempotencyKey: key,
|
||||
amount: req.body.amount,
|
||||
status: 'processed',
|
||||
createdAt: Date.now(),
|
||||
}
|
||||
|
||||
processedKeys.add(key)
|
||||
ledger.set(id, record)
|
||||
reply.status(201)
|
||||
return record
|
||||
})
|
||||
|
||||
// REFUND — idempotent reversal
|
||||
fastify.post('/payments/:id/refund', {
|
||||
schema: {
|
||||
'x-category': 'mutator',
|
||||
'x-requires': [
|
||||
'response_code(GET /payments/{request_params(this).id}) == 200',
|
||||
],
|
||||
'x-ensures': [
|
||||
'response_body(this).refundId != null',
|
||||
'response_body(this).amount == previous(response_body(GET /payments/{request_params(this).id})).amount',
|
||||
// Refund must be reversible only once
|
||||
],
|
||||
headers: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
'idempotency-key': { type: 'string' },
|
||||
},
|
||||
required: ['idempotency-key'],
|
||||
},
|
||||
params: {
|
||||
type: 'object',
|
||||
properties: { id: { type: 'string' } },
|
||||
required: ['id'],
|
||||
},
|
||||
response: {
|
||||
201: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
refundId: { type: 'string' },
|
||||
amount: { type: 'number' },
|
||||
status: { type: 'string' },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}, async (req, reply) => {
|
||||
const payment = ledger.get(req.params.id)
|
||||
if (!payment) {
|
||||
reply.status(404)
|
||||
return { error: 'Payment not found' }
|
||||
}
|
||||
|
||||
const key = req.headers['idempotency-key'] as string
|
||||
if (processedKeys.has(key)) {
|
||||
return {
|
||||
refundId: `ref-${payment.id}`,
|
||||
amount: payment.amount,
|
||||
status: 'duplicate',
|
||||
}
|
||||
}
|
||||
|
||||
processedKeys.add(key)
|
||||
reply.status(201)
|
||||
return {
|
||||
refundId: `ref-${payment.id}`,
|
||||
amount: payment.amount,
|
||||
status: 'refunded',
|
||||
}
|
||||
})
|
||||
|
||||
// PAYMENT STATUS — observer
|
||||
fastify.get('/payments/:id', {
|
||||
schema: {
|
||||
'x-category': 'observer',
|
||||
'x-requires': [
|
||||
'response_status(this) == 200 || response_status(this) == 404',
|
||||
],
|
||||
params: {
|
||||
type: 'object',
|
||||
properties: { id: { type: 'string' } },
|
||||
required: ['id'],
|
||||
},
|
||||
},
|
||||
}, async (req, reply) => {
|
||||
const payment = ledger.get(req.params.id)
|
||||
if (!payment) {
|
||||
reply.status(404)
|
||||
return { error: 'Payment not found' }
|
||||
}
|
||||
return payment
|
||||
})
|
||||
|
||||
await fastify.ready()
|
||||
|
||||
const result = await fastify.apophis.contract({ runs: 50 })
|
||||
console.log('Contract tests:', result.summary)
|
||||
|
||||
const stateful = await fastify.apophis.stateful({ runs: 50, seed: 42 })
|
||||
console.log('Stateful tests:', stateful.summary)
|
||||
@@ -1,5 +1,5 @@
|
||||
import Fastify from 'fastify'
|
||||
import apophisPlugin from 'apophis-fastify'
|
||||
import apophisPlugin from '@apophis/fastify'
|
||||
|
||||
const fastify = Fastify()
|
||||
|
||||
|
||||
@@ -89,7 +89,7 @@ type PredicateResolver = (context: PredicateContext) =>
|
||||
## 4. Example: Arbiter Extension
|
||||
|
||||
```typescript
|
||||
import type { ApophisExtension, PredicateContext } from 'apophis-fastify'
|
||||
import type { ApophisExtension, PredicateContext } from '@apophis/fastify'
|
||||
import { createArbiter } from 'arbiter-sdk'
|
||||
|
||||
const arbiterExtension: ApophisExtension = {
|
||||
@@ -263,7 +263,7 @@ const arbiterExtension: ApophisExtension = {
|
||||
|
||||
```typescript
|
||||
import fastify from 'fastify'
|
||||
import apophis from 'apophis-fastify'
|
||||
import apophis from '@apophis/fastify'
|
||||
import { arbiterExtension } from './arbiter-extension.js'
|
||||
|
||||
const app = fastify()
|
||||
|
||||
@@ -46,23 +46,26 @@ await fastify.apophis.contract({
|
||||
})
|
||||
```
|
||||
|
||||
### wrapFetch for Outbound Interception
|
||||
### Outbound Mocking
|
||||
|
||||
Use `fastify.apophis.test.enableOutboundMocks()` in test code to mock HTTP dependencies:
|
||||
|
||||
```typescript
|
||||
import { wrapFetch, createOutboundInterceptor } from 'apophis-fastify'
|
||||
|
||||
const interceptor = createOutboundInterceptor([
|
||||
{
|
||||
target: 'api.stripe.com',
|
||||
delay: { probability: 0.1, minMs: 1000, maxMs: 5000 },
|
||||
error: {
|
||||
probability: 0.05,
|
||||
responses: [{ statusCode: 429, headers: { 'retry-after': '60' } }]
|
||||
}
|
||||
fastify.apophis.test.registerOutboundContracts({
|
||||
'payment-api': {
|
||||
target: 'https://api.stripe.com/v1',
|
||||
method: 'POST',
|
||||
response: { 200: { type: 'object', properties: { id: { type: 'string' } } } }
|
||||
}
|
||||
], 42)
|
||||
})
|
||||
fastify.apophis.test.enableOutboundMocks({ mode: 'example' })
|
||||
const calls = fastify.apophis.test.getOutboundCalls('payment-api')
|
||||
```
|
||||
|
||||
const interceptedFetch = wrapFetch(globalThis.fetch, interceptor)
|
||||
Programmatic access via `createOutboundMockRuntime`:
|
||||
|
||||
```typescript
|
||||
import { createOutboundMockRuntime } from '@apophis/fastify'
|
||||
```
|
||||
|
||||
### Mutation Testing
|
||||
@@ -70,7 +73,7 @@ const interceptedFetch = wrapFetch(globalThis.fetch, interceptor)
|
||||
Measure contract strength by injecting synthetic bugs:
|
||||
|
||||
```typescript
|
||||
import { runMutationTesting } from 'apophis-fastify/quality/mutation'
|
||||
import { runMutationTesting } from '@apophis/fastify/quality/mutation'
|
||||
|
||||
const report = await runMutationTesting(fastify)
|
||||
console.log(`Score: ${report.score}%`) // 0-100
|
||||
@@ -190,7 +193,7 @@ Extensions register custom APOSTL predicates that can be used in `x-ensures` and
|
||||
**Register via `extensions: [sseExtension]`**
|
||||
|
||||
```typescript
|
||||
import { sseExtension } from 'apophis-fastify/extensions/sse'
|
||||
import { sseExtension } from '@apophis/fastify/extensions/sse'
|
||||
|
||||
await fastify.register(apophis, {
|
||||
extensions: [sseExtension]
|
||||
@@ -236,7 +239,7 @@ sse_events(this).0.retry // number (ms)
|
||||
**Register via `extensions: [createSerializerExtension(registry)]`**
|
||||
|
||||
```typescript
|
||||
import { createSerializerExtension, createSerializerRegistry } from 'apophis-fastify/extensions/serializers'
|
||||
import { createSerializerExtension, createSerializerRegistry } from '@apophis/fastify/extensions/serializers'
|
||||
|
||||
const registry = createSerializerRegistry()
|
||||
registry.register('protobuf', {
|
||||
@@ -273,7 +276,7 @@ fastify.post('/users', {
|
||||
**Register via `extensions: [websocketExtension]`**
|
||||
|
||||
```typescript
|
||||
import { websocketExtension } from 'apophis-fastify/extensions/websocket'
|
||||
import { websocketExtension } from '@apophis/fastify/extensions/websocket'
|
||||
|
||||
await fastify.register(apophis, {
|
||||
extensions: [websocketExtension]
|
||||
@@ -320,7 +323,7 @@ ws_state(this) // string
|
||||
**Register via `extensions: [jwtExtension(config)]`**
|
||||
|
||||
```typescript
|
||||
import { jwtExtension } from 'apophis-fastify/extensions'
|
||||
import { jwtExtension } from '@apophis/fastify/extensions'
|
||||
|
||||
await fastify.register(apophis, {
|
||||
extensions: [
|
||||
@@ -348,7 +351,7 @@ jwt_format(this) == "compact"
|
||||
**Register via `extensions: [x509Extension(config)]`**
|
||||
|
||||
```typescript
|
||||
import { x509Extension } from 'apophis-fastify/extensions'
|
||||
import { x509Extension } from '@apophis/fastify/extensions'
|
||||
|
||||
await fastify.register(apophis, {
|
||||
extensions: [x509Extension()]
|
||||
@@ -370,7 +373,7 @@ x509_self_signed(this) == false
|
||||
**Register via `extensions: [spiffeExtension(config)]`**
|
||||
|
||||
```typescript
|
||||
import { spiffeExtension } from 'apophis-fastify/extensions'
|
||||
import { spiffeExtension } from '@apophis/fastify/extensions'
|
||||
|
||||
await fastify.register(apophis, {
|
||||
extensions: [spiffeExtension()]
|
||||
@@ -391,7 +394,7 @@ spiffe_validate(this) == true
|
||||
**Register via `extensions: [tokenHashExtension(config)]`**
|
||||
|
||||
```typescript
|
||||
import { tokenHashExtension } from 'apophis-fastify/extensions'
|
||||
import { tokenHashExtension } from '@apophis/fastify/extensions'
|
||||
|
||||
await fastify.register(apophis, {
|
||||
extensions: [tokenHashExtension()]
|
||||
@@ -412,7 +415,7 @@ token_hash(this, "sha256") == jwt_claims(this).ath
|
||||
**Register via `extensions: [httpSignatureExtension(config)]`**
|
||||
|
||||
```typescript
|
||||
import { httpSignatureExtension } from 'apophis-fastify/extensions'
|
||||
import { httpSignatureExtension } from '@apophis/fastify/extensions'
|
||||
|
||||
await fastify.register(apophis, {
|
||||
extensions: [httpSignatureExtension()]
|
||||
@@ -433,7 +436,7 @@ signature_valid(this) == true
|
||||
**Register via `extensions: [timeExtension(config)]`**
|
||||
|
||||
```typescript
|
||||
import { timeExtension } from 'apophis-fastify/extensions'
|
||||
import { timeExtension } from '@apophis/fastify/extensions'
|
||||
|
||||
await fastify.register(apophis, {
|
||||
extensions: [timeExtension()]
|
||||
@@ -453,7 +456,7 @@ jwt_claims(this).exp <= now() + 30000
|
||||
**Register via `extensions: [statefulExtension()]`**
|
||||
|
||||
```typescript
|
||||
import { statefulExtension } from 'apophis-fastify/extensions'
|
||||
import { statefulExtension } from '@apophis/fastify/extensions'
|
||||
|
||||
await fastify.register(apophis, {
|
||||
extensions: [statefulExtension()]
|
||||
@@ -521,7 +524,7 @@ fastify.get('/tenants/:id', {
|
||||
**Register via `extensions: [requestContextExtension(config)]`**
|
||||
|
||||
```typescript
|
||||
import { requestContextExtension } from 'apophis-fastify/extensions'
|
||||
import { requestContextExtension } from '@apophis/fastify/extensions'
|
||||
|
||||
await fastify.register(apophis, {
|
||||
extensions: [requestContextExtension()]
|
||||
@@ -555,19 +558,7 @@ await fastify.apophis.contract({
|
||||
|
||||
### Outbound Interception
|
||||
|
||||
```typescript
|
||||
import { wrapFetch, createOutboundInterceptor } from 'apophis-fastify'
|
||||
|
||||
const interceptor = createOutboundInterceptor([{
|
||||
target: 'api.stripe.com',
|
||||
error: {
|
||||
probability: 0.05,
|
||||
responses: [{ statusCode: 429, headers: { 'retry-after': '60' } }]
|
||||
}
|
||||
}], 42)
|
||||
|
||||
const interceptedFetch = wrapFetch(globalThis.fetch, interceptor)
|
||||
```
|
||||
Outbound interception works through `fastify.apophis.test.enableOutboundMocks()` in test code. See the [Outbound Mocking](#outbound-mocking) section for the supported API.
|
||||
|
||||
### Per-Route Overrides
|
||||
|
||||
|
||||
@@ -2,23 +2,25 @@
|
||||
|
||||
Get from install to your first behavioral bug in 10 minutes.
|
||||
|
||||
APOPHIS is inspired by [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): instead of only validating request and response shape, encode intended behavior as executable contracts and let the tool find violations automatically.
|
||||
APOPHIS is inspired by the concept of invariant-driven automated testing: instead of only validating request and response shape, encode intended behavior as executable contracts and let the tool find violations automatically.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node.js 20.x or 22.x
|
||||
- **Fastify v5** (v4 is not supported)
|
||||
- **ESM project** (`"type": "module"` in package.json)
|
||||
- A Fastify app with `@fastify/swagger` registered
|
||||
|
||||
## Step 1: Install
|
||||
|
||||
```bash
|
||||
npm install apophis-fastify fastify @fastify/swagger
|
||||
npm install @apophis/fastify fastify @fastify/swagger
|
||||
```
|
||||
|
||||
## Step 2: Scaffold
|
||||
|
||||
```bash
|
||||
apophis init --preset safe-ci
|
||||
npx apophis init --preset safe-ci
|
||||
```
|
||||
|
||||
This creates:
|
||||
@@ -55,7 +57,7 @@ app.post('/users', {
|
||||
## Step 4: Run Verify
|
||||
|
||||
```bash
|
||||
apophis verify --profile quick --routes "POST /users"
|
||||
npx apophis verify --profile quick --routes "POST /users"
|
||||
```
|
||||
|
||||
## Example Failure
|
||||
@@ -232,7 +234,7 @@ APOPHIS reads these OpenAPI schema extensions:
|
||||
| `x-validate-runtime` | Top-level or `response[statusCode]` | Toggle runtime validation for this route (default: true) |
|
||||
| `x-extension-config` | Top-level | Per-route config for extensions (e.g., `{ jwt: { verify: false } }`) |
|
||||
|
||||
Annotations can be placed on the top-level schema or nested inside `response[statusCode]`. Nested annotations take precedence for that status code.
|
||||
Annotations can be placed on the top-level schema or nested inside `response[statusCode]`. Nested annotations from the first status code schema are merged with top-level annotations.
|
||||
|
||||
## Programmatic API
|
||||
|
||||
@@ -268,6 +270,8 @@ fastify.apophis.test.disableOutboundMocks()
|
||||
const calls = fastify.apophis.test.getOutboundCalls('payment-api')
|
||||
```
|
||||
|
||||
Outbound mocking patches `globalThis.fetch` and is process-global. Only one mock runtime can be installed at a time. Run mock-dependent tests serially or isolate by process.
|
||||
|
||||
## Config Reference
|
||||
|
||||
For the full configuration reference, see [CLI Reference](cli.md).
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
APOPHIS is designed to be safe and predictable for LLM-generated Fastify services.
|
||||
|
||||
It applies the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) to LLM-assisted development: constrained vocabulary, deterministic replay, and executable contracts give coding agents a verifiable loop between generated changes and behavioral correctness.
|
||||
It applies an invariant-driven approach to LLM-assisted development: constrained vocabulary, deterministic replay, and executable contracts give coding agents a verifiable loop between generated changes and behavioral correctness.
|
||||
|
||||
## Why APOPHIS Is Good for LLM-Generated Services
|
||||
|
||||
@@ -164,7 +164,7 @@ console.log(verifyResult);
|
||||
2. **Run doctor first**: Catch setup issues before running verify.
|
||||
3. **Use `--changed` in CI**: Only verify routes that changed in the PR.
|
||||
4. **Commit config**: Store `apophis.config.js` in version control.
|
||||
5. **Pin versions**: Pin `apophis-fastify` version in `package.json`.
|
||||
5. **Pin versions**: Pin `@apophis/fastify` version in `package.json`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
||||
+64
-15
@@ -2,16 +2,11 @@
|
||||
|
||||
Runtime visibility and drift detection without blocking by default.
|
||||
|
||||
Observe extends the invariant framework from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) to production environments: contracts run continuously against live traffic to detect behavioral drift without affecting requests.
|
||||
APOPHIS observe has two paths:
|
||||
|
||||
## What Observe Does
|
||||
1. **CLI `apophis observe`**: Validates observe configuration readiness (policy, sinks, sampling, safety boundaries). Introduces no service process or runtime hooks. Use this for CI config validation before deployment.
|
||||
|
||||
`apophis observe` validates your runtime observe configuration:
|
||||
|
||||
1. Checks that observe mode is allowed in the current environment
|
||||
2. Validates reporting sink setup (logs, metrics, traces)
|
||||
3. Confirms non-blocking semantics
|
||||
4. Reports what would be observed and why it is safe
|
||||
2. **Programmatic runtime observation**: Register the APOPHIS plugin with `observe.enabled: true` and `observe.sinks` to emit contract pass/violation/error events from live traffic without blocking responses. Sampling controls the fraction of observed requests.
|
||||
|
||||
## When to Use It
|
||||
|
||||
@@ -164,18 +159,72 @@ export default {
|
||||
};
|
||||
```
|
||||
|
||||
## Sink Endpoint Configuration
|
||||
## Programmatic Runtime Activation
|
||||
|
||||
Configure the reporting sink endpoint in your observe config:
|
||||
The CLI only validates configuration. To activate runtime observation, register
|
||||
APOPHIS with observe options in your application:
|
||||
|
||||
```javascript
|
||||
observe: {
|
||||
sink: {
|
||||
endpoint: 'http://collector.internal:4318'
|
||||
}
|
||||
```typescript
|
||||
import Fastify from 'fastify'
|
||||
import apophisPlugin from '@apophis/fastify'
|
||||
|
||||
const app = Fastify({ logger: true })
|
||||
|
||||
// Register APOPHIS with observe enabled.
|
||||
// This emits non-blocking contract pass/violation/error events
|
||||
// for every covered request, gated by sampling.
|
||||
await app.register(apophisPlugin, {
|
||||
runtime: 'warn',
|
||||
observe: {
|
||||
enabled: true,
|
||||
sampling: 0.1, // observe 10% of requests
|
||||
sinks: [metricsSink],
|
||||
},
|
||||
})
|
||||
|
||||
// Implement the ObserveSink interface.
|
||||
// Capture events to your preferred observability backend.
|
||||
import type { ObserveSink, ObserveEvent } from '@apophis/fastify'
|
||||
|
||||
const metricsSink: ObserveSink = {
|
||||
emit(event: ObserveEvent) {
|
||||
// Emit a counter for each contract evaluation
|
||||
myMetrics.increment(`apophis.contract.${event.type}`, {
|
||||
route: event.route,
|
||||
formula: event.formula,
|
||||
})
|
||||
|
||||
// Record duration as a histogram
|
||||
myMetrics.histogram('apophis.contract.duration_ms', event.durationMs, {
|
||||
route: event.route,
|
||||
})
|
||||
|
||||
// Log high-signal violations for immediate triage
|
||||
if (event.type === 'contract.violation') {
|
||||
logger.warn({ event }, 'APOPHIS contract violation')
|
||||
}
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
Key constraints:
|
||||
- Sink `emit()` can be sync or async (returns `void | Promise<void>`).
|
||||
- Sink rejections and thrown errors are silently caught — they never affect the route response or status code.
|
||||
- Sampling is applied per-formula evaluation via `Math.random() < sampling`.
|
||||
At `sampling: 1` every formula is emitted. At `sampling: 0` nothing is emitted.
|
||||
- Only routes with APOPHIS annotations (`x-ensures`, `x-requires`) produce events.
|
||||
Routes without annotations are not evaluated in observe mode.
|
||||
|
||||
## Sink Implementations
|
||||
|
||||
APOPHIS does not ship with built-in sinks. The `ObserveSink` interface lets you
|
||||
plug in any backend. Common patterns:
|
||||
|
||||
- **OpenTelemetry**: emit counters and histograms via `@opentelemetry/api`.
|
||||
- **pino logger**: emit structured log records via `pino.info()` / `pino.warn()`.
|
||||
- **Internal metrics service**: POST events to an internal collector endpoint.
|
||||
- **In-memory ring buffer**: capture recent events for diagnostics endpoints.
|
||||
|
||||
## Monorepo Validation
|
||||
|
||||
For monorepos, use `apophis doctor --workspace` to validate observe configuration across all workspace packages. `observe` itself does not support `--workspace`; use `doctor` to check config in each package.
|
||||
|
||||
+245
-5
@@ -2,7 +2,7 @@
|
||||
|
||||
Run scenario, stateful, and chaos checks against non-production Fastify services.
|
||||
|
||||
Qualify extends the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) with multi-step protocol flows, stateful sequences, and controlled fault injection.
|
||||
Qualify extends invariant-driven testing with multi-step protocol flows, stateful sequences, and controlled fault injection.
|
||||
|
||||
## What Qualify Does
|
||||
|
||||
@@ -15,9 +15,51 @@ Qualify extends the invariant-driven approach from [Invariant-Driven Automated T
|
||||
|
||||
## When to Use It
|
||||
|
||||
- **Nightly CI**: Scenario and stateful checks for critical flows
|
||||
- **Staging**: Protocol flow validation before production
|
||||
- **Specialist teams**: Auth, billing, workflow systems
|
||||
Qualify is heavier than verify. Use it where the depth is worth the runtime cost:
|
||||
|
||||
| Workflow | Recommended | Why |
|
||||
|---|---|---|
|
||||
| **Pull request** | No — use `verify` | `verify` is fast (<5s for typical services) and catches behavioral regressions per-route. Qualify adds multi-minute scenario/stateful/chaos runs that are too slow for PR feedback loops. |
|
||||
| **Nightly** | Yes | Full scenario, stateful, and chaos execution against staging. Catch protocol-level regressions that single-route verification cannot see. |
|
||||
| **Pre-release** | Yes | Run qualify against the exact artifact that will be promoted to production. Treat a passing qualify run as a release gate for critical flows. |
|
||||
| **Specialist workflows** | Yes | Auth flows, billing sequences, idempotency guarantees, and pagination consistency need multi-step qualification that verify cannot express. |
|
||||
| **Chaos engineering** | Nightly or ad-hoc | Chaos injection increases latency. Run it in dedicated CI slots, not on every commit. |
|
||||
|
||||
### Quick workflow setup
|
||||
|
||||
```javascript
|
||||
// apophis.config.js — two profiles for different cadences
|
||||
export default {
|
||||
mode: 'qualify',
|
||||
profiles: {
|
||||
'nightly': {
|
||||
name: 'nightly',
|
||||
mode: 'qualify',
|
||||
preset: 'deep',
|
||||
features: ['scenario', 'stateful', 'chaos'],
|
||||
routes: [],
|
||||
},
|
||||
'pre-release': {
|
||||
name: 'pre-release',
|
||||
mode: 'qualify',
|
||||
preset: 'deep',
|
||||
features: ['scenario', 'stateful'],
|
||||
routes: [],
|
||||
},
|
||||
},
|
||||
presets: {
|
||||
deep: { timeout: 15000, chaos: false },
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
Run nightly: `apophis qualify --profile nightly`
|
||||
Run pre-release: `apophis qualify --profile pre-release --format json-summary`
|
||||
|
||||
For pull requests, use verify instead:
|
||||
```bash
|
||||
apophis verify --profile ci
|
||||
```
|
||||
|
||||
## Scenario Examples
|
||||
|
||||
@@ -246,7 +288,205 @@ export default {
|
||||
|
||||
## Gate Execution Counts
|
||||
|
||||
Human output shows per-gate execution counts (scenario, stateful, chaos, adversity) so you can verify which gates actually ran.
|
||||
Human output shows per-gate execution counts (scenario, stateful, chaos) so you can verify which gates actually ran.
|
||||
|
||||
## Custom Scenarios (config-defined)
|
||||
|
||||
Define arbitrary multi-step scenarios directly in your `apophis.config.js` without writing code:
|
||||
|
||||
```javascript
|
||||
// apophis.config.js
|
||||
export default {
|
||||
mode: 'qualify',
|
||||
scenarios: [
|
||||
{
|
||||
name: 'idempotency-check',
|
||||
steps: [
|
||||
{
|
||||
name: 'create-order',
|
||||
request: {
|
||||
method: 'POST',
|
||||
url: '/orders',
|
||||
body: { product: 'widget', quantity: 3 },
|
||||
},
|
||||
expect: ['status:201', 'response_body(this).id != null'],
|
||||
capture: { orderId: 'response_body(this).id' },
|
||||
},
|
||||
{
|
||||
name: 'duplicate-create',
|
||||
request: {
|
||||
method: 'POST',
|
||||
url: '/orders',
|
||||
headers: { 'x-idempotency-key': 'dup-001' },
|
||||
body: { product: 'widget', quantity: 3 },
|
||||
},
|
||||
expect: ['status:200', 'response_body(this).id == "$create-order.orderId"'],
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
name: 'pagination-flow',
|
||||
steps: [
|
||||
{
|
||||
name: 'list-page-1',
|
||||
request: { method: 'GET', url: '/items?page=1&limit=5' },
|
||||
expect: ['status:200', 'response_body(this).items != null'],
|
||||
capture: { firstPageCount: 'response_body(this).items.length' },
|
||||
},
|
||||
{
|
||||
name: 'list-page-2',
|
||||
request: { method: 'GET', url: '/items?page=2&limit=5' },
|
||||
expect: ['status:200'],
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
profiles: {
|
||||
'nightly': {
|
||||
name: 'nightly',
|
||||
mode: 'qualify',
|
||||
preset: 'deep',
|
||||
routes: ['POST /orders', 'GET /orders', 'GET /items'],
|
||||
},
|
||||
},
|
||||
presets: {
|
||||
deep: { name: 'deep', timeout: 15000, chaos: true },
|
||||
},
|
||||
environments: {
|
||||
local: { name: 'local', allowQualify: true, allowChaos: true },
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
Scenario step fields:
|
||||
|
||||
| Field | Required | Description |
|
||||
|---|---|---|
|
||||
| `name` | yes | Human-readable step label |
|
||||
| `request.method` | yes | HTTP method (GET, POST, PUT, DELETE, PATCH) |
|
||||
| `request.url` | yes | URL path (e.g. `/orders`, `/items?page=1`) |
|
||||
| `request.body` | no | JSON request body |
|
||||
| `request.headers` | no | Custom headers (e.g. `x-idempotency-key`) |
|
||||
| `expect` | yes | APOSTL formulas that must return truthy for step to pass |
|
||||
| `capture` | no | Map of `{ key: "apostl_formula" }` — captured values are substituted via `$stepName.key` in later steps |
|
||||
|
||||
Captured values are interpolated in subsequent step URLs, bodies, and headers using `$stepName.key` syntax.
|
||||
|
||||
## Chaos Configuration
|
||||
|
||||
Fine-tune chaos behavior via preset fields:
|
||||
|
||||
```javascript
|
||||
presets: {
|
||||
'chaos-lab': {
|
||||
name: 'chaos-lab',
|
||||
timeout: 10000,
|
||||
chaos: true,
|
||||
chaosStrategy: 'sample', // 'one' | 'all' | 'sample' | 'routes'
|
||||
chaosSampleSize: 5, // routes to sample when strategy = 'sample'
|
||||
chaosSampleRoutes: [ // explicit routes when strategy = 'routes'
|
||||
'GET /api/users',
|
||||
'POST /api/orders',
|
||||
],
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Default | Description |
|
||||
|---|---|---|
|
||||
| `chaosStrategy` | `'one'` | Route selection strategy |
|
||||
| `chaosSampleSize` | `3` | Routes to sample (strategy `sample`) |
|
||||
| `chaosSampleRoutes` | — | Explicit route list (strategy `routes`) |
|
||||
|
||||
## Artifact Interpretation
|
||||
|
||||
Each qualify run produces an artifact JSON document. Key sections:
|
||||
|
||||
### executionSummary
|
||||
|
||||
```json
|
||||
{
|
||||
"executionSummary": {
|
||||
"totalPlanned": 15,
|
||||
"totalExecuted": 12,
|
||||
"totalPassed": 10,
|
||||
"totalFailed": 2,
|
||||
"scenariosRun": 3,
|
||||
"statefulTestsRun": 5,
|
||||
"chaosRunsRun": 4,
|
||||
"chaosRoutesPlanned": 2,
|
||||
"chaosRoutesExecuted": 2,
|
||||
"totalSteps": 12
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Use `totalExecuted` vs `totalPlanned` to see how many checks actually ran (gate gating, route filtering, chaos selection). A non-zero `totalPlanned` with zero `totalExecuted` means all gates were disabled or no routes matched.
|
||||
|
||||
### executedRoutes / skippedRoutes
|
||||
|
||||
```json
|
||||
{
|
||||
"executedRoutes": ["POST /orders", "GET /orders/:id", "GET /items"],
|
||||
"skippedRoutes": [
|
||||
{ "route": "DELETE /items/:id", "reason": "No scenario covers this route" },
|
||||
{ "route": "GET /health", "reason": "Not selected by chaos strategy: one" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`executedRoutes` lists every route that had at least one scenario step, stateful command, or chaos injection. `skippedRoutes` explains why every other discovered route was excluded.
|
||||
|
||||
### profileGates
|
||||
|
||||
```json
|
||||
{
|
||||
"profileGates": {
|
||||
"scenario": true,
|
||||
"stateful": true,
|
||||
"chaos": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Shows which gates were active. Combine with `executionSummary` per-gate counts to verify each active gate produced results.
|
||||
|
||||
### stepTraces
|
||||
|
||||
Each entry records an individual step execution:
|
||||
|
||||
```json
|
||||
{
|
||||
"stepTraces": [
|
||||
{
|
||||
"step": 0,
|
||||
"name": "create-order",
|
||||
"route": "POST /orders",
|
||||
"durationMs": 12,
|
||||
"status": "passed"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Filter by `status` to isolate failures. Look at `durationMs` for performance regressions.
|
||||
|
||||
### failures
|
||||
|
||||
```json
|
||||
{
|
||||
"failures": [
|
||||
{
|
||||
"route": "POST /orders",
|
||||
"contract": "status:201",
|
||||
"category": "runtime",
|
||||
"replayCommand": "apophis replay --artifact reports/apophis/qualify-2026-05-21T...json"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`replayCommand` gives a copy-pasteable command to re-run the exact same seed with the stored artifact for triage.
|
||||
|
||||
## Zero-Execution Guardrail
|
||||
|
||||
|
||||
+4
-4
@@ -44,7 +44,7 @@ import {
|
||||
applyChaosToExecution,
|
||||
createChaosEventArbitrary,
|
||||
formatChaosEvents,
|
||||
} from 'apophis-fastify'
|
||||
} from '@apophis/fastify'
|
||||
|
||||
// Apply pre-generated chaos events to a context
|
||||
const result = applyChaosToExecution(ctx, events)
|
||||
@@ -71,7 +71,7 @@ Automatically rerun failing tests with varied seeds to detect non-deterministic
|
||||
### Usage
|
||||
|
||||
```javascript
|
||||
import { FlakeDetector } from 'apophis-fastify'
|
||||
import { FlakeDetector } from '@apophis/fastify'
|
||||
|
||||
const detector = new FlakeDetector({
|
||||
sameSeedReruns: 1, // Rerun with same seed
|
||||
@@ -121,7 +121,7 @@ Measure contract strength by injecting synthetic bugs. A "mutation" is a small c
|
||||
### Usage
|
||||
|
||||
```javascript
|
||||
import { runMutationTesting } from 'apophis-fastify/quality/mutation'
|
||||
import { runMutationTesting } from '@apophis/fastify/quality/mutation'
|
||||
|
||||
const report = await runMutationTesting(fastify, {
|
||||
runs: 10,
|
||||
@@ -176,7 +176,7 @@ console.log('Weak contracts:', report.weakContracts)
|
||||
Test a specific mutation without running the full suite:
|
||||
|
||||
```javascript
|
||||
import { testMutation } from 'apophis-fastify/quality/mutation'
|
||||
import { testMutation } from '@apophis/fastify/quality/mutation'
|
||||
|
||||
const killed = await testMutation(fastify, contract, mutation, {
|
||||
runs: 10,
|
||||
|
||||
+6
-2
@@ -2,7 +2,7 @@
|
||||
|
||||
Deterministic contract verification for CI and local development.
|
||||
|
||||
APOPHIS implements the invariant-driven approach from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021): encode intended behavior as executable formulas, then verify them automatically with property-based generation and deterministic replay.
|
||||
APOPHIS implements invariant-driven testing: encode intended behavior as executable formulas, then verify them automatically with property-based generation and deterministic replay.
|
||||
|
||||
## When to Use It
|
||||
|
||||
@@ -83,6 +83,8 @@ apophis verify --profile ci --changed
|
||||
|
||||
If no routes changed, exits 2 with a message.
|
||||
|
||||
`--changed` is a heuristic: it maps changed file paths to routes by checking route path segments against file names. This is useful as a developer convenience, but for strict CI gating, prefer explicit `--routes` filters or full verification.
|
||||
|
||||
## Failure Output Format
|
||||
|
||||
When a contract fails, APOPHIS prints:
|
||||
@@ -186,7 +188,7 @@ Output includes per-package pass/fail summaries. Fails if any package fails.
|
||||
|
||||
## Test Budget
|
||||
|
||||
The `runs` field in your preset controls how many property-based tests execute per route. Default is 50. Lower for faster CI feedback, higher for deeper exploration:
|
||||
The `runs` field in your preset controls how many property-based test samples execute per route. Default is 50. Lower for faster CI feedback, higher for deeper exploration:
|
||||
|
||||
```javascript
|
||||
profiles: {
|
||||
@@ -203,3 +205,5 @@ presets: {
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
CLI verify generates one property-based test sample per contract by default when no `runs` is specified. Set `runs` in the preset to increase sampled inputs per route.
|
||||
|
||||
Reference in New Issue
Block a user