- P0: CLI verify now honors test budget with seeded multi-sample - P0: Observe sampling enforced via Math.random() gate in hook-validator - P1: Remove misleading undici-mock-agent isolation option - P1: Qualify reuses shared discoverRouteDetails() with warnings - P1: Chaos/scenario config exposed via preset schema - P1: README/docs limitations updated to current state - P2: Nested response annotations prefer 2xx deterministically - P2: --changed documented as heuristic in verify.md - Add observe sink tests (sampling 0/1, sink failure non-interference) - Add verify runs regression tests (scale, determinism, variants) - Add configured-scenario qualify test (independent of OAuth fixture) - Add coverageBreakdown to qualify artifacts (per-gate route coverage) - Add production-style observe example with real sink in docs/observe.md - Add nightly/staging vs PR gating guidance to docs/qualify.md - Enrich VerifyFailure with formula-aware diagnostics: status:201 => 'HTTP 200', body field checks => actual values - Remove stale observe CLI activation message - Document outbound mocks as process-global in getting-started.md - Refresh APOPHIS_ADOPTION_AUDIT.md with current state 903 tests pass, build clean, typecheck clean.
9.1 KiB
Getting Started with APOPHIS
Get from install to your first behavioral bug in 10 minutes.
APOPHIS is inspired by the concept of invariant-driven automated testing: instead of only validating request and response shape, encode intended behavior as executable contracts and let the tool find violations automatically.
Prerequisites
- Node.js 20.x or 22.x
- Fastify v5 (v4 is not supported)
- ESM project (
"type": "module"in package.json) - A Fastify app with
@fastify/swaggerregistered
Step 1: Install
npm install @apophis/fastify fastify @fastify/swagger
Step 2: Scaffold
npx apophis init --preset safe-ci
This creates:
apophis.config.js— config with aquickprofileAPOPHIS.md— preset-specific guidance- Package script:
npm run apophis:verify
Step 3: Add One Behavioral Contract
Pick one important route. Add an x-ensures clause that checks behavior across operations:
import crypto from 'crypto';
app.post('/users', {
schema: {
'x-category': 'constructor',
'x-ensures': [
// BEHAVIORAL: Creating a user must make it retrievable
'response_code(GET /users/{response_body(this).id}) == 200'
]
}
}, async (request, reply) => {
const { name } = request.body;
const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
reply.status(201);
return { id, name };
});
Warning: Using
Date.now()orMath.random()in handlers breaks determinism and replay. Use a stable function of the input instead. APOPHIS does not proactively detect nondeterministic handlers; it warns only when a replay diverges from the original run.
Step 4: Run Verify
npx apophis verify --profile quick --routes "POST /users"
Example Failure
If your GET /users/:id handler has a bug (always returns 404), APOPHIS catches it:
Contract violation
POST /users
Profile: quick
Seed: 42
Expected
response_code(GET /users/{response_body(this).id}) == 200
Observed
GET /users/usr-7d865e returned 404
Why this matters
The resource created by POST /users is not retrievable.
Replay
apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json
Next
Check the create/read consistency for POST /users and GET /users/{id}.
Step 5: Replay and Fix
Copy the replay command and run it:
apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json
Fix the bug in your handler. Re-run verify. The failure should now pass.
Behavioral vs Structural Contracts
APOPHIS contracts should verify behavior, not structure. Fastify and @fastify/swagger already enforce status codes, required fields, and types. Behavioral contracts catch what schemas cannot:
| Structural (avoid) | Behavioral (prefer) |
|---|---|
status:200 |
response_body(this) == request_body(this) |
response_body(this).id != null |
response_code(GET /users/{response_body(this).id}) == 200 |
response_body(this).name != null |
response_body(GET /users/{id}).name == previous(response_body(this).name) |
Good behavioral patterns (from the paper):
- Constructor precondition: Resource must not exist before creation
response_code(GET /users/{request_body(this).email}) == 404 - Round-trip equality: POST response matches the request body
response_body(this) == request_body(this) - Cross-route retrievability: Creating a resource makes it readable via GET
response_code(GET /users/{response_body(this).id}) == 200 - State-change verification: DELETE causes subsequent GET to return 404
response_code(GET /users/{request_params(this).id}) == 404 - Previous state preservation: DELETE returns the last known state
response_body(this) == previous(response_body(GET /users/{request_params(this).id})) - Invariant over collections: All resources satisfy a cross-resource constraint
for t in response_body(GET /tournaments) :- response_body(GET /tournaments/{t.id}/players).length <= t.capacity
Anti-patterns to avoid:
- Checking status codes (handled by schema validation)
- Checking field existence (handled by schema validation)
- Checking field types (handled by schema validation)
Next Steps
- Add more routes to your profile:
apophis verify --profile quick --routes "POST /users,PUT /users/:id" - Use wildcards to match route patterns:
apophis verify --routes 'POST /api/*' - Run all routes:
apophis verify --profile quick - Run only changed routes in CI:
apophis verify --profile ci --changed- Requires a git repository.
- Use machine-readable output in CI:
apophis verify --profile ci --format json-summary - Add observe mode for runtime drift detection: see observe.md
- Add qualify mode for scenario, stateful, and chaos checks: see qualify.md
Variants
Test the same route with different headers or content types:
await fastify.apophis.contract({
variants: [
{ name: 'json', headers: { accept: 'application/json' } },
{ name: 'xml', headers: { accept: 'application/xml' } }
]
})
Or declare variants in the route schema:
app.get('/users', {
schema: {
'x-variants': [
{ name: 'json', headers: { accept: 'application/json' } }
]
}
})
Plugin Options
When registering the APOPHIS plugin, you can pass these options:
await fastify.register(apophis, {
// Swagger config passthrough (if @fastify/swagger is not already registered)
swagger: { openapi: { info: { title: 'API', version: '1.0.0' } } },
// Runtime contract validation hooks: 'off', 'warn', or 'error'
// Only active in non-production environments
runtime: 'warn',
// Automatically clean up tracked resources after tests
cleanup: true,
// Global timeout in milliseconds for all requests
timeout: 5000,
// Tenant isolation scopes
scopes: {
tenant1: { headers: { 'x-tenant-id': '1' } },
tenant2: { headers: { 'x-tenant-id': '2' } },
},
// Auth and protocol extensions
extensions: [jwtAuth, apiKeyAuth],
// Plugin hook-phase contracts
pluginContracts: {
'rate-limit': { appliesTo: 'POST /users', ensures: ['status != 429'] },
},
// Outbound dependency contracts
outboundContracts: {
'payment-api': {
target: 'https://payments.example.com',
method: 'POST',
response: { 200: { type: 'object', properties: { id: { type: 'string' } } } }
}
}
})
Schema Annotations
APOPHIS reads these OpenAPI schema extensions:
| Annotation | Location | Description |
|---|---|---|
x-category |
Top-level | Route classification: constructor, mutator, observer, destructor, utility |
x-ensures |
Top-level or response[statusCode] |
Post-condition contracts (APOSTL formulas) |
x-requires |
Top-level or response[statusCode] |
Pre-condition contracts (APOSTL formulas) |
x-variants |
Top-level | Request variants for content-type negotiation or feature flags |
x-timeout |
Top-level or response[statusCode] |
Per-route timeout in milliseconds |
x-outbound |
Top-level | Outbound dependency contracts for this route |
x-streaming |
Top-level | Mark route as streaming (populates chunks and streamDurationMs in eval context) |
x-validate-runtime |
Top-level or response[statusCode] |
Toggle runtime validation for this route (default: true) |
x-extension-config |
Top-level | Per-route config for extensions (e.g., { jwt: { verify: false } }) |
Annotations can be placed on the top-level schema or nested inside response[statusCode]. Nested annotations from the first status code schema are merged with top-level annotations.
Programmatic API
After registration, fastify.apophis provides:
// Run contract tests for all routes
const suite = await fastify.apophis.contract({ runs: 50, seed: 42 })
// Run stateful tests
const stateful = await fastify.apophis.stateful({ runs: 50, seed: 42 })
// Run a single scenario
const scenario = await fastify.apophis.scenario({
name: 'oauth-basic',
steps: [...]
})
// Check a single route
const result = await fastify.apophis.check('GET', '/users/:id')
// Get enriched OpenAPI spec with contract metadata
const spec = fastify.apophis.spec()
// Clean up tracked resources
await fastify.apophis.cleanup()
// Test-only utilities (NODE_ENV=test only)
fastify.apophis.test.registerPluginContracts('name', spec)
fastify.apophis.test.registerOutboundContracts({ ... })
fastify.apophis.test.enableOutboundMocks({ mode: 'example' })
fastify.apophis.test.disableOutboundMocks()
const calls = fastify.apophis.test.getOutboundCalls('payment-api')
Outbound mocking patches globalThis.fetch and is process-global. Only one mock runtime can be installed at a time. Run mock-dependent tests serially or isolate by process.
Config Reference
For the full configuration reference, see CLI Reference.
Monorepo Workspaces
Use --workspace to run verify or doctor across all packages:
apophis verify --workspace --profile quick --format json
See CLI Reference for workspace output format and exit codes.