Files
apophis-fastify/docs/getting-started.md
T
John Dvorak d0523fcc2d fix: harden engine, enrich failure diagnostics, close adoption gaps
- P0: CLI verify now honors  test budget with seeded multi-sample
- P0: Observe sampling enforced via Math.random() gate in hook-validator
- P1: Remove misleading undici-mock-agent isolation option
- P1: Qualify reuses shared discoverRouteDetails() with warnings
- P1: Chaos/scenario config exposed via preset schema
- P1: README/docs limitations updated to current state
- P2: Nested response annotations prefer 2xx deterministically
- P2: --changed documented as heuristic in verify.md

- Add observe sink tests (sampling 0/1, sink failure non-interference)
- Add verify runs regression tests (scale, determinism, variants)
- Add configured-scenario qualify test (independent of OAuth fixture)
- Add coverageBreakdown to qualify artifacts (per-gate route coverage)
- Add production-style observe example with real sink in docs/observe.md
- Add nightly/staging vs PR gating guidance to docs/qualify.md

- Enrich VerifyFailure with formula-aware diagnostics:
  status:201 => 'HTTP 200', body field checks => actual values
- Remove stale observe CLI activation message
- Document outbound mocks as process-global in getting-started.md
- Refresh APOPHIS_ADOPTION_AUDIT.md with current state

903 tests pass, build clean, typecheck clean.
2026-05-21 20:39:36 -07:00

9.1 KiB

Getting Started with APOPHIS

Get from install to your first behavioral bug in 10 minutes.

APOPHIS is inspired by the concept of invariant-driven automated testing: instead of only validating request and response shape, encode intended behavior as executable contracts and let the tool find violations automatically.

Prerequisites

  • Node.js 20.x or 22.x
  • Fastify v5 (v4 is not supported)
  • ESM project ("type": "module" in package.json)
  • A Fastify app with @fastify/swagger registered

Step 1: Install

npm install @apophis/fastify fastify @fastify/swagger

Step 2: Scaffold

npx apophis init --preset safe-ci

This creates:

  • apophis.config.js — config with a quick profile
  • APOPHIS.md — preset-specific guidance
  • Package script: npm run apophis:verify

Step 3: Add One Behavioral Contract

Pick one important route. Add an x-ensures clause that checks behavior across operations:

import crypto from 'crypto';

app.post('/users', {
  schema: {
    'x-category': 'constructor',
    'x-ensures': [
      // BEHAVIORAL: Creating a user must make it retrievable
      'response_code(GET /users/{response_body(this).id}) == 200'
    ]
  }
}, async (request, reply) => {
  const { name } = request.body;
  const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
  reply.status(201);
  return { id, name };
});

Warning: Using Date.now() or Math.random() in handlers breaks determinism and replay. Use a stable function of the input instead. APOPHIS does not proactively detect nondeterministic handlers; it warns only when a replay diverges from the original run.

Step 4: Run Verify

npx apophis verify --profile quick --routes "POST /users"

Example Failure

If your GET /users/:id handler has a bug (always returns 404), APOPHIS catches it:

Contract violation
POST /users
Profile: quick
Seed: 42

Expected
  response_code(GET /users/{response_body(this).id}) == 200

Observed
  GET /users/usr-7d865e returned 404

Why this matters
  The resource created by POST /users is not retrievable.

Replay
  apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json

Next
  Check the create/read consistency for POST /users and GET /users/{id}.

Step 5: Replay and Fix

Copy the replay command and run it:

apophis replay --artifact reports/apophis/failure-2026-04-28T12-30-22Z.json

Fix the bug in your handler. Re-run verify. The failure should now pass.

Behavioral vs Structural Contracts

APOPHIS contracts should verify behavior, not structure. Fastify and @fastify/swagger already enforce status codes, required fields, and types. Behavioral contracts catch what schemas cannot:

Structural (avoid) Behavioral (prefer)
status:200 response_body(this) == request_body(this)
response_body(this).id != null response_code(GET /users/{response_body(this).id}) == 200
response_body(this).name != null response_body(GET /users/{id}).name == previous(response_body(this).name)

Good behavioral patterns (from the paper):

  • Constructor precondition: Resource must not exist before creation
    response_code(GET /users/{request_body(this).email}) == 404
    
  • Round-trip equality: POST response matches the request body
    response_body(this) == request_body(this)
    
  • Cross-route retrievability: Creating a resource makes it readable via GET
    response_code(GET /users/{response_body(this).id}) == 200
    
  • State-change verification: DELETE causes subsequent GET to return 404
    response_code(GET /users/{request_params(this).id}) == 404
    
  • Previous state preservation: DELETE returns the last known state
    response_body(this) == previous(response_body(GET /users/{request_params(this).id}))
    
  • Invariant over collections: All resources satisfy a cross-resource constraint
    for t in response_body(GET /tournaments) :-
      response_body(GET /tournaments/{t.id}/players).length <= t.capacity
    

Anti-patterns to avoid:

  • Checking status codes (handled by schema validation)
  • Checking field existence (handled by schema validation)
  • Checking field types (handled by schema validation)

Next Steps

  • Add more routes to your profile: apophis verify --profile quick --routes "POST /users,PUT /users/:id"
  • Use wildcards to match route patterns: apophis verify --routes 'POST /api/*'
  • Run all routes: apophis verify --profile quick
  • Run only changed routes in CI: apophis verify --profile ci --changed
    • Requires a git repository.
  • Use machine-readable output in CI: apophis verify --profile ci --format json-summary
  • Add observe mode for runtime drift detection: see observe.md
  • Add qualify mode for scenario, stateful, and chaos checks: see qualify.md

Variants

Test the same route with different headers or content types:

await fastify.apophis.contract({
  variants: [
    { name: 'json', headers: { accept: 'application/json' } },
    { name: 'xml', headers: { accept: 'application/xml' } }
  ]
})

Or declare variants in the route schema:

app.get('/users', {
  schema: {
    'x-variants': [
      { name: 'json', headers: { accept: 'application/json' } }
    ]
  }
})

Plugin Options

When registering the APOPHIS plugin, you can pass these options:

await fastify.register(apophis, {
  // Swagger config passthrough (if @fastify/swagger is not already registered)
  swagger: { openapi: { info: { title: 'API', version: '1.0.0' } } },

  // Runtime contract validation hooks: 'off', 'warn', or 'error'
  // Only active in non-production environments
  runtime: 'warn',

  // Automatically clean up tracked resources after tests
  cleanup: true,

  // Global timeout in milliseconds for all requests
  timeout: 5000,

  // Tenant isolation scopes
  scopes: {
    tenant1: { headers: { 'x-tenant-id': '1' } },
    tenant2: { headers: { 'x-tenant-id': '2' } },
  },

  // Auth and protocol extensions
  extensions: [jwtAuth, apiKeyAuth],

  // Plugin hook-phase contracts
  pluginContracts: {
    'rate-limit': { appliesTo: 'POST /users', ensures: ['status != 429'] },
  },

  // Outbound dependency contracts
  outboundContracts: {
    'payment-api': {
      target: 'https://payments.example.com',
      method: 'POST',
      response: { 200: { type: 'object', properties: { id: { type: 'string' } } } }
    }
  }
})

Schema Annotations

APOPHIS reads these OpenAPI schema extensions:

Annotation Location Description
x-category Top-level Route classification: constructor, mutator, observer, destructor, utility
x-ensures Top-level or response[statusCode] Post-condition contracts (APOSTL formulas)
x-requires Top-level or response[statusCode] Pre-condition contracts (APOSTL formulas)
x-variants Top-level Request variants for content-type negotiation or feature flags
x-timeout Top-level or response[statusCode] Per-route timeout in milliseconds
x-outbound Top-level Outbound dependency contracts for this route
x-streaming Top-level Mark route as streaming (populates chunks and streamDurationMs in eval context)
x-validate-runtime Top-level or response[statusCode] Toggle runtime validation for this route (default: true)
x-extension-config Top-level Per-route config for extensions (e.g., { jwt: { verify: false } })

Annotations can be placed on the top-level schema or nested inside response[statusCode]. Nested annotations from the first status code schema are merged with top-level annotations.

Programmatic API

After registration, fastify.apophis provides:

// Run contract tests for all routes
const suite = await fastify.apophis.contract({ runs: 50, seed: 42 })

// Run stateful tests
const stateful = await fastify.apophis.stateful({ runs: 50, seed: 42 })

// Run a single scenario
const scenario = await fastify.apophis.scenario({
  name: 'oauth-basic',
  steps: [...]
})

// Check a single route
const result = await fastify.apophis.check('GET', '/users/:id')

// Get enriched OpenAPI spec with contract metadata
const spec = fastify.apophis.spec()

// Clean up tracked resources
await fastify.apophis.cleanup()

// Test-only utilities (NODE_ENV=test only)
fastify.apophis.test.registerPluginContracts('name', spec)
fastify.apophis.test.registerOutboundContracts({ ... })
fastify.apophis.test.enableOutboundMocks({ mode: 'example' })
fastify.apophis.test.disableOutboundMocks()
const calls = fastify.apophis.test.getOutboundCalls('payment-api')

Outbound mocking patches globalThis.fetch and is process-global. Only one mock runtime can be installed at a time. Run mock-dependent tests serially or isolate by process.

Config Reference

For the full configuration reference, see CLI Reference.

Monorepo Workspaces

Use --workspace to run verify or doctor across all packages:

apophis verify --workspace --profile quick --format json

See CLI Reference for workspace output format and exit codes.