apophis-fastify/docs/qualify.md

Qualify Mode

Run scenario, stateful, and chaos checks against non-production Fastify services.

Qualify extends invariant-driven testing with multi-step protocol flows, stateful sequences, and controlled fault injection.

What Qualify Does

apophis qualify runs deeper testing than verify:

• Scenario execution: Multi-step protocol flows with capture/rebind
• Stateful testing: Constructor/mutator/observer/destructor sequences
• Chaos engineering: Controlled fault injection
• Adversity checks: Failure-path and edge-case validation

When to Use It

Qualify is heavier than verify. Use it where the depth is worth the runtime cost:

Workflow	Recommended	Why
Pull request	No — use verify	verify is fast (<5s for typical services) and catches behavioral regressions per-route. Qualify adds multi-minute scenario/stateful/chaos runs that are too slow for PR feedback loops.
Nightly	Yes	Full scenario, stateful, and chaos execution against staging. Catch protocol-level regressions that single-route verification cannot see.
Pre-release	Yes	Run qualify against the exact artifact that will be promoted to production. Treat a passing qualify run as a release gate for critical flows.
Specialist workflows	Yes	Auth flows, billing sequences, idempotency guarantees, and pagination consistency need multi-step qualification that verify cannot express.
Chaos engineering	Nightly or ad-hoc	Chaos injection increases latency. Run it in dedicated CI slots, not on every commit.

Quick workflow setup

// apophis.config.js — two profiles for different cadences
export default {
  mode: 'qualify',
  profiles: {
    'nightly': {
      name: 'nightly',
      mode: 'qualify',
      preset: 'deep',
      features: ['scenario', 'stateful', 'chaos'],
      routes: [],
    },
    'pre-release': {
      name: 'pre-release',
      mode: 'qualify',
      preset: 'deep',
      features: ['scenario', 'stateful'],
      routes: [],
    },
  },
  presets: {
    deep: { timeout: 15000, chaos: false },
  },
}

Run nightly: apophis qualify --profile nightly Run pre-release: apophis qualify --profile pre-release --format json-summary

For pull requests, use verify instead:

apophis verify --profile ci

Scenario Examples

OAuth Flow

profiles: {
  'oauth-nightly': {
    name: 'oauth-nightly',
    mode: 'qualify',
    preset: 'protocol-lab',
    routes: [],
    seed: 42
  }
}

Run with: apophis qualify --profile oauth-nightly --seed 42

Lifecycle Deep

profiles: {
  'lifecycle-deep': {
    name: 'lifecycle-deep',
    mode: 'qualify',
    preset: 'protocol-lab',
    routes: [],
    seed: 42
  }
}

Scenario Definitions

Scenarios are multi-step flows with capture and rebind:

await fastify.apophis.scenario({
  name: 'oauth-basic',
    steps: [
    {
      name: 'authorize',
      request: { method: 'GET', url: '/oauth/authorize?client_id=web&response_type=code&state=abc123' },
      // Behavioral: state parameter round-trips for CSRF protection
      expect: ['response_payload(this).state == request_query(this).state'],
      capture: { code: 'response_payload(this).code' }
    },
    {
      name: 'token',
      request: {
        method: 'POST',
        url: '/oauth/token',
        form: { grant_type: 'authorization_code', code: '$authorize.code', scope: 'read' }
      },
      // Behavioral: issued token preserves the requested scope
      expect: ['response_payload(this).scope == request_body(this).scope']
    }
  ]
})

Scenario behavior:

1. Cookie jar persists Set-Cookie values across steps.
2. Step-level headers.cookie overrides jar values for that step.
3. form sends application/x-www-form-urlencoded payloads.

Stateful Testing

Stateful tests generate sequences of operations and track resources:

1. Constructor: Create resources (POST)
2. Mutator: Modify resources (PUT, PATCH)
3. Observer: Read resources (GET)
4. Destructor: Remove resources (DELETE)

APOPHIS tracks created resources and runs cleanup after test completion.

Run stateful tests via the API:

const stateful = await fastify.apophis.stateful({ runs: 50, seed: 42 })
console.log('Stateful tests:', stateful.summary)

Route Transparency

Artifacts include executedRoutes and skippedRoutes arrays. skippedRoutes contains reasons such as mode mismatch, environment policy, or route filter exclusion.

Chaos and Adversity

Chaos testing injects controlled failures:

• Delay: Slow responses
• Error: Return error status codes
• Dropout: Connection failures
• Truncate: Truncated response bodies
• Malformed: Invalid JSON or content-type
• Field-corrupt: Random field mutation in response objects

Configure chaos in your preset:

presets: {
  'protocol-lab': {
    name: 'protocol-lab',
    depth: 'deep',
    timeout: 15000,
    parallel: false,
    chaos: true,
    observe: false
  }
}

Non-Prod Boundaries

Qualify mode is gated away from production by default:

Environment	Scenario	Stateful	Chaos
local	enabled	enabled	enabled
test/CI	enabled	enabled	enabled
staging	enabled with allowlist	enabled	blocked on protected routes
production	disabled by default	disabled by default	disabled by default

Machine Output for CI

Qualify can produce large output. Use machine-readable formats and event filtering to keep CI logs manageable:

Concise formats

• --format json-summary — emits a single JSON document with summary, failures, and warnings. Omits per-step traces and cleanup outcomes.
• --format ndjson-summary — emits three NDJSON lines: run.started, run.summary, run.completed. No per-route events.

Filtering examples

# Extract only failed routes from full ndjson
apophis qualify --profile oauth-nightly --format ndjson | jq 'select(.type == "route.failed")'

# Write artifact to disk and parse the file instead of stdout
apophis qualify --profile oauth-nightly --format json --artifact-dir reports/apophis

Recommended CI retention strategy

• Keep artifacts for 30 days in CI storage (S3, GCS, Artifactory).
• Use --artifact-dir to write artifacts automatically.
• Parse json-summary output for dashboards; keep full json artifacts for debugging.

Exit Codes

Code	Meaning
0	All qualifications passed
1	One or more qualifications failed
2	Safety violation or invalid config
3	Internal APOPHIS error
130	Interrupted (SIGINT)

Config Example

// apophis.config.js
export default {
  mode: 'qualify',
  profile: 'oauth-nightly',
  profiles: {
    'oauth-nightly': {
      name: 'oauth-nightly',
      mode: 'qualify',
      preset: 'protocol-lab',
      routes: [],
      seed: 42
    },
    'lifecycle-deep': {
      name: 'lifecycle-deep',
      mode: 'qualify',
      preset: 'protocol-lab',
      routes: [],
      seed: 42
    }
  },
  presets: {
    'protocol-lab': {
      name: 'protocol-lab',
    runs: 200,
      timeout: 15000,
      parallel: false,
      chaos: true,
      observe: false
    }
  },
  environments: {
    local: {
      name: 'local',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: true,
      allowBlocking: true,
      requireSink: false
    },
    test: {
      name: 'test',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: true,
      allowBlocking: true,
      requireSink: false
    },
    staging: {
      name: 'staging',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: false,
      allowBlocking: false,
      requireSink: true
    }
  }
};

Gate Execution Counts

Human output shows per-gate execution counts (scenario, stateful, chaos) so you can verify which gates actually ran.

Custom Scenarios (config-defined)

Define arbitrary multi-step scenarios directly in your apophis.config.js without writing code:

// apophis.config.js
export default {
  mode: 'qualify',
  scenarios: [
    {
      name: 'idempotency-check',
      steps: [
        {
          name: 'create-order',
          request: {
            method: 'POST',
            url: '/orders',
            body: { product: 'widget', quantity: 3 },
          },
          expect: ['status:201', 'response_body(this).id != null'],
          capture: { orderId: 'response_body(this).id' },
        },
        {
          name: 'duplicate-create',
          request: {
            method: 'POST',
            url: '/orders',
            headers: { 'x-idempotency-key': 'dup-001' },
            body: { product: 'widget', quantity: 3 },
          },
          expect: ['status:200', 'response_body(this).id == "$create-order.orderId"'],
        },
      ],
    },
    {
      name: 'pagination-flow',
      steps: [
        {
          name: 'list-page-1',
          request: { method: 'GET', url: '/items?page=1&limit=5' },
          expect: ['status:200', 'response_body(this).items != null'],
          capture: { firstPageCount: 'response_body(this).items.length' },
        },
        {
          name: 'list-page-2',
          request: { method: 'GET', url: '/items?page=2&limit=5' },
          expect: ['status:200'],
        },
      ],
    },
  ],
  profiles: {
    'nightly': {
      name: 'nightly',
      mode: 'qualify',
      preset: 'deep',
      routes: ['POST /orders', 'GET /orders', 'GET /items'],
    },
  },
  presets: {
    deep: { name: 'deep', timeout: 15000, chaos: true },
  },
  environments: {
    local: { name: 'local', allowQualify: true, allowChaos: true },
  },
};

Scenario step fields:

Field	Required	Description
name	yes	Human-readable step label
request.method	yes	HTTP method (GET, POST, PUT, DELETE, PATCH)
request.url	yes	URL path (e.g. /orders, /items?page=1)
request.body	no	JSON request body
request.headers	no	Custom headers (e.g. x-idempotency-key)
expect	yes	APOSTL formulas that must return truthy for step to pass
capture	no	Map of { key: "apostl_formula" } — captured values are substituted via $stepName.key in later steps

Captured values are interpolated in subsequent step URLs, bodies, and headers using $stepName.key syntax.

Chaos Configuration

Fine-tune chaos behavior via preset fields:

presets: {
  'chaos-lab': {
    name: 'chaos-lab',
    timeout: 10000,
    chaos: true,
    chaosStrategy: 'sample',   // 'one' | 'all' | 'sample' | 'routes'
    chaosSampleSize: 5,        // routes to sample when strategy = 'sample'
    chaosSampleRoutes: [       // explicit routes when strategy = 'routes'
      'GET /api/users',
      'POST /api/orders',
    ],
  },
}

Field	Default	Description
chaosStrategy	'one'	Route selection strategy
chaosSampleSize	3	Routes to sample (strategy sample)
chaosSampleRoutes	—	Explicit route list (strategy routes)

Artifact Interpretation

Each qualify run produces an artifact JSON document. Key sections:

executionSummary

{
  "executionSummary": {
    "totalPlanned": 15,
    "totalExecuted": 12,
    "totalPassed": 10,
    "totalFailed": 2,
    "scenariosRun": 3,
    "statefulTestsRun": 5,
    "chaosRunsRun": 4,
    "chaosRoutesPlanned": 2,
    "chaosRoutesExecuted": 2,
    "totalSteps": 12
  }
}

Use totalExecuted vs totalPlanned to see how many checks actually ran (gate gating, route filtering, chaos selection). A non-zero totalPlanned with zero totalExecuted means all gates were disabled or no routes matched.

executedRoutes / skippedRoutes

{
  "executedRoutes": ["POST /orders", "GET /orders/:id", "GET /items"],
  "skippedRoutes": [
    { "route": "DELETE /items/:id", "reason": "No scenario covers this route" },
    { "route": "GET /health", "reason": "Not selected by chaos strategy: one" }
  ]
}

executedRoutes lists every route that had at least one scenario step, stateful command, or chaos injection. skippedRoutes explains why every other discovered route was excluded.

profileGates

{
  "profileGates": {
    "scenario": true,
    "stateful": true,
    "chaos": false
  }
}

Shows which gates were active. Combine with executionSummary per-gate counts to verify each active gate produced results.

stepTraces

Each entry records an individual step execution:

{
  "stepTraces": [
    {
      "step": 0,
      "name": "create-order",
      "route": "POST /orders",
      "durationMs": 12,
      "status": "passed"
    }
  ]
}

Filter by status to isolate failures. Look at durationMs for performance regressions.

failures

{
  "failures": [
    {
      "route": "POST /orders",
      "contract": "status:201",
      "category": "runtime",
      "replayCommand": "apophis replay --artifact reports/apophis/qualify-2026-05-21T...json"
    }
  ]
}

replayCommand gives a copy-pasteable command to re-run the exact same seed with the stored artifact for triage.

Zero-Execution Guardrail

Qualify exits with code 1 if zero checks executed. This prevents silent passes when all routes are filtered out or gates are disabled.

Test Budget

The runs field in your preset controls how many property-based tests execute per route. Default is 50. Lower for faster CI feedback, higher for deeper exploration:

presets: {
  'protocol-lab': {
    runs: 200,
    timeout: 15000
  }
}