apophis-fastify/docs/qualify.md

Qualify Mode

Run scenario, stateful, and chaos checks against non-production Fastify services.

Qualify extends invariant-driven testing with multi-step protocol flows, stateful sequences, and controlled fault injection.

What Qualify Does

apophis qualify runs deeper testing than verify:

• Scenario execution: Multi-step protocol flows with capture/rebind
• Stateful testing: Constructor/mutator/observer/destructor sequences
• Chaos engineering: Controlled fault injection
• Adversity checks: Failure-path and edge-case validation

When to Use It

Qualify is heavier than verify. Use it where the depth is worth the runtime cost:

Workflow	Recommended	Why
Pull request	No — use verify	verify is fast (<5s for typical services) and catches behavioral regressions per-route. Qualify adds multi-minute scenario/stateful/chaos runs that are too slow for PR feedback loops.
Nightly	Yes	Full scenario, stateful, and chaos execution against staging. Catch protocol-level regressions that single-route verification cannot see.
Pre-release	Yes	Run qualify against the exact artifact that will be promoted to production. Treat a passing qualify run as a release gate for critical flows.
Specialist workflows	Yes	Auth flows, billing sequences, idempotency guarantees, and pagination consistency need multi-step qualification that verify cannot express.
Chaos engineering	Nightly or ad-hoc	Chaos injection increases latency. Run it in dedicated CI slots, not on every commit.

Quick workflow setup

// apophis.config.js — two profiles for different cadences
export default {
  mode: 'qualify',
  profiles: {
    'nightly': {
      name: 'nightly',
      mode: 'qualify',
      preset: 'deep',
      features: ['scenario', 'stateful', 'chaos'],
      routes: [],
    },
    'pre-release': {
      name: 'pre-release',
      mode: 'qualify',
      preset: 'deep',
      features: ['scenario', 'stateful'],
      routes: [],
    },
  },
  presets: {
    deep: { timeout: 15000, chaos: false },
  },
}

Run nightly: apophis qualify --profile nightly Run pre-release: apophis qualify --profile pre-release --format json-summary

For pull requests, use verify instead:

apophis verify --profile ci

Scenario Examples

OAuth Flow

profiles: {
  'oauth-nightly': {
    name: 'oauth-nightly',
    mode: 'qualify',
    preset: 'protocol-lab',
    routes: [],
    seed: 42
  }
}

Run with: apophis qualify --profile oauth-nightly --seed 42

Lifecycle Deep

profiles: {
  'lifecycle-deep': {
    name: 'lifecycle-deep',
    mode: 'qualify',
    preset: 'protocol-lab',
    routes: [],
    seed: 42
  }
}

Scenario Definitions

Scenarios are multi-step flows with capture and rebind:

await fastify.apophis.scenario({
  name: 'oauth-basic',
    steps: [
    {
      name: 'authorize',
      request: { method: 'GET', url: '/oauth/authorize?client_id=web&response_type=code&state=abc123' },
      // Behavioral: state parameter round-trips for CSRF protection
      expect: ['response_payload(this).state == request_query(this).state'],
      capture: { code: 'response_payload(this).code' }
    },
    {
      name: 'token',
      request: {
        method: 'POST',
        url: '/oauth/token',
        form: { grant_type: 'authorization_code', code: '$authorize.code', scope: 'read' }
      },
      // Behavioral: issued token preserves the requested scope
      expect: ['response_payload(this).scope == request_body(this).scope']
    }
  ]
})

Scenario behavior:

1. Cookie jar persists Set-Cookie values across steps.
2. Step-level headers.cookie overrides jar values for that step.
3. form sends application/x-www-form-urlencoded payloads.

Stateful Testing

Stateful tests generate sequences of operations and track resources:

1. Constructor: Create resources (POST)
2. Mutator: Modify resources (PUT, PATCH)
3. Observer: Read resources (GET)
4. Destructor: Remove resources (DELETE)

APOPHIS tracks created resources and runs cleanup after test completion.

Run stateful tests via the API:

const stateful = await fastify.apophis.stateful({ runs: 50, seed: 42 })
console.log('Stateful tests:', stateful.summary)

Route Transparency

Artifacts include executedRoutes and skippedRoutes arrays. skippedRoutes contains reasons such as mode mismatch, environment policy, or route filter exclusion.

Chaos and Adversity

Chaos testing injects controlled failures:

• Delay: Slow responses
• Error: Return error status codes
• Dropout: Connection failures
• Truncate: Truncated response bodies
• Malformed: Invalid JSON or content-type
• Field-corrupt: Random field mutation in response objects

Configure chaos in your preset:

presets: {
  'protocol-lab': {
    name: 'protocol-lab',
    depth: 'deep',
    timeout: 15000,
    parallel: false,
    chaos: true,
    observe: false
  }
}

Non-Prod Boundaries

Qualify mode is gated away from production by default:

Environment	Scenario	Stateful	Chaos
local	enabled	enabled	enabled
test/CI	enabled	enabled	enabled
staging	enabled (blockQualify: false)	enabled (blockQualify: false)	blocked on protected routes (allowChaosOnProtected: false)

Qualify gates are not individually gated per environment. The blockQualify flag controls all qualify execution, and allowChaosOnProtected controls chaos on protected routes. | production | disabled by default | disabled by default | disabled by default |

Machine Output for CI

Qualify can produce large output. Use machine-readable formats and event filtering to keep CI logs manageable:

Concise formats

• --format json-summary — emits a single JSON document with summary, failures, and warnings. Omits per-step traces and cleanup outcomes.
• --format ndjson-summary — emits three NDJSON lines: run.started, run.summary, run.completed. No per-route events.

Filtering examples

# Extract only failed routes from full ndjson
apophis qualify --profile oauth-nightly --format ndjson | jq 'select(.type == "route.failed")'

# Write artifact to disk and parse the file instead of stdout
apophis qualify --profile oauth-nightly --format json --artifact-dir reports/apophis

Recommended CI retention strategy

• Keep artifacts for 30 days in CI storage (S3, GCS, Artifactory).
• Use --artifact-dir to write artifacts automatically.
• Parse json-summary output for dashboards; keep full json artifacts for debugging.

Exit Codes

Code	Meaning
0	All qualifications passed
1	One or more qualifications failed
2	Safety violation or invalid config
3	Internal APOPHIS error
130	Interrupted (SIGINT)

Config Example

// apophis.config.js
export default {
  mode: 'qualify',
  profile: 'oauth-nightly',
  profiles: {
    'oauth-nightly': {
      name: 'oauth-nightly',
      mode: 'qualify',
      preset: 'protocol-lab',
      routes: [],
      seed: 42
    },
    'lifecycle-deep': {
      name: 'lifecycle-deep',
      mode: 'qualify',
      preset: 'protocol-lab',
      routes: [],
      seed: 42
    }
  },
  presets: {
    'protocol-lab': {
      name: 'protocol-lab',
    runs: 200,
      timeout: 15000,
      parallel: false,
      chaos: true,
      observe: false
    }
  },
  environments: {
    local: {
      name: 'local',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: true,
      allowBlocking: true,
      requireSink: false
    },
    test: {
      name: 'test',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: true,
      allowBlocking: true,
      requireSink: false
    },
    staging: {
      name: 'staging',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: false,
      allowBlocking: false,
      requireSink: true
    }
  }
};

Gate Execution Counts

Human output shows per-gate execution counts (scenario, stateful, chaos) so you can verify which gates actually ran.

Custom Scenarios (config-defined)

Define arbitrary multi-step scenarios directly in your apophis.config.js without writing code:

// apophis.config.js
export default {
  mode: 'qualify',
  scenarios: [
    {
      name: 'idempotency-check',
      steps: [
        {
          name: 'create-order',
          request: {
            method: 'POST',
            url: '/orders',
            body: { product: 'widget', quantity: 3 },
          },
          expect: ['status:201', 'response_body(this).id != null'],
          capture: { orderId: 'response_body(this).id' },
        },
        {
          name: 'duplicate-create',
          request: {
            method: 'POST',
            url: '/orders',
            headers: { 'x-idempotency-key': 'dup-001' },
            body: { product: 'widget', quantity: 3 },
          },
          expect: ['status:200', 'response_body(this).id == "$create-order.orderId"'],
        },
      ],
    },
    {
      name: 'pagination-flow',
      steps: [
        {
          name: 'list-page-1',
          request: { method: 'GET', url: '/items?page=1&limit=5' },
          expect: ['status:200', 'response_body(this).items != null'],
          capture: { firstPageCount: 'response_body(this).items.length' },
        },
        {
          name: 'list-page-2',
          request: { method: 'GET', url: '/items?page=2&limit=5' },
          expect: ['status:200'],
        },
      ],
    },
  ],
  profiles: {
    'nightly': {
      name: 'nightly',
      mode: 'qualify',
      preset: 'deep',
      routes: ['POST /orders', 'GET /orders', 'GET /items'],
    },
  },
  presets: {
    deep: { name: 'deep', timeout: 15000, chaos: true },
  },
  environments: {
    local: { name: 'local', allowQualify: true, allowChaos: true },
  },
};

Scenario step fields:

Field	Required	Description
name	yes	Human-readable step label
request.method	yes	HTTP method (GET, POST, PUT, DELETE, PATCH)
request.url	yes	URL path (e.g. /orders, /items?page=1)
request.body	no	JSON request body
request.headers	no	Custom headers (e.g. x-idempotency-key)
expect	yes	APOSTL formulas that must return truthy for step to pass
capture	no	Map of { key: "apostl_formula" } — captured values are substituted via $stepName.key in later steps

Captured values are interpolated in subsequent step URLs, bodies, and headers using $stepName.key syntax.

Chaos Configuration

Fine-tune chaos behavior via preset fields:

presets: {
  'chaos-lab': {
    name: 'chaos-lab',
    timeout: 10000,
    chaos: true,
    chaosStrategy: 'sample',   // 'one' | 'all' | 'sample' | 'routes'
    chaosSampleSize: 5,        // routes to sample when strategy = 'sample'
    chaosSampleRoutes: [       // explicit routes when strategy = 'routes'
      'GET /api/users',
      'POST /api/orders',
    ],
  },
}

Field	Default	Description
chaosStrategy	'one'	Route selection strategy
chaosSampleSize	3	Routes to sample (strategy sample)
chaosSampleRoutes	—	Explicit route list (strategy routes)

Artifact Interpretation

Each qualify run produces an artifact JSON document. Key sections:

executionSummary

{
  "executionSummary": {
    "totalPlanned": 15,
    "totalExecuted": 12,
    "totalPassed": 10,
    "totalFailed": 2,
    "scenariosRun": 3,
    "statefulTestsRun": 5,
    "chaosRunsRun": 4,
    "chaosRoutesPlanned": 2,
    "chaosRoutesExecuted": 2,
    "totalSteps": 12
  }
}

Use totalExecuted vs totalPlanned to see how many checks actually ran (gate gating, route filtering, chaos selection). A non-zero totalPlanned with zero totalExecuted means all gates were disabled or no routes matched.

executedRoutes / skippedRoutes

{
  "executedRoutes": ["POST /orders", "GET /orders/:id", "GET /items"],
  "skippedRoutes": [
    { "route": "DELETE /items/:id", "reason": "No scenario covers this route" },
    { "route": "GET /health", "reason": "Not selected by chaos strategy: one" }
  ]
}

executedRoutes lists every route that had at least one scenario step, stateful command, or chaos injection. skippedRoutes explains why every other discovered route was excluded.

profileGates

{
  "profileGates": {
    "scenario": true,
    "stateful": true,
    "chaos": false
  }
}

Shows which gates were active. Combine with executionSummary per-gate counts to verify each active gate produced results.

stepTraces

Each entry records an individual step execution:

{
  "stepTraces": [
    {
      "step": 0,
      "name": "create-order",
      "route": "POST /orders",
      "durationMs": 12,
      "status": "passed"
    }
  ]
}

Filter by status to isolate failures. Look at durationMs for performance regressions.

failures

{
  "failures": [
    {
      "route": "POST /orders",
      "contract": "status:201",
      "category": "runtime",
      "replayCommand": "apophis replay --artifact reports/apophis/qualify-2026-05-21T...json"
    }
  ]
}

replayCommand gives a copy-pasteable command to re-run the exact same seed with the stored artifact for triage.

Zero-Execution Guardrail

Qualify exits with code 1 if zero checks executed. This prevents silent passes when all routes are filtered out or gates are disabled.

Test Budget

The runs field in your preset controls how many property-based tests execute per route. Default is 50. Lower for faster CI feedback, higher for deeper exploration:

presets: {
  'protocol-lab': {
    runs: 200,
    timeout: 15000
  }
}