apophis-fastify/docs/qualify.md

# Qualify Mode

Run scenario, stateful, and chaos checks against non-production Fastify services.

Qualify extends invariant-driven testing with multi-step protocol flows, stateful sequences, and controlled fault injection.

## What Qualify Does

`apophis qualify` runs deeper testing than verify:

- **Scenario execution**: Multi-step protocol flows with capture/rebind
- **Stateful testing**: Constructor/mutator/observer/destructor sequences
- **Chaos engineering**: Controlled fault injection
- **Adversity checks**: Failure-path and edge-case validation

## When to Use It

Qualify is heavier than verify. Use it where the depth is worth the runtime cost:

| Workflow | Recommended | Why |
|---|---|---|
| **Pull request** | No — use `verify` | `verify` is fast (<5s for typical services) and catches behavioral regressions per-route. Qualify adds multi-minute scenario/stateful/chaos runs that are too slow for PR feedback loops. |
| **Nightly** | Yes | Full scenario, stateful, and chaos execution against staging. Catch protocol-level regressions that single-route verification cannot see. |
| **Pre-release** | Yes | Run qualify against the exact artifact that will be promoted to production. Treat a passing qualify run as a release gate for critical flows. |
| **Specialist workflows** | Yes | Auth flows, billing sequences, idempotency guarantees, and pagination consistency need multi-step qualification that verify cannot express. |
| **Chaos engineering** | Nightly or ad-hoc | Chaos injection increases latency. Run it in dedicated CI slots, not on every commit. |

### Quick workflow setup

```javascript
// apophis.config.js — two profiles for different cadences
export default {
  mode: 'qualify',
  profiles: {
    'nightly': {
      name: 'nightly',
      mode: 'qualify',
      preset: 'deep',
      features: ['scenario', 'stateful', 'chaos'],
      routes: [],
    },
    'pre-release': {
      name: 'pre-release',
      mode: 'qualify',
      preset: 'deep',
      features: ['scenario', 'stateful'],
      routes: [],
    },
  },
  presets: {
    deep: { timeout: 15000, chaos: false },
  },
}
```

Run nightly: `apophis qualify --profile nightly`
Run pre-release: `apophis qualify --profile pre-release --format json-summary`

For pull requests, use verify instead:
```bash
apophis verify --profile ci
```

## Scenario Examples

### OAuth Flow

```javascript
profiles: {
  'oauth-nightly': {
    name: 'oauth-nightly',
    mode: 'qualify',
    preset: 'protocol-lab',
    routes: [],
    seed: 42
  }
}
```

Run with: `apophis qualify --profile oauth-nightly --seed 42`

### Lifecycle Deep

```javascript
profiles: {
  'lifecycle-deep': {
    name: 'lifecycle-deep',
    mode: 'qualify',
    preset: 'protocol-lab',
    routes: [],
    seed: 42
  }
}
```

## Scenario Definitions

Scenarios are multi-step flows with capture and rebind:

```javascript
await fastify.apophis.scenario({
  name: 'oauth-basic',
    steps: [
    {
      name: 'authorize',
      request: { method: 'GET', url: '/oauth/authorize?client_id=web&response_type=code&state=abc123' },
      // Behavioral: state parameter round-trips for CSRF protection
      expect: ['response_payload(this).state == request_query(this).state'],
      capture: { code: 'response_payload(this).code' }
    },
    {
      name: 'token',
      request: {
        method: 'POST',
        url: '/oauth/token',
        form: { grant_type: 'authorization_code', code: '$authorize.code', scope: 'read' }
      },
      // Behavioral: issued token preserves the requested scope
      expect: ['response_payload(this).scope == request_body(this).scope']
    }
  ]
})
```

Scenario behavior:
1. Cookie jar persists `Set-Cookie` values across steps.
2. Step-level `headers.cookie` overrides jar values for that step.
3. `form` sends `application/x-www-form-urlencoded` payloads.

## Stateful Testing

Stateful tests generate sequences of operations and track resources:

1. **Constructor**: Create resources (POST)
2. **Mutator**: Modify resources (PUT, PATCH)
3. **Observer**: Read resources (GET)
4. **Destructor**: Remove resources (DELETE)

APOPHIS tracks created resources and runs cleanup after test completion.

Run stateful tests via the API:

```javascript
const stateful = await fastify.apophis.stateful({ runs: 50, seed: 42 })
console.log('Stateful tests:', stateful.summary)
```

## Route Transparency

Artifacts include `executedRoutes` and `skippedRoutes` arrays. `skippedRoutes` contains reasons such as mode mismatch, environment policy, or route filter exclusion.

## Chaos and Adversity

Chaos testing injects controlled failures:

- **Delay**: Slow responses
- **Error**: Return error status codes
- **Dropout**: Connection failures
- **Truncate**: Truncated response bodies
- **Malformed**: Invalid JSON or content-type
- **Field-corrupt**: Random field mutation in response objects

Configure chaos in your preset:

```javascript
presets: {
  'protocol-lab': {
    name: 'protocol-lab',
    depth: 'deep',
    timeout: 15000,
    parallel: false,
    chaos: true,
    observe: false
  }
}
```

## Non-Prod Boundaries

Qualify mode is gated away from production by default:

| Environment | Scenario | Stateful | Chaos |
|---|---|---|---|
| local | enabled | enabled | enabled |
| test/CI | enabled | enabled | enabled |
| staging | enabled with allowlist | enabled | blocked on protected routes |
| production | disabled by default | disabled by default | disabled by default |

## Machine Output for CI

Qualify can produce large output. Use machine-readable formats and event filtering to keep CI logs manageable:

### Concise formats

- `--format json-summary` — emits a single JSON document with summary, failures, and warnings. Omits per-step traces and cleanup outcomes.
- `--format ndjson-summary` — emits three NDJSON lines: `run.started`, `run.summary`, `run.completed`. No per-route events.

### Filtering examples

```bash
# Extract only failed routes from full ndjson
apophis qualify --profile oauth-nightly --format ndjson | jq 'select(.type == "route.failed")'

# Write artifact to disk and parse the file instead of stdout
apophis qualify --profile oauth-nightly --format json --artifact-dir reports/apophis
```

### Recommended CI retention strategy

- Keep artifacts for 30 days in CI storage (S3, GCS, Artifactory).
- Use `--artifact-dir` to write artifacts automatically.
- Parse `json-summary` output for dashboards; keep full `json` artifacts for debugging.

## Exit Codes

| Code | Meaning |
|---|---|
| 0 | All qualifications passed |
| 1 | One or more qualifications failed |
| 2 | Safety violation or invalid config |
| 3 | Internal APOPHIS error |
| 130 | Interrupted (SIGINT) |

## Config Example

```javascript
// apophis.config.js
export default {
  mode: 'qualify',
  profile: 'oauth-nightly',
  profiles: {
    'oauth-nightly': {
      name: 'oauth-nightly',
      mode: 'qualify',
      preset: 'protocol-lab',
      routes: [],
      seed: 42
    },
    'lifecycle-deep': {
      name: 'lifecycle-deep',
      mode: 'qualify',
      preset: 'protocol-lab',
      routes: [],
      seed: 42
    }
  },
  presets: {
    'protocol-lab': {
      name: 'protocol-lab',
    runs: 200,
      timeout: 15000,
      parallel: false,
      chaos: true,
      observe: false
    }
  },
  environments: {
    local: {
      name: 'local',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: true,
      allowBlocking: true,
      requireSink: false
    },
    test: {
      name: 'test',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: true,
      allowBlocking: true,
      requireSink: false
    },
    staging: {
      name: 'staging',
      allowVerify: true,
      allowObserve: true,
      allowQualify: true,
      allowChaos: false,
      allowBlocking: false,
      requireSink: true
    }
  }
};
```

## Gate Execution Counts

Human output shows per-gate execution counts (scenario, stateful, chaos) so you can verify which gates actually ran.

## Custom Scenarios (config-defined)

Define arbitrary multi-step scenarios directly in your `apophis.config.js` without writing code:

```javascript
// apophis.config.js
export default {
  mode: 'qualify',
  scenarios: [
    {
      name: 'idempotency-check',
      steps: [
        {
          name: 'create-order',
          request: {
            method: 'POST',
            url: '/orders',
            body: { product: 'widget', quantity: 3 },
          },
          expect: ['status:201', 'response_body(this).id != null'],
          capture: { orderId: 'response_body(this).id' },
        },
        {
          name: 'duplicate-create',
          request: {
            method: 'POST',
            url: '/orders',
            headers: { 'x-idempotency-key': 'dup-001' },
            body: { product: 'widget', quantity: 3 },
          },
          expect: ['status:200', 'response_body(this).id == "$create-order.orderId"'],
        },
      ],
    },
    {
      name: 'pagination-flow',
      steps: [
        {
          name: 'list-page-1',
          request: { method: 'GET', url: '/items?page=1&limit=5' },
          expect: ['status:200', 'response_body(this).items != null'],
          capture: { firstPageCount: 'response_body(this).items.length' },
        },
        {
          name: 'list-page-2',
          request: { method: 'GET', url: '/items?page=2&limit=5' },
          expect: ['status:200'],
        },
      ],
    },
  ],
  profiles: {
    'nightly': {
      name: 'nightly',
      mode: 'qualify',
      preset: 'deep',
      routes: ['POST /orders', 'GET /orders', 'GET /items'],
    },
  },
  presets: {
    deep: { name: 'deep', timeout: 15000, chaos: true },
  },
  environments: {
    local: { name: 'local', allowQualify: true, allowChaos: true },
  },
};
```

Scenario step fields:

| Field | Required | Description |
|---|---|---|
| `name` | yes | Human-readable step label |
| `request.method` | yes | HTTP method (GET, POST, PUT, DELETE, PATCH) |
| `request.url` | yes | URL path (e.g. `/orders`, `/items?page=1`) |
| `request.body` | no | JSON request body |
| `request.headers` | no | Custom headers (e.g. `x-idempotency-key`) |
| `expect` | yes | APOSTL formulas that must return truthy for step to pass |
| `capture` | no | Map of `{ key: "apostl_formula" }` — captured values are substituted via `$stepName.key` in later steps |

Captured values are interpolated in subsequent step URLs, bodies, and headers using `$stepName.key` syntax.

## Chaos Configuration

Fine-tune chaos behavior via preset fields:

```javascript
presets: {
  'chaos-lab': {
    name: 'chaos-lab',
    timeout: 10000,
    chaos: true,
    chaosStrategy: 'sample',   // 'one' | 'all' | 'sample' | 'routes'
    chaosSampleSize: 5,        // routes to sample when strategy = 'sample'
    chaosSampleRoutes: [       // explicit routes when strategy = 'routes'
      'GET /api/users',
      'POST /api/orders',
    ],
  },
}
```

| Field | Default | Description |
|---|---|---|
| `chaosStrategy` | `'one'` | Route selection strategy |
| `chaosSampleSize` | `3` | Routes to sample (strategy `sample`) |
| `chaosSampleRoutes` | — | Explicit route list (strategy `routes`) |

## Artifact Interpretation

Each qualify run produces an artifact JSON document. Key sections:

### executionSummary

```json
{
  "executionSummary": {
    "totalPlanned": 15,
    "totalExecuted": 12,
    "totalPassed": 10,
    "totalFailed": 2,
    "scenariosRun": 3,
    "statefulTestsRun": 5,
    "chaosRunsRun": 4,
    "chaosRoutesPlanned": 2,
    "chaosRoutesExecuted": 2,
    "totalSteps": 12
  }
}
```

Use `totalExecuted` vs `totalPlanned` to see how many checks actually ran (gate gating, route filtering, chaos selection). A non-zero `totalPlanned` with zero `totalExecuted` means all gates were disabled or no routes matched.

### executedRoutes / skippedRoutes

```json
{
  "executedRoutes": ["POST /orders", "GET /orders/:id", "GET /items"],
  "skippedRoutes": [
    { "route": "DELETE /items/:id", "reason": "No scenario covers this route" },
    { "route": "GET /health", "reason": "Not selected by chaos strategy: one" }
  ]
}
```

`executedRoutes` lists every route that had at least one scenario step, stateful command, or chaos injection. `skippedRoutes` explains why every other discovered route was excluded.

### profileGates

```json
{
  "profileGates": {
    "scenario": true,
    "stateful": true,
    "chaos": false
  }
}
```

Shows which gates were active. Combine with `executionSummary` per-gate counts to verify each active gate produced results.

### stepTraces

Each entry records an individual step execution:

```json
{
  "stepTraces": [
    {
      "step": 0,
      "name": "create-order",
      "route": "POST /orders",
      "durationMs": 12,
      "status": "passed"
    }
  ]
}
```

Filter by `status` to isolate failures. Look at `durationMs` for performance regressions.

### failures

```json
{
  "failures": [
    {
      "route": "POST /orders",
      "contract": "status:201",
      "category": "runtime",
      "replayCommand": "apophis replay --artifact reports/apophis/qualify-2026-05-21T...json"
    }
  ]
}
```

`replayCommand` gives a copy-pasteable command to re-run the exact same seed with the stored artifact for triage.

## Zero-Execution Guardrail

Qualify exits with code 1 if zero checks executed. This prevents silent passes when all routes are filtered out or gates are disabled.

## Test Budget

The `runs` field in your preset controls how many property-based tests execute per route. Default is 50. Lower for faster CI feedback, higher for deeper exploration:

```javascript
presets: {
  'protocol-lab': {
    runs: 200,
    timeout: 15000
  }
}
```