- P0: CLI verify now honors test budget with seeded multi-sample - P0: Observe sampling enforced via Math.random() gate in hook-validator - P1: Remove misleading undici-mock-agent isolation option - P1: Qualify reuses shared discoverRouteDetails() with warnings - P1: Chaos/scenario config exposed via preset schema - P1: README/docs limitations updated to current state - P2: Nested response annotations prefer 2xx deterministically - P2: --changed documented as heuristic in verify.md - Add observe sink tests (sampling 0/1, sink failure non-interference) - Add verify runs regression tests (scale, determinism, variants) - Add configured-scenario qualify test (independent of OAuth fixture) - Add coverageBreakdown to qualify artifacts (per-gate route coverage) - Add production-style observe example with real sink in docs/observe.md - Add nightly/staging vs PR gating guidance to docs/qualify.md - Enrich VerifyFailure with formula-aware diagnostics: status:201 => 'HTTP 200', body field checks => actual values - Remove stale observe CLI activation message - Document outbound mocks as process-global in getting-started.md - Refresh APOPHIS_ADOPTION_AUDIT.md with current state 903 tests pass, build clean, typecheck clean.
4.6 KiB
LLM-Safe Adoption
APOPHIS is designed to be safe and predictable for LLM-generated Fastify services.
It applies an invariant-driven approach to LLM-assisted development: constrained vocabulary, deterministic replay, and executable contracts give coding agents a verifiable loop between generated changes and behavioral correctness.
Why APOPHIS Is Good for LLM-Generated Services
Coding agents benefit from:
- Constrained vocabulary: Small set of CLI commands and config options
- Official scaffolds: Tested templates that produce valid config
- Policy guards: CI catches unsafe modes and malformed setup
- Deterministic output: Fixed seed, config, schemas, and deterministic handlers produce repeatable output
- Behavioral contracts: Agents write
x-ensuresclauses, APOPHIS verifies them
Official Scaffolds
Use apophis init with a preset:
| Preset | Use Case |
|---|---|
safe-ci |
Minimal CI-safe preset (default) |
llm-safe |
Minimal preset for LLM-generated codebases |
platform-observe |
Production-ready with observe mode |
protocol-lab |
Multi-step flow and stateful testing |
apophis init --preset llm-safe
apophis doctor Checks
Run apophis doctor to validate your setup:
- Dependencies: Checks for
fastify,@fastify/swagger - Config validation: Rejects unknown keys, unsafe modes
- Route discovery: Confirms routes are discoverable
- Safety checks: Blocks qualify in production, missing sinks
- Docs drift: Validates examples in CI mode
apophis doctor
CI Policy Guards
Add these checks to your CI pipeline:
name: APOPHIS Checks
on: [push, pull_request]
jobs:
apophis:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- run: npm ci
- run: npx apophis doctor
- run: npx apophis verify --profile ci --changed
Template Examples
Minimal LLM-Safe Config
// apophis.config.js
export default {
mode: 'verify',
profile: 'llm-check',
profiles: {
'llm-check': {
name: 'llm-check',
mode: 'verify',
preset: 'llm-safe',
routes: []
}
},
presets: {
'llm-safe': {
name: 'llm-safe',
timeout: 3000,
parallel: false,
chaos: false,
observe: false
}
},
environments: {
local: {
name: 'local',
allowVerify: true,
allowObserve: false,
allowQualify: false,
allowChaos: false,
allowBlocking: false,
requireSink: false
}
}
};
Route Template with Behavioral Contract
import crypto from 'crypto';
app.post('/users', {
schema: {
'x-category': 'constructor',
'x-ensures': [
// BEHAVIORAL: Creating a user must make it retrievable
'response_code(GET /users/{response_body(this).id}) == 200'
],
body: {
type: 'object',
properties: {
name: { type: 'string', minLength: 1 }
},
required: ['name']
},
response: {
201: {
type: 'object',
properties: {
id: { type: 'string' },
name: { type: 'string' }
}
}
}
}
}, async (request, reply) => {
const { name } = request.body;
const id = `usr-${crypto.createHash('sha256').update(name).digest('hex').slice(0, 8)}`;
reply.status(201);
return { id, name };
});
CI Policy Guard Script
// scripts/apophis-ci-guard.js
import { execSync } from 'node:child_process';
// Run doctor
const doctorResult = execSync('npx apophis doctor', { encoding: 'utf-8' });
console.log(doctorResult);
// Run verify
const verifyResult = execSync('npx apophis verify --profile ci --changed', { encoding: 'utf-8' });
console.log(verifyResult);
Best Practices
- Start with presets: Avoid raw manual config until the project needs explicit overrides.
- Run doctor first: Catch setup issues before running verify.
- Use
--changedin CI: Only verify routes that changed in the PR. - Commit config: Store
apophis.config.jsin version control. - Pin versions: Pin
@apophis/fastifyversion inpackage.json.
Troubleshooting
"Unknown config key"
APOPHIS rejects unknown keys to prevent hallucinated config. Check the key name against the config schema.
"Qualify blocked in production"
Qualify mode is blocked in production by default. Use a non-production environment or explicitly allow it in your environment policy.
"Missing sink config"
Observe mode requires a sink config in staging/production. Add requireSink: true to your environment policy and configure a sink.