docs: audit + fix — remove observe CLI refs, correct defaults, fix public API imports, add historical banners

This commit is contained in:
John Dvorak
2026-05-22 17:07:12 -07:00
parent 4be5fd74cb
commit fa0f6e1fe5
15 changed files with 52 additions and 55 deletions
+5 -3
View File
@@ -34,10 +34,12 @@ Observed results:
|---|---:| |---|---:|
| Typecheck | pass | | Typecheck | pass |
| Build | pass | | Build | pass |
| Source tests | 590 pass, 0 fail | | Source tests | 590 pass, 0 fail (at audit time) |
| CLI tests | 278 pass, 0 fail | | CLI tests | 278 pass, 0 fail (at audit time) |
| Docs smoke tests | 4 pass, 0 fail | | Docs smoke tests | 4 pass, 0 fail |
| Total tests | 879 pass, 0 fail | | Total tests | 879 pass, 0 fail (at audit time) |
Test counts are snapshots at audit time (2026-05-21) and may differ in subsequent development.
The working tree contains many broader project changes unrelated to this audit. This document evaluates the current working tree state. The working tree contains many broader project changes unrelated to this audit. This document evaluates the current working tree state.
+1 -1
View File
@@ -109,7 +109,7 @@ const schema = {
#### Mutation Testing #### Mutation Testing
- **New**: `src/quality/mutation.ts` — synthetic bug injection to measure contract strength. - **New**: `src/quality/mutation.ts` — synthetic bug injection to measure contract strength.
- **New**: `runMutationTesting()` — generates mutations and verifies tests catch them. - **New**: `runMutationTesting()` — generates mutations and verifies tests catch them. Internal API only; not part of the public `@apophis/fastify` export.
- **New**: Mutation score reporting (0-100%) with weak contract identification. - **New**: Mutation score reporting (0-100%) with weak contract identification.
### Changed ### Changed
+4 -4
View File
@@ -72,10 +72,10 @@ JSON Schema cannot express this relationship. APOPHIS turns it into an executabl
| Mode | Purpose | Default Environments | | Mode | Purpose | Default Environments |
|---|---|---| |---|---|---|
| `verify` | Deterministic CI and local contract verification | local, test, CI | | `verify` | Deterministic CI and local contract verification | local, test, CI |
| `observe` | Runtime visibility and drift detection without blocking | staging, prod | | `observe` | Programmatic runtime visibility and drift detection without blocking (validated via `doctor --mode observe`) | staging, prod |
| `qualify` | Exercise scenarios, stateful flows, and configured chaos checks before release | local, test, staging | | `qualify` | Exercise scenarios, stateful flows, and configured chaos checks before release | local, test, staging |
## Quickstart: 3 Commands ## Quickstart
```bash ```bash
# 1. Install # 1. Install
@@ -99,7 +99,7 @@ See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough.
- **Explicit test budget**: Control how many tests run with `runs: 10` in your preset. - **Explicit test budget**: Control how many tests run with `runs: 10` in your preset.
- **CI-safe default path**: `verify` is deterministic and safe for CI pipelines. - **CI-safe default path**: `verify` is deterministic and safe for CI pipelines.
- **Machine-readable output**: `--format json-summary` and `--format ndjson-summary` for CI dashboards. - **Machine-readable output**: `--format json-summary` and `--format ndjson-summary` for CI dashboards.
- **Production-safe observe path**: `observe` is non-blocking by default. Blocking behavior requires explicit break-glass policy. - **Production-safe observe path**: `observe` is the programmatic runtime plugin (no CLI command). Non-blocking by default. Blocking behavior requires explicit break-glass policy.
- **Qualify path gated away from prod**: `qualify` is blocked in production by default. - **Qualify path gated away from prod**: `qualify` is blocked in production by default.
- **Monorepo workspace support**: `--workspace` fans out `verify` and `doctor` across all packages. - **Monorepo workspace support**: `--workspace` fans out `verify` and `doctor` across all packages.
- **Explicit environment boundaries**: Config rejects unknown keys and unsafe environment mixes. - **Explicit environment boundaries**: Config rejects unknown keys and unsafe environment mixes.
@@ -118,7 +118,7 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
## Full Documentation ## Full Documentation
- [Getting Started](docs/getting-started.md) — First route, first verify run, first replay - [Getting Started](docs/getting-started.md) — First route, first verify run, first replay
- [CLI Reference](docs/cli.md) — All 7 commands, global flags, exit codes - [CLI Reference](docs/cli.md) — All 6 commands, global flags, exit codes
- [Verify Mode](docs/verify.md) — Deterministic contract verification - [Verify Mode](docs/verify.md) — Deterministic contract verification
- [Observe Mode](docs/observe.md) — Runtime visibility and drift detection - [Observe Mode](docs/observe.md) — Runtime visibility and drift detection
- [Qualify Mode](docs/qualify.md) — Scenarios, stateful testing, chaos - [Qualify Mode](docs/qualify.md) — Scenarios, stateful testing, chaos
+2
View File
@@ -1,5 +1,7 @@
# Cache & CI/CD Integration # Cache & CI/CD Integration
> **Note**: The incremental cache is an internal mechanism used by programmatic test runners (petit-runner, stateful-runner). CLI users do not interact with `.apophis-cache.json` directly. This document describes the internal caching behavior for advanced programmatic integration.
APOPHIS includes an incremental test cache that speeds up test runs by skipping unchanged routes. This document covers cache invalidation strategies and CI/CD integration. APOPHIS includes an incremental test cache that speeds up test runs by skipping unchanged routes. This document covers cache invalidation strategies and CI/CD integration.
## How the Cache Works ## How the Cache Works
+2 -21
View File
@@ -28,7 +28,7 @@ Adds artificial latency. Tests timeout contracts:
response_time(this) < 1000 response_time(this) < 1000
``` ```
**Note**: Delay events are generated by the chaos arbitrary but the inbound delay handler is currently a no-op. Use this for timeout contract documentation; actual delay injection requires the outbound delay strategy or a custom handler. Delay chaos strategies (`inbound-delay`, `outbound-delay`) are applied at the transport level between request execution and contract evaluation. The inline chaos handlers for these strategies are no-ops because `sleep()` handles delay application out-of-band. Delay contracts such as `response_time(this) < 1000` will still work correctly with chaos injection.
### Error ### Error
@@ -69,26 +69,7 @@ Built-in strategies are content-type agnostic:
Extension strategies can add content-type-specific behavior if needed. Extension strategies can add content-type-specific behavior if needed.
## Custom Corruption via Extensions **Note**: Extension-defined corruption strategies are documented for future implementation. Currently, corruption strategies (`truncate`, `malformed`, `field-corrupt`) are hardcoded in the chaos engine.
```javascript
const myExtension = {
name: 'custom-corrupt',
corruptionStrategies: {
'application/vnd.api+json': (data) => ({
...data,
corrupted: true,
}),
'text/*': (data) => `CORRUPTED:${String(data)}`,
},
};
await fastify.register(apophis, {
extensions: [myExtension],
});
```
Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`) match any subtype.
## Environment Guard ## Environment Guard
+3 -3
View File
@@ -58,11 +58,11 @@ apophis verify --profile quick --routes "POST /users"
``` ```
| Flag | Description | | Flag | Description |
|---|---| |---|---|---|
| `--profile <name>` | Profile name from config | | `--profile <name>` | Profile name from config |
| `--routes <filter>` | Route filter pattern (comma-separated, supports wildcards) |
| `--seed <number>` | Deterministic seed (generated and printed if omitted) | | `--seed <number>` | Deterministic seed (generated and printed if omitted) |
| `--changed` | Filter to git-modified routes only | | `--changed` | Filter to git-modified routes only |
| `--changed` | Filter to git-modified routes only |
| `--workspace` | Run across all workspace packages | | `--workspace` | Run across all workspace packages |
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` | | `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` |
@@ -198,7 +198,7 @@ apophis replay --artifact reports/apophis/failure-*.json
- `--changed` requires a git repository - `--changed` requires a git repository
- `migrate` defaults to `--dry-run` (safe by default) - `migrate` defaults to `--dry-run` (safe by default)
- `--workspace` is fully implemented by `verify` and `doctor`. `observe` and `qualify` accept the flag but run in the current package only. - `--workspace` is fully implemented by `verify` and `doctor`. Other commands do not support `--workspace`.
- Seeds ensure deterministic generation; handler nondeterminism (e.g., `Date.now()`) can still cause replay divergence - Seeds ensure deterministic generation; handler nondeterminism (e.g., `Date.now()`) can still cause replay divergence
## Exit Codes ## Exit Codes
@@ -1,5 +1,7 @@
# APOPHIS v1.1 Architecture — Hybrid Core + Extensions # APOPHIS v1.1 Architecture — Hybrid Core + Extensions
> **HISTORICAL**: This is a v1.1 architecture design specification. For current extension documentation, see [EXTENSION-PLUGIN-SYSTEM.md](EXTENSION-PLUGIN-SYSTEM.md) and [QUICK-REFERENCE.md](QUICK-REFERENCE.md).
## Status: Architecture Specification ## Status: Architecture Specification
## Date: 2026-04-24 ## Date: 2026-04-24
## Scope: v1.1 First-Class Features & Extension Ecosystem ## Scope: v1.1 First-Class Features & Extension Ecosystem
+1 -1
View File
@@ -392,7 +392,7 @@ await app.register(apophis, {
| `src/test/extension.test.ts` | Extension system tests | | `src/test/extension.test.ts` | Extension system tests |
| `src/formula/evaluator.ts` | APOSTL evaluator with extension predicate resolution | | `src/formula/evaluator.ts` | APOSTL evaluator with extension predicate resolution |
| `src/domain/contract-validation.ts` | Passes extension registry to evaluator | | `src/domain/contract-validation.ts` | Passes extension registry to evaluator |
| `src/test/petit-runner.ts` | Calls extension hooks | | `src/quality/petit-runner.ts` | Calls extension hooks |
| `src/plugin/index.ts` | Creates and passes ExtensionRegistry | | `src/plugin/index.ts` | Creates and passes ExtensionRegistry |
--- ---
@@ -1,4 +1,4 @@
# APOPHIS v1.0 Extension Specification: Timeouts and Redirects # APOPHIS Extension Specification: Timeouts and Redirects
## Document Information ## Document Information
- **Version**: 1.0 - **Version**: 1.0
@@ -64,6 +64,8 @@ fastify.get('/slow-endpoint', {
When a timeout is configured, `executeHttp` uses an abortable timer where supported. The timeout must be cleared in `finally`; Fastify injection may continue running after timeout if the underlying transport cannot be cancelled. When a timeout is configured, `executeHttp` uses an abortable timer where supported. The timeout must be cleared in `finally`; Fastify injection may continue running after timeout if the underlying transport cannot be cancelled.
> **Note**: The code examples in this section are illustrative representations of the timeout, redirect, and concurrency mechanisms. The actual implementation may differ in detail.
```typescript ```typescript
// In src/infrastructure/http-executor.ts // In src/infrastructure/http-executor.ts
if (timeoutMs && timeoutMs > 0) { if (timeoutMs && timeoutMs > 0) {
+2 -2
View File
@@ -6,7 +6,7 @@ APOPHIS is inspired by the concept of invariant-driven automated testing: instea
## Prerequisites ## Prerequisites
- Node.js 20.x or 22.x - Node.js >=20.18.1 (20.x) or >=22 (22.x)
- **Fastify v5** (v4 is not supported) - **Fastify v5** (v4 is not supported)
- **ESM project** (`"type": "module"` in package.json) - **ESM project** (`"type": "module"` in package.json)
- A Fastify app with `@fastify/swagger` registered - A Fastify app with `@fastify/swagger` registered
@@ -143,7 +143,7 @@ APOPHIS contracts should verify **behavior**, not structure. Fastify and `@fasti
- Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"` - Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"`
- Use wildcards to match route patterns: `apophis verify --routes 'POST /api/*'` - Use wildcards to match route patterns: `apophis verify --routes 'POST /api/*'`
- Run all routes: `apophis verify --profile quick` - Run all routes: `apophis verify --profile quick`
- Run only changed routes in CI: `apophis verify --profile ci --changed` - Run only changed routes in CI: `apophis verify --profile quick --changed`
- Requires a git repository. - Requires a git repository.
- Use machine-readable output in CI: `apophis verify --profile ci --format json-summary` - Use machine-readable output in CI: `apophis verify --profile ci --format json-summary`
- Add observe mode for runtime drift detection: see [observe.md](observe.md) - Add observe mode for runtime drift detection: see [observe.md](observe.md)
+12 -12
View File
@@ -29,21 +29,19 @@ Observe mode requires a reporting sink. Configure it in your environment policy:
environments: { environments: {
staging: { staging: {
name: 'staging', name: 'staging',
allowVerify: true, allowedModes: ['verify', 'observe'],
allowObserve: true, blockQualify: true,
allowQualify: false, requireSink: true,
allowChaos: false, allowBlocking: false
allowBlocking: false,
requireSink: true
} }
} }
``` ```
APOPHIS supports these sink types: You implement sinks for your observability backend. Common categories:
- **Logs**: Structured logging of contract violations - **Logs**: Structured logging of contract violations (pino, winston)
- **Metrics**: Counter and histogram metrics for violation rates - **Metrics**: Counter and histogram metrics for violation rates (Prometheus, OpenTelemetry)
- **Traces**: Distributed tracing integration for violation context - **Traces**: Distributed tracing integration for violation context (OpenTelemetry, Jaeger)
## Sampling ## Sampling
@@ -97,10 +95,12 @@ profiles: {
Validate your observe config before deployment with doctor: Validate your observe config before deployment with doctor:
## Exit Codes ## Validation (via Doctor)
Validate observe configuration with `apophis doctor --mode observe`:
| Code | Meaning | | Code | Meaning |
|---|---| |---|---|---|
| 0 | Observe config is valid and safe | | 0 | Observe config is valid and safe |
| 2 | Safety violation or invalid config | | 2 | Safety violation or invalid config |
+3 -1
View File
@@ -183,7 +183,9 @@ Qualify mode is gated away from production by default:
|---|---|---|---| |---|---|---|---|
| local | enabled | enabled | enabled | | local | enabled | enabled | enabled |
| test/CI | enabled | enabled | enabled | | test/CI | enabled | enabled | enabled |
| staging | enabled with allowlist | enabled | blocked on protected routes | | staging | enabled (blockQualify: false) | enabled (blockQualify: false) | blocked on protected routes (allowChaosOnProtected: false) |
Qualify gates are not individually gated per environment. The `blockQualify` flag controls all qualify execution, and `allowChaosOnProtected` controls chaos on protected routes.
| production | disabled by default | disabled by default | disabled by default | | production | disabled by default | disabled by default | disabled by default |
## Machine Output for CI ## Machine Output for CI
+9 -3
View File
@@ -71,7 +71,9 @@ Automatically rerun failing tests with varied seeds to detect non-deterministic
### Usage ### Usage
```javascript ```javascript
import { FlakeDetector } from '@apophis/fastify' // FlakeDetector is an internal API. It is not part of the public @apophis/fastify export.
// For programmatic use, import from the internal path:
// import { FlakeDetector } from '@apophis/fastify/src/quality/flake.js'
const detector = new FlakeDetector({ const detector = new FlakeDetector({
sameSeedReruns: 1, // Rerun with same seed sameSeedReruns: 1, // Rerun with same seed
@@ -121,7 +123,9 @@ Measure contract strength by injecting synthetic bugs. A "mutation" is a small c
### Usage ### Usage
```javascript ```javascript
import { runMutationTesting } from '@apophis/fastify/quality/mutation' // Mutation testing is an internal API. It is not part of the public @apophis/fastify export.
// For programmatic use, import from the internal path:
// import { runMutationTesting } from '@apophis/fastify/src/quality/mutation.js'
const report = await runMutationTesting(fastify, { const report = await runMutationTesting(fastify, {
runs: 10, runs: 10,
@@ -176,7 +180,9 @@ console.log('Weak contracts:', report.weakContracts)
Test a specific mutation without running the full suite: Test a specific mutation without running the full suite:
```javascript ```javascript
import { testMutation } from '@apophis/fastify/quality/mutation' // Mutation testing is an internal API. It is not part of the public @apophis/fastify export.
// For programmatic use, import from the internal path:
// import { testMutation } from '@apophis/fastify/src/quality/mutation.js'
const killed = await testMutation(fastify, contract, mutation, { const killed = await testMutation(fastify, contract, mutation, {
runs: 10, runs: 10,
+2 -2
View File
@@ -38,10 +38,10 @@ APOPHIS classifies failures into six categories. Lower categories take precedenc
1. Check the route and clause index printed in the error message. 1. Check the route and clause index printed in the error message.
2. Verify APOSTL syntax: use `response_code(this)` not `response_code()`. 2. Verify APOSTL syntax: use `response_code(this)` not `response_code()`.
3. Ensure string literals use single or double quotes consistently. 3. Ensure string literals use single or double quotes consistently.
4. Run `apophis doctor --profile <name>` to validate formulas without executing. 4. Run `apophis doctor` to validate formulas without executing.
**Prevention** **Prevention**
- Run `apophis doctor --profile <name>` to validate formulas without executing. - Run `apophis doctor` to validate formulas without executing.
- Enable editor support for APOSTL syntax highlighting. - Enable editor support for APOSTL syntax highlighting.
--- ---
+1 -1
View File
@@ -206,4 +206,4 @@ presets: {
} }
``` ```
CLI verify generates one property-based test sample per contract by default when no `runs` is specified. Set `runs` in the preset to increase sampled inputs per route. CLI verify defaults to 50 runs per contract. Set `runs` in the preset to adjust the sample count per route. Use `runs: 1` to check each contract once, or `runs: 0` to disable property-based verification.