diff --git a/APOPHIS_ADOPTION_AUDIT.md b/APOPHIS_ADOPTION_AUDIT.md index a00bbfb..b6154bc 100644 --- a/APOPHIS_ADOPTION_AUDIT.md +++ b/APOPHIS_ADOPTION_AUDIT.md @@ -34,10 +34,12 @@ Observed results: |---|---:| | Typecheck | pass | | Build | pass | -| Source tests | 590 pass, 0 fail | -| CLI tests | 278 pass, 0 fail | +| Source tests | 590 pass, 0 fail (at audit time) | +| CLI tests | 278 pass, 0 fail (at audit time) | | Docs smoke tests | 4 pass, 0 fail | -| Total tests | 879 pass, 0 fail | +| Total tests | 879 pass, 0 fail (at audit time) | + +Test counts are snapshots at audit time (2026-05-21) and may differ in subsequent development. The working tree contains many broader project changes unrelated to this audit. This document evaluates the current working tree state. diff --git a/CHANGELOG.md b/CHANGELOG.md index 230fc80..4f3eecd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -109,7 +109,7 @@ const schema = { #### Mutation Testing - **New**: `src/quality/mutation.ts` — synthetic bug injection to measure contract strength. -- **New**: `runMutationTesting()` — generates mutations and verifies tests catch them. +- **New**: `runMutationTesting()` — generates mutations and verifies tests catch them. Internal API only; not part of the public `@apophis/fastify` export. - **New**: Mutation score reporting (0-100%) with weak contract identification. ### Changed diff --git a/README.md b/README.md index c4ebacd..d483da0 100644 --- a/README.md +++ b/README.md @@ -72,10 +72,10 @@ JSON Schema cannot express this relationship. APOPHIS turns it into an executabl | Mode | Purpose | Default Environments | |---|---|---| | `verify` | Deterministic CI and local contract verification | local, test, CI | -| `observe` | Runtime visibility and drift detection without blocking | staging, prod | +| `observe` | Programmatic runtime visibility and drift detection without blocking (validated via `doctor --mode observe`) | staging, prod | | `qualify` | Exercise scenarios, stateful flows, and configured chaos checks before release | local, test, staging | -## Quickstart: 3 Commands +## Quickstart ```bash # 1. Install @@ -99,7 +99,7 @@ See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough. - **Explicit test budget**: Control how many tests run with `runs: 10` in your preset. - **CI-safe default path**: `verify` is deterministic and safe for CI pipelines. - **Machine-readable output**: `--format json-summary` and `--format ndjson-summary` for CI dashboards. -- **Production-safe observe path**: `observe` is non-blocking by default. Blocking behavior requires explicit break-glass policy. +- **Production-safe observe path**: `observe` is the programmatic runtime plugin (no CLI command). Non-blocking by default. Blocking behavior requires explicit break-glass policy. - **Qualify path gated away from prod**: `qualify` is blocked in production by default. - **Monorepo workspace support**: `--workspace` fans out `verify` and `doctor` across all packages. - **Explicit environment boundaries**: Config rejects unknown keys and unsafe environment mixes. @@ -118,7 +118,7 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI ## Full Documentation - [Getting Started](docs/getting-started.md) — First route, first verify run, first replay -- [CLI Reference](docs/cli.md) — All 7 commands, global flags, exit codes +- [CLI Reference](docs/cli.md) — All 6 commands, global flags, exit codes - [Verify Mode](docs/verify.md) — Deterministic contract verification - [Observe Mode](docs/observe.md) — Runtime visibility and drift detection - [Qualify Mode](docs/qualify.md) — Scenarios, stateful testing, chaos diff --git a/docs/cache-and-ci.md b/docs/cache-and-ci.md index 29f9ee2..2f8f870 100644 --- a/docs/cache-and-ci.md +++ b/docs/cache-and-ci.md @@ -1,5 +1,7 @@ # Cache & CI/CD Integration +> **Note**: The incremental cache is an internal mechanism used by programmatic test runners (petit-runner, stateful-runner). CLI users do not interact with `.apophis-cache.json` directly. This document describes the internal caching behavior for advanced programmatic integration. + APOPHIS includes an incremental test cache that speeds up test runs by skipping unchanged routes. This document covers cache invalidation strategies and CI/CD integration. ## How the Cache Works diff --git a/docs/chaos.md b/docs/chaos.md index 75ac4d3..3a74327 100644 --- a/docs/chaos.md +++ b/docs/chaos.md @@ -28,7 +28,7 @@ Adds artificial latency. Tests timeout contracts: response_time(this) < 1000 ``` -**Note**: Delay events are generated by the chaos arbitrary but the inbound delay handler is currently a no-op. Use this for timeout contract documentation; actual delay injection requires the outbound delay strategy or a custom handler. +Delay chaos strategies (`inbound-delay`, `outbound-delay`) are applied at the transport level between request execution and contract evaluation. The inline chaos handlers for these strategies are no-ops because `sleep()` handles delay application out-of-band. Delay contracts such as `response_time(this) < 1000` will still work correctly with chaos injection. ### Error @@ -69,26 +69,7 @@ Built-in strategies are content-type agnostic: Extension strategies can add content-type-specific behavior if needed. -## Custom Corruption via Extensions - -```javascript -const myExtension = { - name: 'custom-corrupt', - corruptionStrategies: { - 'application/vnd.api+json': (data) => ({ - ...data, - corrupted: true, - }), - 'text/*': (data) => `CORRUPTED:${String(data)}`, - }, -}; - -await fastify.register(apophis, { - extensions: [myExtension], -}); -``` - -Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`) match any subtype. +**Note**: Extension-defined corruption strategies are documented for future implementation. Currently, corruption strategies (`truncate`, `malformed`, `field-corrupt`) are hardcoded in the chaos engine. ## Environment Guard diff --git a/docs/cli.md b/docs/cli.md index f925e03..b41a353 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -58,11 +58,11 @@ apophis verify --profile quick --routes "POST /users" ``` | Flag | Description | -|---|---| +|---|---|---| | `--profile ` | Profile name from config | -| `--routes ` | Route filter pattern (comma-separated, supports wildcards) | | `--seed ` | Deterministic seed (generated and printed if omitted) | | `--changed` | Filter to git-modified routes only | +| `--changed` | Filter to git-modified routes only | | `--workspace` | Run across all workspace packages | | `--format ` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` | @@ -198,7 +198,7 @@ apophis replay --artifact reports/apophis/failure-*.json - `--changed` requires a git repository - `migrate` defaults to `--dry-run` (safe by default) -- `--workspace` is fully implemented by `verify` and `doctor`. `observe` and `qualify` accept the flag but run in the current package only. +- `--workspace` is fully implemented by `verify` and `doctor`. Other commands do not support `--workspace`. - Seeds ensure deterministic generation; handler nondeterminism (e.g., `Date.now()`) can still cause replay divergence ## Exit Codes diff --git a/docs/extensions/EXTENSION-ARCHITECTURE.md b/docs/extensions/EXTENSION-ARCHITECTURE.md index ea439a8..ffb8091 100644 --- a/docs/extensions/EXTENSION-ARCHITECTURE.md +++ b/docs/extensions/EXTENSION-ARCHITECTURE.md @@ -1,5 +1,7 @@ # APOPHIS v1.1 Architecture — Hybrid Core + Extensions +> **HISTORICAL**: This is a v1.1 architecture design specification. For current extension documentation, see [EXTENSION-PLUGIN-SYSTEM.md](EXTENSION-PLUGIN-SYSTEM.md) and [QUICK-REFERENCE.md](QUICK-REFERENCE.md). + ## Status: Architecture Specification ## Date: 2026-04-24 ## Scope: v1.1 First-Class Features & Extension Ecosystem diff --git a/docs/extensions/EXTENSION-PLUGIN-SYSTEM.md b/docs/extensions/EXTENSION-PLUGIN-SYSTEM.md index e660c65..2afa6f3 100644 --- a/docs/extensions/EXTENSION-PLUGIN-SYSTEM.md +++ b/docs/extensions/EXTENSION-PLUGIN-SYSTEM.md @@ -392,7 +392,7 @@ await app.register(apophis, { | `src/test/extension.test.ts` | Extension system tests | | `src/formula/evaluator.ts` | APOSTL evaluator with extension predicate resolution | | `src/domain/contract-validation.ts` | Passes extension registry to evaluator | -| `src/test/petit-runner.ts` | Calls extension hooks | +| `src/quality/petit-runner.ts` | Calls extension hooks | | `src/plugin/index.ts` | Creates and passes ExtensionRegistry | --- diff --git a/docs/extensions/TIMEOUTS-REDIRECTS-CONCURRENCY.md b/docs/extensions/TIMEOUTS-REDIRECTS-CONCURRENCY.md index b0d0d26..efa9006 100644 --- a/docs/extensions/TIMEOUTS-REDIRECTS-CONCURRENCY.md +++ b/docs/extensions/TIMEOUTS-REDIRECTS-CONCURRENCY.md @@ -1,4 +1,4 @@ -# APOPHIS v1.0 Extension Specification: Timeouts and Redirects +# APOPHIS Extension Specification: Timeouts and Redirects ## Document Information - **Version**: 1.0 @@ -64,6 +64,8 @@ fastify.get('/slow-endpoint', { When a timeout is configured, `executeHttp` uses an abortable timer where supported. The timeout must be cleared in `finally`; Fastify injection may continue running after timeout if the underlying transport cannot be cancelled. +> **Note**: The code examples in this section are illustrative representations of the timeout, redirect, and concurrency mechanisms. The actual implementation may differ in detail. + ```typescript // In src/infrastructure/http-executor.ts if (timeoutMs && timeoutMs > 0) { diff --git a/docs/getting-started.md b/docs/getting-started.md index d3cfb07..dc16b63 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -6,7 +6,7 @@ APOPHIS is inspired by the concept of invariant-driven automated testing: instea ## Prerequisites -- Node.js 20.x or 22.x +- Node.js >=20.18.1 (20.x) or >=22 (22.x) - **Fastify v5** (v4 is not supported) - **ESM project** (`"type": "module"` in package.json) - A Fastify app with `@fastify/swagger` registered @@ -143,7 +143,7 @@ APOPHIS contracts should verify **behavior**, not structure. Fastify and `@fasti - Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"` - Use wildcards to match route patterns: `apophis verify --routes 'POST /api/*'` - Run all routes: `apophis verify --profile quick` -- Run only changed routes in CI: `apophis verify --profile ci --changed` +- Run only changed routes in CI: `apophis verify --profile quick --changed` - Requires a git repository. - Use machine-readable output in CI: `apophis verify --profile ci --format json-summary` - Add observe mode for runtime drift detection: see [observe.md](observe.md) diff --git a/docs/observe.md b/docs/observe.md index cec2f8f..e909ee0 100644 --- a/docs/observe.md +++ b/docs/observe.md @@ -29,21 +29,19 @@ Observe mode requires a reporting sink. Configure it in your environment policy: environments: { staging: { name: 'staging', - allowVerify: true, - allowObserve: true, - allowQualify: false, - allowChaos: false, - allowBlocking: false, - requireSink: true + allowedModes: ['verify', 'observe'], + blockQualify: true, + requireSink: true, + allowBlocking: false } } ``` -APOPHIS supports these sink types: +You implement sinks for your observability backend. Common categories: -- **Logs**: Structured logging of contract violations -- **Metrics**: Counter and histogram metrics for violation rates -- **Traces**: Distributed tracing integration for violation context +- **Logs**: Structured logging of contract violations (pino, winston) +- **Metrics**: Counter and histogram metrics for violation rates (Prometheus, OpenTelemetry) +- **Traces**: Distributed tracing integration for violation context (OpenTelemetry, Jaeger) ## Sampling @@ -97,10 +95,12 @@ profiles: { Validate your observe config before deployment with doctor: -## Exit Codes +## Validation (via Doctor) + +Validate observe configuration with `apophis doctor --mode observe`: | Code | Meaning | -|---|---| +|---|---|---| | 0 | Observe config is valid and safe | | 2 | Safety violation or invalid config | diff --git a/docs/qualify.md b/docs/qualify.md index abc5e23..0de1f10 100644 --- a/docs/qualify.md +++ b/docs/qualify.md @@ -183,7 +183,9 @@ Qualify mode is gated away from production by default: |---|---|---|---| | local | enabled | enabled | enabled | | test/CI | enabled | enabled | enabled | -| staging | enabled with allowlist | enabled | blocked on protected routes | +| staging | enabled (blockQualify: false) | enabled (blockQualify: false) | blocked on protected routes (allowChaosOnProtected: false) | + +Qualify gates are not individually gated per environment. The `blockQualify` flag controls all qualify execution, and `allowChaosOnProtected` controls chaos on protected routes. | production | disabled by default | disabled by default | disabled by default | ## Machine Output for CI diff --git a/docs/quality.md b/docs/quality.md index 9c16938..c02a95e 100644 --- a/docs/quality.md +++ b/docs/quality.md @@ -71,7 +71,9 @@ Automatically rerun failing tests with varied seeds to detect non-deterministic ### Usage ```javascript -import { FlakeDetector } from '@apophis/fastify' +// FlakeDetector is an internal API. It is not part of the public @apophis/fastify export. +// For programmatic use, import from the internal path: +// import { FlakeDetector } from '@apophis/fastify/src/quality/flake.js' const detector = new FlakeDetector({ sameSeedReruns: 1, // Rerun with same seed @@ -121,7 +123,9 @@ Measure contract strength by injecting synthetic bugs. A "mutation" is a small c ### Usage ```javascript -import { runMutationTesting } from '@apophis/fastify/quality/mutation' +// Mutation testing is an internal API. It is not part of the public @apophis/fastify export. +// For programmatic use, import from the internal path: +// import { runMutationTesting } from '@apophis/fastify/src/quality/mutation.js' const report = await runMutationTesting(fastify, { runs: 10, @@ -176,7 +180,9 @@ console.log('Weak contracts:', report.weakContracts) Test a specific mutation without running the full suite: ```javascript -import { testMutation } from '@apophis/fastify/quality/mutation' +// Mutation testing is an internal API. It is not part of the public @apophis/fastify export. +// For programmatic use, import from the internal path: +// import { testMutation } from '@apophis/fastify/src/quality/mutation.js' const killed = await testMutation(fastify, contract, mutation, { runs: 10, diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 7ad80de..a613d42 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -38,10 +38,10 @@ APOPHIS classifies failures into six categories. Lower categories take precedenc 1. Check the route and clause index printed in the error message. 2. Verify APOSTL syntax: use `response_code(this)` not `response_code()`. 3. Ensure string literals use single or double quotes consistently. -4. Run `apophis doctor --profile ` to validate formulas without executing. +4. Run `apophis doctor` to validate formulas without executing. **Prevention** -- Run `apophis doctor --profile ` to validate formulas without executing. +- Run `apophis doctor` to validate formulas without executing. - Enable editor support for APOSTL syntax highlighting. --- diff --git a/docs/verify.md b/docs/verify.md index b30d9ef..c2c9bfb 100644 --- a/docs/verify.md +++ b/docs/verify.md @@ -206,4 +206,4 @@ presets: { } ``` -CLI verify generates one property-based test sample per contract by default when no `runs` is specified. Set `runs` in the preset to increase sampled inputs per route. +CLI verify defaults to 50 runs per contract. Set `runs` in the preset to adjust the sample count per route. Use `runs: 1` to check each contract once, or `runs: 0` to disable property-based verification.