docs: audit + fix — remove observe CLI refs, correct defaults, fix public API imports, add historical banners
This commit is contained in:
@@ -34,10 +34,12 @@ Observed results:
|
|||||||
|---|---:|
|
|---|---:|
|
||||||
| Typecheck | pass |
|
| Typecheck | pass |
|
||||||
| Build | pass |
|
| Build | pass |
|
||||||
| Source tests | 590 pass, 0 fail |
|
| Source tests | 590 pass, 0 fail (at audit time) |
|
||||||
| CLI tests | 278 pass, 0 fail |
|
| CLI tests | 278 pass, 0 fail (at audit time) |
|
||||||
| Docs smoke tests | 4 pass, 0 fail |
|
| Docs smoke tests | 4 pass, 0 fail |
|
||||||
| Total tests | 879 pass, 0 fail |
|
| Total tests | 879 pass, 0 fail (at audit time) |
|
||||||
|
|
||||||
|
Test counts are snapshots at audit time (2026-05-21) and may differ in subsequent development.
|
||||||
|
|
||||||
The working tree contains many broader project changes unrelated to this audit. This document evaluates the current working tree state.
|
The working tree contains many broader project changes unrelated to this audit. This document evaluates the current working tree state.
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -109,7 +109,7 @@ const schema = {
|
|||||||
#### Mutation Testing
|
#### Mutation Testing
|
||||||
|
|
||||||
- **New**: `src/quality/mutation.ts` — synthetic bug injection to measure contract strength.
|
- **New**: `src/quality/mutation.ts` — synthetic bug injection to measure contract strength.
|
||||||
- **New**: `runMutationTesting()` — generates mutations and verifies tests catch them.
|
- **New**: `runMutationTesting()` — generates mutations and verifies tests catch them. Internal API only; not part of the public `@apophis/fastify` export.
|
||||||
- **New**: Mutation score reporting (0-100%) with weak contract identification.
|
- **New**: Mutation score reporting (0-100%) with weak contract identification.
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|||||||
@@ -72,10 +72,10 @@ JSON Schema cannot express this relationship. APOPHIS turns it into an executabl
|
|||||||
| Mode | Purpose | Default Environments |
|
| Mode | Purpose | Default Environments |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `verify` | Deterministic CI and local contract verification | local, test, CI |
|
| `verify` | Deterministic CI and local contract verification | local, test, CI |
|
||||||
| `observe` | Runtime visibility and drift detection without blocking | staging, prod |
|
| `observe` | Programmatic runtime visibility and drift detection without blocking (validated via `doctor --mode observe`) | staging, prod |
|
||||||
| `qualify` | Exercise scenarios, stateful flows, and configured chaos checks before release | local, test, staging |
|
| `qualify` | Exercise scenarios, stateful flows, and configured chaos checks before release | local, test, staging |
|
||||||
|
|
||||||
## Quickstart: 3 Commands
|
## Quickstart
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 1. Install
|
# 1. Install
|
||||||
@@ -99,7 +99,7 @@ See [docs/getting-started.md](docs/getting-started.md) for the full walkthrough.
|
|||||||
- **Explicit test budget**: Control how many tests run with `runs: 10` in your preset.
|
- **Explicit test budget**: Control how many tests run with `runs: 10` in your preset.
|
||||||
- **CI-safe default path**: `verify` is deterministic and safe for CI pipelines.
|
- **CI-safe default path**: `verify` is deterministic and safe for CI pipelines.
|
||||||
- **Machine-readable output**: `--format json-summary` and `--format ndjson-summary` for CI dashboards.
|
- **Machine-readable output**: `--format json-summary` and `--format ndjson-summary` for CI dashboards.
|
||||||
- **Production-safe observe path**: `observe` is non-blocking by default. Blocking behavior requires explicit break-glass policy.
|
- **Production-safe observe path**: `observe` is the programmatic runtime plugin (no CLI command). Non-blocking by default. Blocking behavior requires explicit break-glass policy.
|
||||||
- **Qualify path gated away from prod**: `qualify` is blocked in production by default.
|
- **Qualify path gated away from prod**: `qualify` is blocked in production by default.
|
||||||
- **Monorepo workspace support**: `--workspace` fans out `verify` and `doctor` across all packages.
|
- **Monorepo workspace support**: `--workspace` fans out `verify` and `doctor` across all packages.
|
||||||
- **Explicit environment boundaries**: Config rejects unknown keys and unsafe environment mixes.
|
- **Explicit environment boundaries**: Config rejects unknown keys and unsafe environment mixes.
|
||||||
@@ -118,7 +118,7 @@ See [docs/llm-safe-adoption.md](docs/llm-safe-adoption.md) for templates and CI
|
|||||||
## Full Documentation
|
## Full Documentation
|
||||||
|
|
||||||
- [Getting Started](docs/getting-started.md) — First route, first verify run, first replay
|
- [Getting Started](docs/getting-started.md) — First route, first verify run, first replay
|
||||||
- [CLI Reference](docs/cli.md) — All 7 commands, global flags, exit codes
|
- [CLI Reference](docs/cli.md) — All 6 commands, global flags, exit codes
|
||||||
- [Verify Mode](docs/verify.md) — Deterministic contract verification
|
- [Verify Mode](docs/verify.md) — Deterministic contract verification
|
||||||
- [Observe Mode](docs/observe.md) — Runtime visibility and drift detection
|
- [Observe Mode](docs/observe.md) — Runtime visibility and drift detection
|
||||||
- [Qualify Mode](docs/qualify.md) — Scenarios, stateful testing, chaos
|
- [Qualify Mode](docs/qualify.md) — Scenarios, stateful testing, chaos
|
||||||
|
|||||||
@@ -1,5 +1,7 @@
|
|||||||
# Cache & CI/CD Integration
|
# Cache & CI/CD Integration
|
||||||
|
|
||||||
|
> **Note**: The incremental cache is an internal mechanism used by programmatic test runners (petit-runner, stateful-runner). CLI users do not interact with `.apophis-cache.json` directly. This document describes the internal caching behavior for advanced programmatic integration.
|
||||||
|
|
||||||
APOPHIS includes an incremental test cache that speeds up test runs by skipping unchanged routes. This document covers cache invalidation strategies and CI/CD integration.
|
APOPHIS includes an incremental test cache that speeds up test runs by skipping unchanged routes. This document covers cache invalidation strategies and CI/CD integration.
|
||||||
|
|
||||||
## How the Cache Works
|
## How the Cache Works
|
||||||
|
|||||||
+2
-21
@@ -28,7 +28,7 @@ Adds artificial latency. Tests timeout contracts:
|
|||||||
response_time(this) < 1000
|
response_time(this) < 1000
|
||||||
```
|
```
|
||||||
|
|
||||||
**Note**: Delay events are generated by the chaos arbitrary but the inbound delay handler is currently a no-op. Use this for timeout contract documentation; actual delay injection requires the outbound delay strategy or a custom handler.
|
Delay chaos strategies (`inbound-delay`, `outbound-delay`) are applied at the transport level between request execution and contract evaluation. The inline chaos handlers for these strategies are no-ops because `sleep()` handles delay application out-of-band. Delay contracts such as `response_time(this) < 1000` will still work correctly with chaos injection.
|
||||||
|
|
||||||
### Error
|
### Error
|
||||||
|
|
||||||
@@ -69,26 +69,7 @@ Built-in strategies are content-type agnostic:
|
|||||||
|
|
||||||
Extension strategies can add content-type-specific behavior if needed.
|
Extension strategies can add content-type-specific behavior if needed.
|
||||||
|
|
||||||
## Custom Corruption via Extensions
|
**Note**: Extension-defined corruption strategies are documented for future implementation. Currently, corruption strategies (`truncate`, `malformed`, `field-corrupt`) are hardcoded in the chaos engine.
|
||||||
|
|
||||||
```javascript
|
|
||||||
const myExtension = {
|
|
||||||
name: 'custom-corrupt',
|
|
||||||
corruptionStrategies: {
|
|
||||||
'application/vnd.api+json': (data) => ({
|
|
||||||
...data,
|
|
||||||
corrupted: true,
|
|
||||||
}),
|
|
||||||
'text/*': (data) => `CORRUPTED:${String(data)}`,
|
|
||||||
},
|
|
||||||
};
|
|
||||||
|
|
||||||
await fastify.register(apophis, {
|
|
||||||
extensions: [myExtension],
|
|
||||||
});
|
|
||||||
```
|
|
||||||
|
|
||||||
Extension strategies take precedence over built-ins. Wildcard patterns (`text/*`) match any subtype.
|
|
||||||
|
|
||||||
## Environment Guard
|
## Environment Guard
|
||||||
|
|
||||||
|
|||||||
+3
-3
@@ -58,11 +58,11 @@ apophis verify --profile quick --routes "POST /users"
|
|||||||
```
|
```
|
||||||
|
|
||||||
| Flag | Description |
|
| Flag | Description |
|
||||||
|---|---|
|
|---|---|---|
|
||||||
| `--profile <name>` | Profile name from config |
|
| `--profile <name>` | Profile name from config |
|
||||||
| `--routes <filter>` | Route filter pattern (comma-separated, supports wildcards) |
|
|
||||||
| `--seed <number>` | Deterministic seed (generated and printed if omitted) |
|
| `--seed <number>` | Deterministic seed (generated and printed if omitted) |
|
||||||
| `--changed` | Filter to git-modified routes only |
|
| `--changed` | Filter to git-modified routes only |
|
||||||
|
| `--changed` | Filter to git-modified routes only |
|
||||||
| `--workspace` | Run across all workspace packages |
|
| `--workspace` | Run across all workspace packages |
|
||||||
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` |
|
| `--format <mode>` | Output format: `human`, `json`, `ndjson`, `json-summary`, `ndjson-summary` |
|
||||||
|
|
||||||
@@ -198,7 +198,7 @@ apophis replay --artifact reports/apophis/failure-*.json
|
|||||||
|
|
||||||
- `--changed` requires a git repository
|
- `--changed` requires a git repository
|
||||||
- `migrate` defaults to `--dry-run` (safe by default)
|
- `migrate` defaults to `--dry-run` (safe by default)
|
||||||
- `--workspace` is fully implemented by `verify` and `doctor`. `observe` and `qualify` accept the flag but run in the current package only.
|
- `--workspace` is fully implemented by `verify` and `doctor`. Other commands do not support `--workspace`.
|
||||||
- Seeds ensure deterministic generation; handler nondeterminism (e.g., `Date.now()`) can still cause replay divergence
|
- Seeds ensure deterministic generation; handler nondeterminism (e.g., `Date.now()`) can still cause replay divergence
|
||||||
|
|
||||||
## Exit Codes
|
## Exit Codes
|
||||||
|
|||||||
@@ -1,5 +1,7 @@
|
|||||||
# APOPHIS v1.1 Architecture — Hybrid Core + Extensions
|
# APOPHIS v1.1 Architecture — Hybrid Core + Extensions
|
||||||
|
|
||||||
|
> **HISTORICAL**: This is a v1.1 architecture design specification. For current extension documentation, see [EXTENSION-PLUGIN-SYSTEM.md](EXTENSION-PLUGIN-SYSTEM.md) and [QUICK-REFERENCE.md](QUICK-REFERENCE.md).
|
||||||
|
|
||||||
## Status: Architecture Specification
|
## Status: Architecture Specification
|
||||||
## Date: 2026-04-24
|
## Date: 2026-04-24
|
||||||
## Scope: v1.1 First-Class Features & Extension Ecosystem
|
## Scope: v1.1 First-Class Features & Extension Ecosystem
|
||||||
|
|||||||
@@ -392,7 +392,7 @@ await app.register(apophis, {
|
|||||||
| `src/test/extension.test.ts` | Extension system tests |
|
| `src/test/extension.test.ts` | Extension system tests |
|
||||||
| `src/formula/evaluator.ts` | APOSTL evaluator with extension predicate resolution |
|
| `src/formula/evaluator.ts` | APOSTL evaluator with extension predicate resolution |
|
||||||
| `src/domain/contract-validation.ts` | Passes extension registry to evaluator |
|
| `src/domain/contract-validation.ts` | Passes extension registry to evaluator |
|
||||||
| `src/test/petit-runner.ts` | Calls extension hooks |
|
| `src/quality/petit-runner.ts` | Calls extension hooks |
|
||||||
| `src/plugin/index.ts` | Creates and passes ExtensionRegistry |
|
| `src/plugin/index.ts` | Creates and passes ExtensionRegistry |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
# APOPHIS v1.0 Extension Specification: Timeouts and Redirects
|
# APOPHIS Extension Specification: Timeouts and Redirects
|
||||||
|
|
||||||
## Document Information
|
## Document Information
|
||||||
- **Version**: 1.0
|
- **Version**: 1.0
|
||||||
@@ -64,6 +64,8 @@ fastify.get('/slow-endpoint', {
|
|||||||
|
|
||||||
When a timeout is configured, `executeHttp` uses an abortable timer where supported. The timeout must be cleared in `finally`; Fastify injection may continue running after timeout if the underlying transport cannot be cancelled.
|
When a timeout is configured, `executeHttp` uses an abortable timer where supported. The timeout must be cleared in `finally`; Fastify injection may continue running after timeout if the underlying transport cannot be cancelled.
|
||||||
|
|
||||||
|
> **Note**: The code examples in this section are illustrative representations of the timeout, redirect, and concurrency mechanisms. The actual implementation may differ in detail.
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// In src/infrastructure/http-executor.ts
|
// In src/infrastructure/http-executor.ts
|
||||||
if (timeoutMs && timeoutMs > 0) {
|
if (timeoutMs && timeoutMs > 0) {
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ APOPHIS is inspired by the concept of invariant-driven automated testing: instea
|
|||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
- Node.js 20.x or 22.x
|
- Node.js >=20.18.1 (20.x) or >=22 (22.x)
|
||||||
- **Fastify v5** (v4 is not supported)
|
- **Fastify v5** (v4 is not supported)
|
||||||
- **ESM project** (`"type": "module"` in package.json)
|
- **ESM project** (`"type": "module"` in package.json)
|
||||||
- A Fastify app with `@fastify/swagger` registered
|
- A Fastify app with `@fastify/swagger` registered
|
||||||
@@ -143,7 +143,7 @@ APOPHIS contracts should verify **behavior**, not structure. Fastify and `@fasti
|
|||||||
- Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"`
|
- Add more routes to your profile: `apophis verify --profile quick --routes "POST /users,PUT /users/:id"`
|
||||||
- Use wildcards to match route patterns: `apophis verify --routes 'POST /api/*'`
|
- Use wildcards to match route patterns: `apophis verify --routes 'POST /api/*'`
|
||||||
- Run all routes: `apophis verify --profile quick`
|
- Run all routes: `apophis verify --profile quick`
|
||||||
- Run only changed routes in CI: `apophis verify --profile ci --changed`
|
- Run only changed routes in CI: `apophis verify --profile quick --changed`
|
||||||
- Requires a git repository.
|
- Requires a git repository.
|
||||||
- Use machine-readable output in CI: `apophis verify --profile ci --format json-summary`
|
- Use machine-readable output in CI: `apophis verify --profile ci --format json-summary`
|
||||||
- Add observe mode for runtime drift detection: see [observe.md](observe.md)
|
- Add observe mode for runtime drift detection: see [observe.md](observe.md)
|
||||||
|
|||||||
+12
-12
@@ -29,21 +29,19 @@ Observe mode requires a reporting sink. Configure it in your environment policy:
|
|||||||
environments: {
|
environments: {
|
||||||
staging: {
|
staging: {
|
||||||
name: 'staging',
|
name: 'staging',
|
||||||
allowVerify: true,
|
allowedModes: ['verify', 'observe'],
|
||||||
allowObserve: true,
|
blockQualify: true,
|
||||||
allowQualify: false,
|
requireSink: true,
|
||||||
allowChaos: false,
|
allowBlocking: false
|
||||||
allowBlocking: false,
|
|
||||||
requireSink: true
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
APOPHIS supports these sink types:
|
You implement sinks for your observability backend. Common categories:
|
||||||
|
|
||||||
- **Logs**: Structured logging of contract violations
|
- **Logs**: Structured logging of contract violations (pino, winston)
|
||||||
- **Metrics**: Counter and histogram metrics for violation rates
|
- **Metrics**: Counter and histogram metrics for violation rates (Prometheus, OpenTelemetry)
|
||||||
- **Traces**: Distributed tracing integration for violation context
|
- **Traces**: Distributed tracing integration for violation context (OpenTelemetry, Jaeger)
|
||||||
|
|
||||||
## Sampling
|
## Sampling
|
||||||
|
|
||||||
@@ -97,10 +95,12 @@ profiles: {
|
|||||||
|
|
||||||
Validate your observe config before deployment with doctor:
|
Validate your observe config before deployment with doctor:
|
||||||
|
|
||||||
## Exit Codes
|
## Validation (via Doctor)
|
||||||
|
|
||||||
|
Validate observe configuration with `apophis doctor --mode observe`:
|
||||||
|
|
||||||
| Code | Meaning |
|
| Code | Meaning |
|
||||||
|---|---|
|
|---|---|---|
|
||||||
| 0 | Observe config is valid and safe |
|
| 0 | Observe config is valid and safe |
|
||||||
| 2 | Safety violation or invalid config |
|
| 2 | Safety violation or invalid config |
|
||||||
|
|
||||||
|
|||||||
+3
-1
@@ -183,7 +183,9 @@ Qualify mode is gated away from production by default:
|
|||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| local | enabled | enabled | enabled |
|
| local | enabled | enabled | enabled |
|
||||||
| test/CI | enabled | enabled | enabled |
|
| test/CI | enabled | enabled | enabled |
|
||||||
| staging | enabled with allowlist | enabled | blocked on protected routes |
|
| staging | enabled (blockQualify: false) | enabled (blockQualify: false) | blocked on protected routes (allowChaosOnProtected: false) |
|
||||||
|
|
||||||
|
Qualify gates are not individually gated per environment. The `blockQualify` flag controls all qualify execution, and `allowChaosOnProtected` controls chaos on protected routes.
|
||||||
| production | disabled by default | disabled by default | disabled by default |
|
| production | disabled by default | disabled by default | disabled by default |
|
||||||
|
|
||||||
## Machine Output for CI
|
## Machine Output for CI
|
||||||
|
|||||||
+9
-3
@@ -71,7 +71,9 @@ Automatically rerun failing tests with varied seeds to detect non-deterministic
|
|||||||
### Usage
|
### Usage
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
import { FlakeDetector } from '@apophis/fastify'
|
// FlakeDetector is an internal API. It is not part of the public @apophis/fastify export.
|
||||||
|
// For programmatic use, import from the internal path:
|
||||||
|
// import { FlakeDetector } from '@apophis/fastify/src/quality/flake.js'
|
||||||
|
|
||||||
const detector = new FlakeDetector({
|
const detector = new FlakeDetector({
|
||||||
sameSeedReruns: 1, // Rerun with same seed
|
sameSeedReruns: 1, // Rerun with same seed
|
||||||
@@ -121,7 +123,9 @@ Measure contract strength by injecting synthetic bugs. A "mutation" is a small c
|
|||||||
### Usage
|
### Usage
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
import { runMutationTesting } from '@apophis/fastify/quality/mutation'
|
// Mutation testing is an internal API. It is not part of the public @apophis/fastify export.
|
||||||
|
// For programmatic use, import from the internal path:
|
||||||
|
// import { runMutationTesting } from '@apophis/fastify/src/quality/mutation.js'
|
||||||
|
|
||||||
const report = await runMutationTesting(fastify, {
|
const report = await runMutationTesting(fastify, {
|
||||||
runs: 10,
|
runs: 10,
|
||||||
@@ -176,7 +180,9 @@ console.log('Weak contracts:', report.weakContracts)
|
|||||||
Test a specific mutation without running the full suite:
|
Test a specific mutation without running the full suite:
|
||||||
|
|
||||||
```javascript
|
```javascript
|
||||||
import { testMutation } from '@apophis/fastify/quality/mutation'
|
// Mutation testing is an internal API. It is not part of the public @apophis/fastify export.
|
||||||
|
// For programmatic use, import from the internal path:
|
||||||
|
// import { testMutation } from '@apophis/fastify/src/quality/mutation.js'
|
||||||
|
|
||||||
const killed = await testMutation(fastify, contract, mutation, {
|
const killed = await testMutation(fastify, contract, mutation, {
|
||||||
runs: 10,
|
runs: 10,
|
||||||
|
|||||||
@@ -38,10 +38,10 @@ APOPHIS classifies failures into six categories. Lower categories take precedenc
|
|||||||
1. Check the route and clause index printed in the error message.
|
1. Check the route and clause index printed in the error message.
|
||||||
2. Verify APOSTL syntax: use `response_code(this)` not `response_code()`.
|
2. Verify APOSTL syntax: use `response_code(this)` not `response_code()`.
|
||||||
3. Ensure string literals use single or double quotes consistently.
|
3. Ensure string literals use single or double quotes consistently.
|
||||||
4. Run `apophis doctor --profile <name>` to validate formulas without executing.
|
4. Run `apophis doctor` to validate formulas without executing.
|
||||||
|
|
||||||
**Prevention**
|
**Prevention**
|
||||||
- Run `apophis doctor --profile <name>` to validate formulas without executing.
|
- Run `apophis doctor` to validate formulas without executing.
|
||||||
- Enable editor support for APOSTL syntax highlighting.
|
- Enable editor support for APOSTL syntax highlighting.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|||||||
+1
-1
@@ -206,4 +206,4 @@ presets: {
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
CLI verify generates one property-based test sample per contract by default when no `runs` is specified. Set `runs` in the preset to increase sampled inputs per route.
|
CLI verify defaults to 50 runs per contract. Set `runs` in the preset to adjust the sample count per route. Use `runs: 1` to check each contract once, or `runs: 0` to disable property-based verification.
|
||||||
|
|||||||
Reference in New Issue
Block a user