fix: harden engine, enrich failure diagnostics, close adoption gaps

- P0: CLI verify now honors  test budget with seeded multi-sample
- P0: Observe sampling enforced via Math.random() gate in hook-validator
- P1: Remove misleading undici-mock-agent isolation option
- P1: Qualify reuses shared discoverRouteDetails() with warnings
- P1: Chaos/scenario config exposed via preset schema
- P1: README/docs limitations updated to current state
- P2: Nested response annotations prefer 2xx deterministically
- P2: --changed documented as heuristic in verify.md

- Add observe sink tests (sampling 0/1, sink failure non-interference)
- Add verify runs regression tests (scale, determinism, variants)
- Add configured-scenario qualify test (independent of OAuth fixture)
- Add coverageBreakdown to qualify artifacts (per-gate route coverage)
- Add production-style observe example with real sink in docs/observe.md
- Add nightly/staging vs PR gating guidance to docs/qualify.md

- Enrich VerifyFailure with formula-aware diagnostics:
  status:201 => 'HTTP 200', body field checks => actual values
- Remove stale observe CLI activation message
- Document outbound mocks as process-global in getting-started.md
- Refresh APOPHIS_ADOPTION_AUDIT.md with current state

903 tests pass, build clean, typecheck clean.
This commit is contained in:
John Dvorak
2026-05-21 20:39:36 -07:00
parent 55b0262799
commit d0523fcc2d
128 changed files with 4004 additions and 3631 deletions
+64 -15
View File
@@ -2,16 +2,11 @@
Runtime visibility and drift detection without blocking by default.
Observe extends the invariant framework from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) to production environments: contracts run continuously against live traffic to detect behavioral drift without affecting requests.
APOPHIS observe has two paths:
## What Observe Does
1. **CLI `apophis observe`**: Validates observe configuration readiness (policy, sinks, sampling, safety boundaries). Introduces no service process or runtime hooks. Use this for CI config validation before deployment.
`apophis observe` validates your runtime observe configuration:
1. Checks that observe mode is allowed in the current environment
2. Validates reporting sink setup (logs, metrics, traces)
3. Confirms non-blocking semantics
4. Reports what would be observed and why it is safe
2. **Programmatic runtime observation**: Register the APOPHIS plugin with `observe.enabled: true` and `observe.sinks` to emit contract pass/violation/error events from live traffic without blocking responses. Sampling controls the fraction of observed requests.
## When to Use It
@@ -164,18 +159,72 @@ export default {
};
```
## Sink Endpoint Configuration
## Programmatic Runtime Activation
Configure the reporting sink endpoint in your observe config:
The CLI only validates configuration. To activate runtime observation, register
APOPHIS with observe options in your application:
```javascript
observe: {
sink: {
endpoint: 'http://collector.internal:4318'
}
```typescript
import Fastify from 'fastify'
import apophisPlugin from '@apophis/fastify'
const app = Fastify({ logger: true })
// Register APOPHIS with observe enabled.
// This emits non-blocking contract pass/violation/error events
// for every covered request, gated by sampling.
await app.register(apophisPlugin, {
runtime: 'warn',
observe: {
enabled: true,
sampling: 0.1, // observe 10% of requests
sinks: [metricsSink],
},
})
// Implement the ObserveSink interface.
// Capture events to your preferred observability backend.
import type { ObserveSink, ObserveEvent } from '@apophis/fastify'
const metricsSink: ObserveSink = {
emit(event: ObserveEvent) {
// Emit a counter for each contract evaluation
myMetrics.increment(`apophis.contract.${event.type}`, {
route: event.route,
formula: event.formula,
})
// Record duration as a histogram
myMetrics.histogram('apophis.contract.duration_ms', event.durationMs, {
route: event.route,
})
// Log high-signal violations for immediate triage
if (event.type === 'contract.violation') {
logger.warn({ event }, 'APOPHIS contract violation')
}
},
}
```
Key constraints:
- Sink `emit()` can be sync or async (returns `void | Promise<void>`).
- Sink rejections and thrown errors are silently caught — they never affect the route response or status code.
- Sampling is applied per-formula evaluation via `Math.random() < sampling`.
At `sampling: 1` every formula is emitted. At `sampling: 0` nothing is emitted.
- Only routes with APOPHIS annotations (`x-ensures`, `x-requires`) produce events.
Routes without annotations are not evaluated in observe mode.
## Sink Implementations
APOPHIS does not ship with built-in sinks. The `ObserveSink` interface lets you
plug in any backend. Common patterns:
- **OpenTelemetry**: emit counters and histograms via `@opentelemetry/api`.
- **pino logger**: emit structured log records via `pino.info()` / `pino.warn()`.
- **Internal metrics service**: POST events to an internal collector endpoint.
- **In-memory ring buffer**: capture recent events for diagnostics endpoints.
## Monorepo Validation
For monorepos, use `apophis doctor --workspace` to validate observe configuration across all workspace packages. `observe` itself does not support `--workspace`; use `doctor` to check config in each package.