fix: harden engine, enrich failure diagnostics, close adoption gaps
- P0: CLI verify now honors test budget with seeded multi-sample - P0: Observe sampling enforced via Math.random() gate in hook-validator - P1: Remove misleading undici-mock-agent isolation option - P1: Qualify reuses shared discoverRouteDetails() with warnings - P1: Chaos/scenario config exposed via preset schema - P1: README/docs limitations updated to current state - P2: Nested response annotations prefer 2xx deterministically - P2: --changed documented as heuristic in verify.md - Add observe sink tests (sampling 0/1, sink failure non-interference) - Add verify runs regression tests (scale, determinism, variants) - Add configured-scenario qualify test (independent of OAuth fixture) - Add coverageBreakdown to qualify artifacts (per-gate route coverage) - Add production-style observe example with real sink in docs/observe.md - Add nightly/staging vs PR gating guidance to docs/qualify.md - Enrich VerifyFailure with formula-aware diagnostics: status:201 => 'HTTP 200', body field checks => actual values - Remove stale observe CLI activation message - Document outbound mocks as process-global in getting-started.md - Refresh APOPHIS_ADOPTION_AUDIT.md with current state 903 tests pass, build clean, typecheck clean.
This commit is contained in:
+64
-15
@@ -2,16 +2,11 @@
|
||||
|
||||
Runtime visibility and drift detection without blocking by default.
|
||||
|
||||
Observe extends the invariant framework from [Invariant-Driven Automated Testing](https://arxiv.org/abs/2602.23922) (Malhado Ribeiro, 2021) to production environments: contracts run continuously against live traffic to detect behavioral drift without affecting requests.
|
||||
APOPHIS observe has two paths:
|
||||
|
||||
## What Observe Does
|
||||
1. **CLI `apophis observe`**: Validates observe configuration readiness (policy, sinks, sampling, safety boundaries). Introduces no service process or runtime hooks. Use this for CI config validation before deployment.
|
||||
|
||||
`apophis observe` validates your runtime observe configuration:
|
||||
|
||||
1. Checks that observe mode is allowed in the current environment
|
||||
2. Validates reporting sink setup (logs, metrics, traces)
|
||||
3. Confirms non-blocking semantics
|
||||
4. Reports what would be observed and why it is safe
|
||||
2. **Programmatic runtime observation**: Register the APOPHIS plugin with `observe.enabled: true` and `observe.sinks` to emit contract pass/violation/error events from live traffic without blocking responses. Sampling controls the fraction of observed requests.
|
||||
|
||||
## When to Use It
|
||||
|
||||
@@ -164,18 +159,72 @@ export default {
|
||||
};
|
||||
```
|
||||
|
||||
## Sink Endpoint Configuration
|
||||
## Programmatic Runtime Activation
|
||||
|
||||
Configure the reporting sink endpoint in your observe config:
|
||||
The CLI only validates configuration. To activate runtime observation, register
|
||||
APOPHIS with observe options in your application:
|
||||
|
||||
```javascript
|
||||
observe: {
|
||||
sink: {
|
||||
endpoint: 'http://collector.internal:4318'
|
||||
}
|
||||
```typescript
|
||||
import Fastify from 'fastify'
|
||||
import apophisPlugin from '@apophis/fastify'
|
||||
|
||||
const app = Fastify({ logger: true })
|
||||
|
||||
// Register APOPHIS with observe enabled.
|
||||
// This emits non-blocking contract pass/violation/error events
|
||||
// for every covered request, gated by sampling.
|
||||
await app.register(apophisPlugin, {
|
||||
runtime: 'warn',
|
||||
observe: {
|
||||
enabled: true,
|
||||
sampling: 0.1, // observe 10% of requests
|
||||
sinks: [metricsSink],
|
||||
},
|
||||
})
|
||||
|
||||
// Implement the ObserveSink interface.
|
||||
// Capture events to your preferred observability backend.
|
||||
import type { ObserveSink, ObserveEvent } from '@apophis/fastify'
|
||||
|
||||
const metricsSink: ObserveSink = {
|
||||
emit(event: ObserveEvent) {
|
||||
// Emit a counter for each contract evaluation
|
||||
myMetrics.increment(`apophis.contract.${event.type}`, {
|
||||
route: event.route,
|
||||
formula: event.formula,
|
||||
})
|
||||
|
||||
// Record duration as a histogram
|
||||
myMetrics.histogram('apophis.contract.duration_ms', event.durationMs, {
|
||||
route: event.route,
|
||||
})
|
||||
|
||||
// Log high-signal violations for immediate triage
|
||||
if (event.type === 'contract.violation') {
|
||||
logger.warn({ event }, 'APOPHIS contract violation')
|
||||
}
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
Key constraints:
|
||||
- Sink `emit()` can be sync or async (returns `void | Promise<void>`).
|
||||
- Sink rejections and thrown errors are silently caught — they never affect the route response or status code.
|
||||
- Sampling is applied per-formula evaluation via `Math.random() < sampling`.
|
||||
At `sampling: 1` every formula is emitted. At `sampling: 0` nothing is emitted.
|
||||
- Only routes with APOPHIS annotations (`x-ensures`, `x-requires`) produce events.
|
||||
Routes without annotations are not evaluated in observe mode.
|
||||
|
||||
## Sink Implementations
|
||||
|
||||
APOPHIS does not ship with built-in sinks. The `ObserveSink` interface lets you
|
||||
plug in any backend. Common patterns:
|
||||
|
||||
- **OpenTelemetry**: emit counters and histograms via `@opentelemetry/api`.
|
||||
- **pino logger**: emit structured log records via `pino.info()` / `pino.warn()`.
|
||||
- **Internal metrics service**: POST events to an internal collector endpoint.
|
||||
- **In-memory ring buffer**: capture recent events for diagnostics endpoints.
|
||||
|
||||
## Monorepo Validation
|
||||
|
||||
For monorepos, use `apophis doctor --workspace` to validate observe configuration across all workspace packages. `observe` itself does not support `--workspace`; use `doctor` to check config in each package.
|
||||
|
||||
Reference in New Issue
Block a user