feat: plugin contract e2e, qualify --changed, production observe, regressions

2026-05-22 11:05:52 -07:00
parent d0523fcc2d
commit 1de735ee08
34 changed files with 1392 additions and 122 deletions
@@ -0,0 +1,431 @@
+# Plugin Contracts
+
+Define behavioral rules that apply across all matching routes and are automatically injected into every matching route's contract evaluation.
+
+Plugin contracts let teams codify cross-cutting expectations — "all API routes must carry an auth header," "all POST handlers must close the database connection," "all responses must include a request ID." They're defined once in config and evaluated alongside route-level contracts without per-route boilerplate.
+
+## Table of Contents
+
+- [When to Use It](#when-to-use-it)
+- [Config Format](#config-format)
+- [PluginContractSpec Reference](#plugincontractspec-reference)
+- [Route Pattern Matching](#route-pattern-matching)
+- [Precondition → Skip Semantics](#precondition--skip-semantics)
+- [Auto-Injected Headers](#auto-injected-headers)
+- [Failure Attribution](#failure-attribution)
+- [Extension Dependencies](#extension-dependencies)
+- [Production Safety](#production-safety)
+- [Complete Example](#complete-example)
+- [Limitations](#limitations)
+
+## When to Use It
+
+Plugin contracts are ideal for rules that span many routes and don't belong on a single handler:
+
+- Auth header presence across all API routes
+- Response metadata (request IDs, timing headers, version stamps)
+- Tenant isolation checks that apply to every route in a prefix
+- Database connection lifecycle guards on write endpoints
+- Content-type enforcement on all response paths
+
+Don't use plugin contracts for route-specific behavioral rules — those belong on the route schema via `x-requires` and `x-ensures`.
+
+## Config Format
+
+### apophis.config.js
+
+```javascript
+export default {
+  pluginContracts: {
+    "auth-presence": {
+      appliesTo: "/api/**",
+      hooks: {
+        onRequest: {
+          requires: ["request_headers(this).authorization != null"]
+        }
+      }
+    },
+    "rate-limit-respected": {
+      appliesTo: "POST /api/**",
+      hooks: {
+        onResponse: {
+          ensures: ["status != 429"]
+        }
+      },
+      meta: { name: "rate-limit-respected", description: "Rate limit should not be hit under normal CI load" }
+    }
+  }
+}
+```
+
+### Programmatic (Fastify plugin registration)
+
+```javascript
+await app.register(apophisPlugin, {
+  pluginContracts: {
+    "auth-presence": {
+      appliesTo: "/api/**",
+      hooks: {
+        onRequest: { requires: ["request_headers(this).authorization != null"] }
+      }
+    }
+  }
+})
+```
+
+Contracts registered programmatically take the same path through the plugin contract registry as config-defined contracts. Both sources are merged into the composed contract at evaluation time.
+
+## PluginContractSpec Reference
+
+| Field | Required | Type | Description |
+|---|---|---|---|
+| `appliesTo` | yes | `string` | Route pattern (see [Route Pattern Matching](#route-pattern-matching)) |
+| `hooks` | yes | `object` | Hook-phase contracts keyed by Fastify lifecycle hook name |
+| `hooks.<phase>.requires` | no | `string[]` | APOSTL formulas that must be `true` **before** this phase executes |
+| `hooks.<phase>.ensures` | no | `string[]` | APOSTL formulas that must be `true` **after** this phase executes |
+| `meta` | no | `object` | Display metadata |
+| `meta.name` | no | `string` | Human-readable plugin name |
+| `meta.version` | no | `string` | Plugin version |
+| `meta.description` | no | `string` | Description of what the plugin enforces |
+| `extensions` | no | `array` | APOPHIS extensions this plugin depends on |
+
+### Supported hook phases
+
+Keys in `hooks` correspond to Fastify lifecycle hook names:
+
+- `onRequest` — evaluated before the route handler runs
+- `preHandler` — evaluated before the handler, after shared hooks
+- `preSerialization` — evaluated before the response body is serialized
+- `onSend` — evaluated before the response is sent
+- `onResponse` — evaluated after the response is sent (body already consumed)
+
+At least one of `requires` or `ensures` must be present per phase. Empty phases are ignored.
+
+### Extension dependencies
+
+```javascript
+pluginContracts: {
+  "jwt-auth": {
+    appliesTo: "/api/**",
+    extensions: [{ name: "jwt-decoder", required: true }],
+    hooks: {
+      onRequest: { requires: ["decode_jwt(this).valid == true"] }
+    }
+  }
+}
+```
+
+If a required extension is missing at registration time, the plugin's contracts are skipped with a warning. Extensions are resolved from the APOPHIS extension registry at evaluation time, not at route discovery time.
+
+## Route Pattern Matching
+
+The `appliesTo` field controls which routes a plugin contract covers.
+
+### Pattern syntax
+
+| Pattern | Matches | Does NOT match |
+|---|---|---|
+| `"/api/users"` | `/api/users` (exact) | `/api/users/42`, `/api/v2/users` |
+| `"/api/**"` | `/api/users`, `/api/users/42`, `/api/users/:id/posts` | `/v2/api/users` |
+| `"/api/*"` | `/api/users`, `/api/status` | `/api/users/42`, `/api/users/:id/posts` |
+| `"**"` | Everything | — |
+| `"POST /api/**"` | `POST /api/users`, `POST /api/orders/42` | `GET /api/users`, `POST /v2/api/users` |
+
+The `/**` suffix matches any path at any depth below the prefix. The `/*` suffix matches exactly one segment below the prefix.
+
+### Method-prefixed patterns
+
+Prepend the HTTP method to restrict a pattern to a specific verb:
+
+```javascript
+{
+  appliesTo: "POST /api/**",
+  hooks: {
+    preHandler: { requires: ["request_body(this) != null"] }
+  }
+}
+```
+
+When a method is specified, the pattern only matches routes registered with that HTTP method. The method must be a valid HTTP method in uppercase.
+
+Pattern matching is evaluated against the route's path string only (the raw argument passed to `fastify.get()`, `fastify.post()`, etc.), not against the dereferenced URL with prefix applied. Fastify prefix application happens before pattern matching, so a route registered as `fastify.get("/users")` under a `prefix: "/api"` is matched against `/api/users`.
+
+## Precondition → Skip Semantics
+
+When a plugin contract defines `requires` formulas, the runner evaluates them before `ensures`. If any require formula fails:
+
+1. Require failure is **not** counted as a contract failure
+2. All `ensures` formulas for that route+run are **skipped**
+3. The skip is counted separately from failures in the artifact summary
+
+This distinguishes "the plugin's prerequisites aren't met" from "the route's behavior violated the contract."
+
+### Example
+
+```javascript
+pluginContracts: {
+  "tenant-check": {
+    appliesTo: "/api/**",
+    hooks: {
+      onRequest: {
+        requires: ["request_headers(this).x-tenant-id != null"],
+        ensures: ["status != 500"]
+      }
+    }
+  }
+}
+```
+
+For a route without the `x-tenant-id` header:
+- The `requires` check fails → the `ensures` (`status != 500`) is **skipped**
+- `skippedCount` increments; `failedCount` does not
+
+For a route with the `x-tenant-id` header:
+- The `requires` check passes → the `ensures` is evaluated normally
+- If `status == 500`, it's counted as a plugin contract **failure**
+
+## Auto-Injected Headers
+
+To avoid every test being skipped when a plugin contract checks for header presence, APOPHIS automatically injects headers into test requests when the precondition pattern is simple enough.
+
+### Supported patterns
+
+| Formula pattern | Injected header |
+|---|---|
+| `request_headers(this).<name> != null` | `<name>: test-value` |
+| `request_headers(this).<name> == "value"` | `<name>: value` |
+
+### Example
+
+```javascript
+pluginContracts: {
+  "auth-presence": {
+    appliesTo: "/api/**",
+    hooks: {
+      onRequest: {
+        requires: ["request_headers(this).authorization != null"]
+      }
+    }
+  }
+}
+```
+
+APOPHIS detects the `!= null` pattern on `authorization` and injects `authorization: test-value` into every test request. The precondition passes automatically without user intervention.
+
+Auto-injection works in all runners: verify, qualify (scenario, stateful, chaos), and replay.
+
+### Unsupported patterns
+
+Complex preconditions are not auto-injected. For these, the requires check will fail and the route's ensures will be skipped:
+
+```javascript
+// NOT auto-injected — the route's ensures will be skipped
+requires: ["request_headers(this).x-count >= 5"]
+```
+
+For these cases, provide the header explicitly via route variants:
+
+```javascript
+await fastify.apophis.contract({
+  runs: 5,
+  variants: [{ name: "with-count", headers: { "x-count": "10" } }]
+})
+```
+
+## Failure Attribution
+
+When a plugin contract formula fails, the violation carries metadata that distinguishes it from route-level failures.
+
+### ContractViolation fields
+
+| Field | Value | Meaning |
+|---|---|---|
+| `source` | `"plugin:auth-presence"` | Which plugin added the formula |
+| `phase` | `"onRequest"` | Which hook phase the formula belongs to |
+
+In human-readable output, plugin failures appear separately from route contract failures:
+
+```text
+Plugin contract violation (plugin:auth-presence)
+GET /api/users
+Phase: onRequest
+
+Expected
+  response_headers(this).x-request-id != null
+
+Observed
+  response_headers(this).x-request-id was null
+```
+
+The artifact summary tracks plugin contract outcomes independently:
+
+```json
+{
+  "summary": {
+    "pluginContractsApplied": 12,
+    "pluginContractsFailed": 2,
+    "failed": 1,
+    "passed": 10
+  }
+}
+```
+
+- `pluginContractsApplied` — total plugin contract formulas evaluated across all routes
+- `pluginContractsFailed` — plugin contract formulas that failed
+- `failed` — total failures (route + plugin)
+- `passed` — route-level contract passes (plugin-only formulas not counted here)
+
+A route can pass its own contracts and still have plugin contract failures. The CLI exit code reflects the aggregate: any failure (route or plugin) produces exit code 1.
+
+## Extension Dependencies
+
+Plugin contracts can declare dependencies on APOPHIS extensions. Extension resolution happens at evaluation time via the extension registry.
+
+### Declaring dependencies
+
+```javascript
+pluginContracts: {
+  "jwt-auth": {
+    appliesTo: "/api/**",
+    extensions: [{ name: "jwt-decoder", required: true }],
+    hooks: {
+      onRequest: { requires: ["decode_jwt(this).valid == true"] }
+    }
+  }
+}
+```
+
+Each extension reference has:
+- `name` (required) — matches the extension's registered name
+- `required` (optional, default `true`) — if `false`, missing extension emits a warning but does not skip the plugin
+
+### Missing extension behavior
+
+If a required extension is not registered, the plugin's contracts are **skipped** for all matching routes. Warnings are collected in the registry and surfaced via `drainWarnings()`:
+
+```text
+Plugin 'jwt-auth' requires extensions [jwt-decoder] which are not registered. Skipping its contracts.
+```
+
+Warnings appear in the CLI output and the artifact's `warnings` array.
+
+### Extension registration
+
+Extensions are registered alongside plugin contracts:
+
+```javascript
+await app.register(apophisPlugin, {
+  extensions: [
+    {
+      name: "jwt-decoder",
+      headers: ["decode_jwt"],
+      predicates: {
+        decode_jwt: (ctx) => ({ value: true, success: true })
+      }
+    }
+  ],
+  pluginContracts: {
+    "jwt-auth": {
+      appliesTo: "/api/**",
+      extensions: [{ name: "jwt-decoder", required: true }],
+      hooks: {
+        onRequest: { requires: ["decode_jwt(this).valid == true"] }
+      }
+    }
+  }
+})
+```
+
+## Production Safety
+
+`pluginContracts` is a **test-only** feature. If `pluginContracts` is non-empty in a production environment (`NODE_ENV=production`), APOPHIS throws at plugin registration:
+
+```text
+APOPHIS: Unsafe options detected in production: pluginContracts.
+These features are test-only and must not be enabled in production.
+Remove them from ApophisOptions or set NODE_ENV=test.
+```
+
+This is enforced by `validateProductionSafety()` before any contract evaluation runs. There is no escape hatch.
+
+## Complete Example
+
+### Config (`apophis.config.js`)
+
+```javascript
+export default {
+  profiles: {
+    quick: { preset: "safe-ci" },
+  },
+  presets: {
+    "safe-ci": { runs: 10 },
+  },
+  pluginContracts: {
+    "auth-required": {
+      appliesTo: "/api/**",
+      hooks: {
+        onRequest: {
+          requires: ["request_headers(this).authorization != null"],
+        },
+      },
+      meta: { name: "auth-required", description: "All API routes require an Authorization header" },
+    },
+    "request-id": {
+      appliesTo: "/api/**",
+      hooks: {
+        onSend: {
+          ensures: ["response_headers(this).x-request-id != null"],
+        },
+      },
+      meta: { name: "request-id", description: "All API responses must include x-request-id" },
+    },
+  },
+}
+```
+
+### Route (`routes/users.js`)
+
+```javascript
+app.get("/api/users", {
+  schema: {
+    "x-category": "observer",
+    "x-ensures": ["status:200", "response_body(this) is Array"],
+    response: {
+      200: {
+        type: "array",
+        items: { type: "object", properties: { id: { type: "number" } } },
+      },
+    },
+  },
+}, async (request, reply) => {
+  return [{ id: 1 }, { id: 2 }]
+})
+```
+
+### Execution (`apophis verify`)
+
+1. Sends a request to `GET /api/users`
+2. `auth-required` plugin: `onRequest.requires` → `request_headers(this).authorization != null` → APOPHIS auto-injects `authorization: test-value` → passes
+3. Route-level ensures: `status:200` → passes; `response_body(this) is Array` → passes
+4. `request-id` plugin: `onSend.ensures` → `response_headers(this).x-request-id != null` → if the handler doesn't set `x-request-id`, this **fails** as a `plugin:request-id` contract violation
+
+```text
+Plugin contract violation (plugin:request-id)
+GET /api/users
+Phase: onSend
+
+Expected
+  response_headers(this).x-request-id != null
+
+Observed
+  response_headers(this).x-request-id was null
+```
+
+## Limitations
+
+- **`appliesTo` matches by route path only** by default. Use method-prefixed syntax (`"POST /api/**"`) for HTTP-method-specific rules.
+- **Header auto-injection only handles simple patterns** (`!= null` and `== "value"`). Complex preconditions like `>= 5` are not auto-injected; the route's ensures will be skipped.
+- **No per-route opt-out.** If a plugin contract matches a route, it always applies. Use narrower `appliesTo` patterns or separate route prefixes to exclude routes.
+- **Extension resolution is lazy.** Extensions are checked at route evaluation time, not at registration time. A missing extension is only discovered when a matching route is evaluated.
+- **At least one of `requires` or `ensures` must be present per phase.** A phase with neither is silently ignored.
+- **Runtime hook ordering follows Fastify's lifecycle.** Plugin contracts in `onResponse` cannot inspect the response body (it has already been sent to the client).