apophis-fastify/docs/extensions/EXTENSION-ARCHITECTURE.md

# APOPHIS v1.1 Architecture — Hybrid Core + Extensions

> **HISTORICAL**: This is a v1.1 architecture design specification. For current extension documentation, see [EXTENSION-PLUGIN-SYSTEM.md](EXTENSION-PLUGIN-SYSTEM.md) and [QUICK-REFERENCE.md](QUICK-REFERENCE.md).

## Status: Architecture Specification
## Date: 2026-04-24
## Scope: v1.1 First-Class Features & Extension Ecosystem

---

## 1. Philosophy: Core HTTP vs Extensions

**First-class**: Standard HTTP features that require deep integration with APOPHIS core:
- Schema-to-arbitrary integration (teaching fast-check to generate custom data)
- Request builder integration (constructing specialized payloads)
- HTTP executor integration (handling specialized responses)
- APOSTL parser/evaluator integration (new operations)

**Extensions**: Specialized protocols or features with heavy dependencies that should be opt-in:
- Different protocols (WebSockets, not HTTP)
- Heavy dependencies (Protobuf, MessagePack)
- Protocol-specific features such as SSE

**This split keeps common HTTP testing in core while moving specialized protocols out of the default path.**

---

## 2. First-Class Features (v1.1 Core)

### 2.1 Multipart File Uploads

**Module**: Core — `src/infrastructure/multipart.ts`, `src/domain/multipart-generator.ts`

**Schema Annotations**:
```typescript
schema: {
  body: {
    type: 'object',
    'x-content-type': 'multipart/form-data',
    'x-multipart-fields': {
      description: { type: 'string', maxLength: 500 }
    },
    'x-multipart-files': {
      avatar: {
        maxSize: 5 * 1024 * 1024,
        mimeTypes: ['image/jpeg', 'image/png'],
        maxCount: 1
      }
    }
  }
}
```

**APOSTL Operations**:
```typescript
// request_files(this).avatar.count == 1
// request_files(this).avatar.size <= 5242880
// request_files(this).avatar.mimetype matches "image/(jpeg|png)"
// request_fields(this).description != null
```

**Core Integration Points**:
1. **Schema-to-arbitrary**: Detect `x-content-type: multipart/form-data`, generate `{ fields: {...}, files: [...] }`
2. **Request builder**: Convert generated data to `multipart` payload on `RequestStructure`
3. **HTTP executor**: Build `FormData` from `request.multipart`, inject via Fastify
4. **Parser**: Add `request_files`, `request_fields` to `VALID_HEADERS`
5. **Evaluator**: Add multipart operations to `resolveOperation`

### 2.2 Streaming / NDJSON

**Module**: Core — `src/infrastructure/stream-collector.ts`

**Schema Annotations**:
```typescript
schema: {
  response: {
    200: {
      type: 'object',
      'x-streaming': true,
      'x-stream-format': 'ndjson',
      'x-stream-max-chunks': 100,
      'x-stream-timeout': 5000
    }
  }
}
```

**APOSTL Operations**:
```typescript
// response_body(this) — array of parsed chunks
// stream_chunks(this) — alias for response_body(this)
// stream_duration(this) — total stream time in ms
```

**Core Integration Points**:
1. **Contract extraction**: Extract `x-streaming`, `x-stream-format`, `x-stream-max-chunks`, `x-stream-timeout`
2. **HTTP executor**: After inject, check if route has streaming config. If so:
   - Read response payload as string
   - Split by `\n`
   - `JSON.parse` each line (for NDJSON)
   - Respect `maxChunks` and `timeoutMs`
   - Store result in `EvalContext.response.body` and `EvalContext.response.chunks`
3. **Parser**: Add `stream_chunks`, `stream_duration` to `VALID_HEADERS`
4. **Evaluator**: Add streaming operations to `resolveOperation`

---

## 3. Extension System (v1.1+ Ecosystem)

The extension system handles features that don't require core HTTP integration.

### 3.1 Extension Interface

```typescript
export interface ApophisExtension {
  /** Unique name. Used for state isolation and error attribution. */
  name: string
  
  /** APOSTL headers this extension adds. Used for parser validation. */
  headers?: string[]
  
  /** APOSTL predicates exposed by this extension. */
  predicates?: Record<string, PredicateResolver>
  
  /** Lifecycle hooks. */
  hooks?: {
    onBuildRequest?: Hook<RequestBuildContext, void>
    onBeforeRequest?: Hook<ExecutionContext, void>
    onAfterRequest?: Hook<ExecutionContext, void>
    onSuiteStart?: Hook<{ routes: RouteContract[] }, void>
    onSuiteEnd?: Hook<{ summary: TestSummary }, void>
    onViolation?: Hook<{ violation: ContractViolation }, void>
  }
  
  /** Severity: 'fatal' (block test), 'warn' (log, don't block). Default: 'fatal'. */
  severity?: 'fatal' | 'warn'
  
  /** Redaction: fields to mask in violation output. */
  redactFields?: string[]
  
  /** Initial state for this extension. Passed to hooks/predicates. */
  state?: Record<string, unknown>
}
```

### 3.2 Extension Registration

```typescript
await fastify.register(apophis, {
  extensions: [
    sseExtension,
    createSerializerExtension(mySerializerRegistry),
    websocketExtension,
  ]
})
```

### 3.3 Extensions Available

#### SSE Extension
**Module**: `src/extensions/sse/`

```typescript
export const sseExtension: ApophisExtension = {
  name: 'sse',
  headers: ['sse_events'],
  predicates: {
    sse_events: (ctx) => {
      const events = ctx.evalContext.response.sseEvents ?? []
      if (ctx.accessor.length === 0) return { value: events, success: true }
      
      const idx = parseInt(ctx.accessor[0], 10)
      const event = events[idx]
      if (!event) return { value: null, success: true }
      
      if (ctx.accessor[1] === 'event') return { value: event.event, success: true }
      if (ctx.accessor[1] === 'data') return { value: event.data, success: true }
      if (ctx.accessor[1] === 'id') return { value: event.id, success: true }
      if (ctx.accessor[1] === 'retry') return { value: event.retry, success: true }
      
      return { value: event, success: true }
    }
  }
}
```

#### Serializers Extension
**Module**: `src/extensions/serializers/`

```typescript
export interface Serializer {
  readonly name: string
  encode(data: unknown): Buffer
  decode(buffer: Buffer): unknown
}

export interface SerializerRegistry {
  get(name: string): Serializer | undefined
  register(name: string, serializer: Serializer): void
}

export const createSerializerExtension = (registry: SerializerRegistry): ApophisExtension => ({
  name: 'serializers',
  hooks: {
    onBuildRequest: async (ctx) => {
      const serializerName = ctx.route.serializer?.name
      if (!serializerName) return
      
      const serializer = registry.get(serializerName)
      if (!serializer) return
      
      // Modify request: encode body, set content-type
      ctx.request.body = serializer.encode(ctx.request.body)
      ctx.request.headers = {
        ...ctx.request.headers,
        'content-type': `application/x-${serializerName}`,
      }
    },
    onAfterRequest: async (ctx) => {
      const serializerName = ctx.route.serializer?.name
      if (!serializerName) return
      
      const serializer = registry.get(serializerName)
      if (!serializer) return
      
      // Modify response: decode body
      const rawBody = Buffer.from(JSON.stringify(ctx.evalContext.response.body))
      ctx.evalContext.response.body = serializer.decode(rawBody)
    }
  }
})
```

#### WebSockets Extension
**Module**: `src/extensions/websocket/`

**Note**: WebSockets are fundamentally different from HTTP. They require a dedicated runner, not just hooks.

```typescript
export const websocketExtension: ApophisExtension = {
  name: 'websocket',
  headers: ['ws_message', 'ws_state'],
  predicates: {
    ws_message: (ctx) => {
      const msg = ctx.evalContext.ws?.message ?? null
      if (ctx.accessor.length === 0) return { value: msg, success: true }
      if (!msg) return { value: null, success: true }
      
      if (ctx.accessor[0] === 'type') return { value: msg.type, success: true }
      if (ctx.accessor[0] === 'payload') return { value: msg.payload, success: true }
      if (ctx.accessor[0] === 'direction') return { value: msg.direction, success: true }
      
      return { value: msg, success: true }
    },
    ws_state: (ctx) => {
      return { value: ctx.evalContext.ws?.state ?? null, success: true }
    }
  },
  hooks: {
    onSuiteStart: async ({ routes }) => {
      // Pre-validate all WS contracts
      const wsRoutes = routes.filter(r => r.ws !== undefined)
      for (const route of wsRoutes) {
        validateWebSocketContract(route.ws!)
      }
    }
  }
}
```

**WebSocket runner**: Invoked by plugin separately from HTTP runners:
```typescript
// In plugin/index.ts
const buildContract = (fastify, scope) => async (opts) => {
  const httpSuite = await runPetitTests(fastify, opts, scope)
  const wsSuite = await runWebSocketTests(fastify, opts, scope) // From extension
  return mergeSuites(httpSuite, wsSuite)
}
```

---

## 4. Core Changes (Phase 1)

### 4.1 Parser Extensibility

**Current**: `VALID_HEADERS` is hardcoded. Extensions can't add headers.

**Solution**: Extensions register headers. Parser validates against registered + core headers.

```typescript
// src/formula/parser.ts
const CORE_HEADERS: OperationHeader[] = [
  'request_body', 'response_body', 'response_code',
  'request_headers', 'response_headers', 'query_params', 'cookies', 'response_time',
  'redirect_count', 'redirect_url', 'redirect_status',
  'timeout_occurred', 'timeout_value',
  // v1.1 first-class
  'request_files', 'request_fields', 'stream_chunks', 'stream_duration',
]

// ExtensionRegistry provides additional headers
function getValidHeaders(registry?: ExtensionRegistry): string[] {
  const extensionHeaders = registry 
    ? registry.extensions.flatMap(e => e.headers ?? [])
    : []
  return [...CORE_HEADERS, ...extensionHeaders]
}

// In parseOperation, validate against getValidHeaders()
```

### 4.2 Evaluator Extensibility

**Current**: `resolveOperation` checks core operations only.

**Solution**: Check extension predicates BEFORE core operations.

```typescript
function resolveOperation(node, ctx, extensionRegistry, route) {
  const { header, accessor } = node
  
  // 1. Check extension predicates FIRST
  if (extensionRegistry) {
    const resolver = extensionRegistry.resolvePredicate(header)
    if (resolver) {
      const ownerName = extensionRegistry.getPredicateOwner(header)
      const extState = ownerName ? (extensionRegistry.getState(ownerName) ?? {}) : {}
      const result = resolver({ route, evalContext: ctx, accessor: accessor ?? [], extensionState: extState })
      if (result && typeof result.then !== 'function') {
        return (result as PredicateResult).value
      }
    }
  }
  
  // 2. Fall back to core operations
  switch (header) {
    // ... core cases ...
  }
}
```

### 4.3 HTTP Executor Hooks

**Current**: `executeHttp` is a monolithic function.

**Solution**: Add `onTransformResponse` hook point for extensions that need to modify responses.

```typescript
export interface ResponseTransformContext {
  responseBody: unknown
  evalContext: EvalContext
  route: RouteContract
}

export type ResponseTransformHook = (ctx: ResponseTransformContext) => EvalContext | Promise<EvalContext>

// In executeHttp:
let ctx = buildEvalContext(request, response, route)

// Apply extension response transforms
for (const ext of (extensionRegistry?.extensions ?? [])) {
  if (ext.hooks?.onAfterRequest) {
    await ext.hooks.onAfterRequest({
      route,
      request,
      evalContext: ctx,
      extensionState: extensionRegistry?.getState(ext.name) ?? {},
    })
  }
}
```

---

## 5. Implementation Order

### Phase 1: Core Extension Points (1-2 days)
1. Make parser accept registered headers (CORE_HEADERS + extension headers)
2. Make evaluator check extension predicates before core operations
3. Add response transform hook point to HTTP executor
4. **Test**: Core operations still work; extension predicates resolve

### Phase 2A: Multipart (First-Class, 2-3 days)
1. Add `MultipartFile`, `MultipartPayload` types
2. Add multipart schema-to-arbitrary handler
3. Add multipart request builder support
4. Add multipart HTTP executor support (FormData construction)
5. Add `request_files`, `request_fields` to parser/evaluator
6. Extract multipart config from schema in contract.ts
7. **Test**: `src/test/multipart.test.ts` (10+ tests)

### Phase 2B: Streaming (First-Class, 2-3 days)
1. Add `chunks`, `streamDurationMs` to `EvalContext.response`
2. Add streaming config extraction from schema
3. Add stream collection to HTTP executor (NDJSON parsing)
4. Add `stream_chunks`, `stream_duration` to parser/evaluator
5. **Test**: `src/test/streaming.test.ts` (8+ tests)

### Phase 2C: Extension System Polish (1 day)
1. Document extension registration API
2. Add `extensions: ApophisExtension[]` to `ApophisOptions`
3. Wire extension headers into parser
4. Wire extension predicates into evaluator

### Phase 3: Extensions (Parallel, after Phase 2C)
- **SSE Extension** (2-3 days)
- **Serializers Extension** (2-3 days)
- **WebSockets Extension** (1-2 weeks)

### Phase 4: Integration (2-3 days)
1. Run full test suite
2. Update README
3. Verify benchmarks

---

## 6. File Layout

```
src/
  # Core v1.1 First-Class Features
  infrastructure/
    http-executor.ts         # ADD: multipart FormData, stream collection
    multipart.ts            # NEW: FormData construction
    stream-collector.ts     # NEW: NDJSON chunk parsing
  domain/
    schema-to-arbitrary.ts   # ADD: multipart schema handler
    request-builder.ts       # ADD: multipart payload construction
    contract.ts             # ADD: multipart/streaming config extraction
  formula/
    parser.ts               # MODIFY: extensible VALID_HEADERS
    evaluator.ts            # MODIFY: extension predicate check
  types.ts                  # ADD: MultipartFile, MultipartPayload, stream fields
  
  # Extension System
  extension/
    types.ts                # ADD: headers, onTransformResponse to interface
    registry.ts             # ADD: collect extension headers
  
  # Extensions (opt-in)
  extensions/
    sse/                    # SSE extension module
    serializers/            # Serializer extension module
    websocket/              # WebSocket extension module
```

---

## 7. Test Strategy

### First-Class Features: Red-Green-Refactor

```typescript
// Example: Multipart
// 1. Test: Parser accepts request_files(this).avatar.size
// 2. Implement: Add request_files to VALID_HEADERS
// 3. Test: Evaluator resolves request_files
// 4. Implement: Add multipart operations to resolveOperation
// 5. Test: Schema-to-arbitrary generates fake files
// 6. Implement: Add multipart handler to convertSchemaInternal
// 7. Test: Request builder constructs multipart payload
// 8. Implement: Add multipart support to buildRequest
// 9. Test: HTTP executor sends multipart request
// 10. Implement: Build FormData in executeHttp
// 11. Test: Integration — upload route works end-to-end
// 12. Implement: Full flow
```

### Extensions: Self-Contained Tests

Each extension module has its own `test.ts`:

```typescript
// src/extensions/sse/test.ts
import { test } from 'node:test'
import assert from 'node:assert'
import { sseExtension } from './extension.js'

test('sse: predicate returns events', () => {
  const resolver = sseExtension.predicates!.sse_events
  const result = resolver({
    route: mockRoute,
    evalContext: { response: { sseEvents: [{ event: 'update', data: {} }] } },
    accessor: [],
    extensionState: {},
  })
  assert.strictEqual((result.value as any[]).length, 1)
})
```

---

## 8. Backward Compatibility

All v1.1 changes are additive:
- Routes without multipart/streaming annotations work unchanged
- Extensions are opt-in via `extensions: [...]` option
- Existing APOSTL formulas work unchanged
- No breaking changes to public API

**Migration path**:
```typescript
// v1.0
await fastify.register(apophis)

// v1.1 (no changes required for existing code)
await fastify.register(apophis)

// v1.1 with extensions
await fastify.register(apophis, {
  extensions: [sseExtension, serializerExtension, websocketExtension]
})
```

---

## 9. Risk Assessment

| Risk | Mitigation |
|------|-----------|
| Parser changes break existing formulas | Comprehensive regression tests before parser modification |
| Multipart adds heavy deps | Only use native FormData/Blob (no external deps) |
| Streaming tests are flaky | Mock streams for unit tests; integration tests with deterministic timeouts |
| Extension conflicts | Namespacing by extension name; `ExtensionRegistry.getState(name)` isolates state |
| WebSocket extension too large | Split into sub-workstreams: client, runner, stateful, validation |

---

## 10. Success Criteria

| Criterion | Verification |
|-----------|-------------|
| Multipart upload routes tested | `multipart.test.ts` passes |
| Streaming routes tested | `streaming.test.ts` passes |
| Extension predicates work | Extension `test.ts` files pass |
| No regression | Full source and CLI test suites pass |
| Benchmark targets met | `benchmark.test.ts` passes |
| Documentation updated | README covers multipart and streaming |

---

## 11. Quick Reference: First-Class vs Extension

| Feature | Type | Core Files | Tests | Effort |
|---------|------|-----------|-------|--------|
| **Multipart** | First-class | `multipart.ts`, `schema-to-arbitrary.ts`, `request-builder.ts`, `http-executor.ts`, `parser.ts`, `evaluator.ts` | `multipart.test.ts` | 2-3 days |
| **Streaming** | First-class | `stream-collector.ts`, `http-executor.ts`, `parser.ts`, `evaluator.ts`, `contract.ts` | `streaming.test.ts` | 2-3 days |
| **SSE** | Extension | `src/extensions/sse/*` | `src/extensions/sse/test.ts` | 2-3 days |
| **Serializers** | Extension | `src/extensions/serializers/*` | `src/extensions/serializers/test.ts` | 2-3 days |
| **WebSockets** | Extension | `src/extensions/websocket/*` | `src/extensions/websocket/test.ts` | 1-2 weeks |