Skip to content

Inference runtime

Inference runtime

Inference is a model call with a schema-checked result. This page is how those calls execute internally: the binding, structured output, streaming, and where the transcript lives.

The ai binding

Calls go through the ai facility port — a host-supplied binding with the concrete provider wired at runtime. The runtime never talks to a provider directly and never holds provider credentials.

Structured output

The runtime asks the model for output conforming to a schema and validates it on the way back. A malformed result is retried or surfaced as an error — it is never passed to tenant code untyped.

  prompt + schema
        │
        ▼
  ┌────────────────────────────┐
  │  ai facility binding       │
  │  (host provider at runtime)│
  └────────────┬───────────────┘
               │  model output
               ▼
  schema validation
        │
        ├── valid ──► typed result to caller
        │
        └── invalid ──► retry or error (never untyped)

Streaming

Inference is facility I/O inside the current invocation. The guest yields while the host calls the model, then the same invocation resumes in the same worker. The worker and capacity slot stay assigned during the wait, but that wait is not metered as worker CPU. Model tokens, cost, and turns are metered on the conversation row.

Session storage

Every conversation is persisted in one tenant-owned chat_session aggregate. Ordered messages, nested turns, title, status and metered usage sync and audit together.

The authoring surface is Infer ; the agentic loop that uses this runtime lives under Agent loop.