Skip to main content
Polyglot is built on a modular, layered architecture that separates concerns and promotes extensibility. Each layer has a clear responsibility, and dependencies flow in one direction — from the public API down to the HTTP transport. Understanding these layers will help you extend the library, contribute to its development, or build your own integrations with new LLM providers.

​
The Four Layers

​
Public Layer

This is what application code usually touches. Three operation facades provide a stable interface for provider interactions:
  • Inference — for chat completions and text generation
  • Embeddings — for generating vector embeddings
  • Decision — for typed Noul, Choice, and Score judgments
These facades build request objects, delegate execution to runtimes, and return normalized responses regardless of the underlying provider. All three follow an immutable, fluent interface pattern, so configurations can branch safely from a shared base.

​
Runtime Layer

Runtimes assemble the moving parts needed for a provider call. They wire together configuration, a driver, the HTTP client, and event dispatch, and own the operation lifecycle and retry policy. The key classes are:
  • InferenceRuntime — coordinates inference execution and creates PendingInference handles
  • EmbeddingsRuntime — coordinates embeddings execution and creates PendingEmbeddings handles
  • DecisionRuntime — coordinates decision execution and creates PendingDecision handles
Each runtime can be constructed from a config object, a provider, or injected directly. When no HTTP client is provided, the runtime builds a default one via HttpClientBuilder. Runtimes also expose onEvent() and wiretap() methods for hooking into the event system.

​
Request and Response Layer

Requests and responses are normalized into package data objects that are provider-agnostic:
  • InferenceRequest — messages, model, tools, tool choice, response format, options, cached context, retry policy, response cache policy
  • InferenceResponse — one ordered assistant Message, usage, finish reason, and raw HTTP response data
  • PartialInferenceDelta — ordered assistant-message chunks plus finish reason, usage, and provider replay data
  • EmbeddingsRequest — input texts, model, options, retry policy
  • EmbeddingsResponse — vectors and usage
  • DecisionRequest — text or structured state, typed questions, model, retry policy, and telemetry correlation
  • DecisionResponse — typed answers, model, usage, provider request ID, and raw HTTP response data
These objects isolate application code from provider-specific response shapes. Request objects support immutable with*() mutators for building modified copies. PendingInference is the laziness boundary — nothing is sent until you ask for a result. Behind it, InferenceExecutionSession drives one request to one response and divides that job with two per-execution collaborators, both in Inference/Core/:
  • InferenceLifecycleEmitter — every lifecycle event, the execution and attempt stopwatches, the attempt counter, and per-attempt telemetry correlation
  • InferenceRetryLoop — the attempt and length-recovery budgets, backoff delay, and length-recovery request rewriting
There is deliberately no response cache. Calling response() repeatedly returns the identical instance because the session reads it back off the InferenceExecution it already holds — which works for every ResponseCachePolicy, not only Memory. ResponseCachePolicy still matters, but at the HTTP layer: it becomes the StreamCachePolicy that decides whether a stream is replayable. See lifecycle.md for the event sequence each path emits.

​
Driver Layer

Drivers translate Polyglot requests into provider-native HTTP payloads and normalize the results back. Drivers implement CanProcessInferenceRequest, CanHandleVectorization, or CanProcessDecisionRequest for their operation family. Inference drivers are further composed from these responsibilities:
  • Request adapters (CanTranslateInferenceRequest) — convert InferenceRequest into an HttpRequest
  • Response adapters (CanTranslateInferenceResponse) — convert raw HttpResponse data into InferenceResponse or stream of PartialInferenceDelta
  • Message formatters (CanMapMessages) — map typed Messages to provider-specific structures, composing a MessageMapper utility for iteration
  • Body formatters (CanMapRequestBody) — assemble the full request body with mode-specific adjustments
  • Usage formatters (CanMapUsage) — extract token usage from provider responses
All inference drivers extend BaseInferenceRequestDriver, which provides the standard HTTP execution flow and stream handling. Every bundled provider is declared as an InferenceDriverSpec — a row naming its adapters and formatters — and built by that shared driver. Providers that assemble their own URL or headers select bespoke request adapters in that row; the provider-specific behavior does not require a driver class. Decision drivers use DecisionRequestAdapter and DecisionResponseAdapter to translate between the stable Decision domain and provider wire formats. TypeSafe, Jeff, and Laya use the System One shape, while classifier.dev uses its dimensions API. See Configuration and runtime for the registry contract.

​
Shared Support

Inference, embeddings, and Decision are separate operation families, but a few primitives belong to none of them. They live under Cognesy\Polyglot\Support\:
  • Support\Redaction\SensitiveDataRedactor — decides what is sensitive: which header names, query parameters and option keys are masked before a value can reach an event payload or an exception message.
  • Support\Redaction\RedactsHttpPayloads — decides where the drivers apply it. BaseInferenceRequestDriver and BaseEmbedDriver both use this trait, so a tightened rule takes effect on both sides at once. Redaction runs on error paths only; nothing here executes on a successful request.
  • Support\Retry\RetryBackoff, RetryJitter, RetryPolicyInvariants — the delay computation, jitter strategies, and constructor invariants shared by inference, embeddings, and Decision retry policies.
  • Support\Pricing\Cost — the value object every cost calculator returns, on both sides. Note that the calculators themselves are not shared: Inference\Pricing\FlatRateCostCalculator prices five token categories and Embeddings\Pricing\FlatRateCostCalculator prices one, and InferencePricing and EmbeddingsPricing carry different rate fields. They share a name, not a signature.
The retry and redaction classes previously lived under Inference\, which forced other operation families to import from the inference subsystem. Cost was already neutral, at a top-level Polyglot\Pricing\; it moved here so that Support\ is the package’s single answer to “where do things no operation family owns live?” rather than one of two competing conventions.

​
How the Layers Connect

The public facade creates a request and hands it to the runtime. The runtime delegates to a driver, which translates the request into an HTTP call and normalizes the response. Events are dispatched at each stage for observability. The result flows back up as a normalized data object.

​
Key Design Decisions

Immutability. The public facades and request/response objects are immutable. Calling methods such as withMessages(), withInput(), or withModel() returns a new instance rather than modifying the original. Lazy execution. Calling create() returns a PendingInference, PendingEmbeddings, or PendingDecision handle without triggering the HTTP call. Execution begins when the application reads from the handle. Driver registry. Inference and Decision use InferenceDriverRegistry and DecisionDriverRegistry; embeddings use EmbeddingsDriverFactory. Each operation family supports custom runtime wiring. Provider-agnostic data. InferenceResponse::message() preserves one ordered assistant turn regardless of provider. Text, reasoning, and tool calls are projections of that message; provider-specific HTTP details remain available through responseData(). EmbeddingsResponse provides the corresponding normalized vector surface for embeddings. DecisionResponse provides typed answers and preserves the provider request ID and normalized HTTP response for diagnostics.