Skip to content

Provider-Agnostic LLM Gateway

Intent

Decouple agentic reasoning workflows, governance evaluations, and autonomous execution pipelines from downstream model providers through a unified invocation contract backed by symmetric live routing and deterministic mock implementations.

Context

Continuous architecture steering and agentic orchestration pipelines require language model reasoning across multi-step execution stages. Directly coupling runner modules to specific model vendor SDKs introduces API churn fragility, impedes multi-cloud routing, and causes flakiness and prohibitive costs in automated CI test environments.

Agentic Profile

Applies across all autonomous agent execution components, governance evaluation steps, and test suites. Autonomous agents invoke standard prompt completion and structured parsing methods without awareness of physical provider routing or testing context.

Forces

The pattern resolves tensions between provider flexibility, deterministic CI testing requirements, and API churn across heterogeneous model ecosystems.

Drivers

  • Elimination of proprietary model provider SDK coupling in agent reasoning tasks
  • Requirement for hermetic, zero-cost, and deterministic pipeline testing in automated CI
  • Need for transparent routing across multi-cloud and local inference backends

Constraints

  • Disparate provider payload contracts and streaming token semantics must be harmonized
  • Deterministic mock test doubles must faithfully reflect production contract schemas
  • Low-latency invocation overhead must be maintained across model gateway routers

Tradeoffs

  • Additional facade abstraction layer over vendor-specific cutting-edge API features
  • Maintenance overhead of keeping mock client response signatures synchronized with evolving model outputs

Solution

Abstract external model interaction behind a unified gateway facade with two symmetric concrete implementations: a production router (LiteLLMClient) that handles dynamic multi-model routing, retries, and schema normalization; and a deterministic test double (MockLLMClient) that provides canned, predictable responses for hermetic unit testing and CI execution.

Operationalisation

Operationalised by establishing the gateway interface as a mandatory integration boundary for all LLM calls, selecting backend implementations via runtime configuration profiles.

Workflow Implications

  • Agent pipelines instantiate client interfaces via dependency injection driven by runtime profile flags
  • CI pipelines execute in offline mode using MockLLMClient without requiring external network access or API credentials
  • Production deployments route through LiteLLMClient with centralized token and latency instrumentation

Automation Opportunities

  • Automated regression gates verifying mock response schema conformity against real provider schema changes
  • Automated normalisation tests verifying identical streaming semantics across downstream models

Implementation Signals

Verified in core system modules through the existence of LiteLLMClient and MockLLMClient implementations within src/cas_runner/llm/ and unified invocation across FlowRunner and AgenticMerger.

Evidence

Supported by active production components, recurring architecture consolidation signals, and complete offline test suite pass rates.

Signals

  • Consolidation of 19 disparate LLM client proposals into a single canonical gateway design
  • Verified production and offline CI usage in FlowRunner and AgenticMerger modules
  • Active AST verification in src/cas_runner/llm/litellm_client.py and mock_client.py

Supporting Evidence

  • aw-ideas-to-life-cas-2026-06-06-001
  • aw-ideas-to-life-cas-20260605-07

Confidence Basis

Empirical implementation in continuous architecture execution runtime with zero-cost deterministic mock pass in CI.

Consequences

Centralises model governance, isolates reasoning logic from vendor changes, and enables offline deterministic testing, balanced against the need to monitor cross-provider streaming nuances.

Benefits

  • Complete vendor portability allowing seamless transitions between cloud models and local instances
  • Hermetic CI test execution eliminating test suite flakiness, credential leakage, and API execution costs
  • Uniform error handling and structured output parsing across all agent steps

Risks

  • Subtle provider streaming semantic differences across multi-model deployments can cause unexpected edge-case behaviors
  • Potential abstraction leakage if vendor-specific response headers or metadata are relied upon

Governance Implications

  • Mandatory architectural facade for all model interactions across continuous architecture workflows
  • Monitoring and normalisation of streaming semantics required during multi-model deployments

Reuse Notes

Globally reusable across all autonomous agent execution frameworks, multi-agent evaluation suites, and continuous delivery pipelines where LLM reasoning must remain decoupled from specific infrastructure vendors.

Confidence

High confidence based on comprehensive implementation, AST evidence, cross-project applicability, and formal Architecture Forum ratification.

Level: high

Rationale

Pattern is fully implemented in production and mock variants within core runner components, with extensive test coverage and verified CI execution.