Skip to content

Provider-Agnostic LLM Client Abstraction

Intent

Decouple agent execution pipelines and reasoning workflows from concrete model provider SDKs through a unified client abstraction paired with deterministic test doubles, guaranteeing vendor portability, standardized telemetry, and zero-cost offline testability.

Context

Language model provider SDKs undergo frequent updates, breaking changes, and divergent invocation semantics across vendors (e.g. OpenAI, Anthropic, Ollama). Autonomous agent execution runtimes and continuous architecture pipelines require reliable, uniform model access without being tightly coupled to specific upstream APIs or incurring live API execution costs during continuous integration.

Agentic Profile

Autonomous reasoning agents, prompt-driven evaluation steps, and workflow mergers require model completion and structured output interfaces that isolate agent business logic from transport, authentication, rate limiting, and vendor error translation.

Forces

Balances provider portability, consistent operational invocation semantics, and zero-cost offline testing against the maintenance of an abstraction layer.

Drivers

  • High churn and divergent invocation semantics across evolving vendor SDKs
  • Requirement for zero-cost, deterministic offline execution in CI/CD pipelines
  • Need for standardized token tracking, error normalization, and structured output parsing

Constraints

  • All platform components must interface with LLMs through the ratified provider-agnostic abstraction
  • Direct usage of vendor-specific SDKs across agent execution pipelines is prohibited

Tradeoffs

  • Abstraction overhead when exposing cutting-edge vendor-specific experimental capabilities
  • Need to maintain interface parity between production router adapters and mock test doubles

Solution

Implement a dual-tier client abstraction conforming to a uniform interface. Production flows invoke a unified multi-model routing adapter (LiteLLMClient) configured with centralized policies, while testing and offline validation flows substitute a deterministic in-memory double (MockLLMClient) to deliver reproducible responses without external network access.

Operationalisation

Integrate the unified client contract across FlowRunner steps, governance validators, and agentic mergers, parameterising client selection via execution profiles.

Workflow Implications

  • FlowRunner and governance steps utilize LiteLLMClient in operational runs and switch to MockLLMClient for deterministic CI/CD execution
  • All new agent steps must consume the standardized LLM client protocol rather than direct provider bindings

Automation Opportunities

  • Automated static analysis to block direct vendor SDK imports across all repository modules
  • Automated recording of production invocation traces to refresh deterministic test doubles

Implementation Signals

Verified presence of LiteLLMClient in src/cas_runner/llm/litellm_client.py, MockLLMClient in src/cas_runner/llm/mock_client.py, and protocol adherence in FlowRunner and AgenticMerger.

Evidence

Ratified Reference Architecture in ArchGov with verified implementations across LiteLLMClient and MockLLMClient test doubles.

Signals

  • Consolidation of 19 redundant provider adapter proposals into a single canonical implementation
  • AST symbols in src/cas_runner/llm/litellm_client.py and src/cas_runner/llm/mock_client.py

Supporting Evidence

  • Ratified Reference Architecture in ArchGov

Confidence Basis

Empirical implementation in core platform runner with verified 100% test pass rate using MockLLMClient in CI validation pipelines.

Consequences

Eliminates vendor SDK coupling, standardizes token usage and error handling, and enables zero-cost deterministic offline execution for integration tests.

Benefits

  • Total decoupling from vendor-specific LLM APIs and SDK breaking changes
  • Zero-cost deterministic offline execution in CI/CD pipelines
  • Standardized error normalization, retries, and token accounting across all steps

Risks

  • Potential abstraction leakage if vendor-specific response fields or error structures are not normalized
  • Latency or feature gaps when cutting-edge proprietary model features emerge

Governance Implications

  • All platform components must interface with LLMs through the ratified provider-agnostic abstraction; direct usage of vendor-specific SDKs is prohibited
  • New agent workflows must provide verified mock fixtures compatible with MockLLMClient

Reuse Notes

Global architectural pattern applicable across all agent execution pipelines, LLM invocations, and test suites within ideas-to-life-cas and downstream projects adopting CAS governance.

Confidence

High confidence supported by ratified Reference Architecture and established operational code.

Level: high

Rationale

Verified production implementations in LiteLLMClient and MockLLMClient test doubles with complete test suite validation and Architecture Forum ratification.