Skip to content

Provider-Agnostic LLM Facade

Intent

Encapsulate large language model invocations behind a unified, provider-agnostic client abstraction paired with a deterministic mock test double to decouple agent orchestration from vendor-specific APIs and enable offline execution.

Context

Applied across agent workflows and orchestration modules where agents execute inference, synthesis, or document reconciliation. In distributed agentic systems, dependencies on external closed-model APIs or proprietary SDKs create provider lock-in, vendor churn, and unreliable test execution in offline or resource-constrained CI/CD environments.

Agentic Profile

Governs autonomous agent steps, flow runners, and semantic merger modules that require LLM interactions. Isolates agent task execution from concrete inference providers, enabling seamless multi-cloud dispatch and zero-cost local test execution.

Forces

Balances multi-model provider flexibility and offline CI testability against the operational discipline required to maintain strict abstraction boundaries.

Drivers

  • Elimination of proprietary vendor SDK lock-in across agent reasoning steps
  • Requirement for zero-cost, hermetic, and deterministic CI/CD testing pipelines
  • Dynamic multi-cloud model routing and token budgeting across diverse execution environments

Constraints

  • Abstract interface contracts must normalize divergent vendor response schemas and error models
  • Test doubles must exhibit behavioural symmetry with live inference endpoints
  • Domain logic must strictly consume the abstraction layer without bypassing it for vendor-specific extensions

Tradeoffs

  • Abstraction overhead and delayed availability of vendor-specific proprietary features
  • Maintenance cost of dual-client parity between production adapters and mock test doubles

Solution

Establish a polymorphic client interface defining standard text generation and structured output contracts. Implement a production gateway adapter utilizing a multi-provider router alongside a symmetric deterministic mock client for offline testing. Agent workflows interact exclusively through the interface protocol, allowing dynamic configuration-driven injection of production or test doubles without code changes.

Operationalisation

Integrate the client interface across all workflow steps and automate runtime selection via environment configuration.

Workflow Implications

  • All workflow runners and step implementations must instantiate client adapters through the base protocol
  • CI pipelines configure mock client test doubles by default to run tests without network credentials or API expenditure
  • Production deployments configure the router adapter with appropriate token budgets, distributed tracing, and retry policies

Automation Opportunities

  • Automated AST verification in CI to detect and reject direct imports of vendor SDKs
  • Continuous fixture generation to keep mock test double response corpora in sync with model schema updates

Implementation Signals

Concrete implementation evidenced by AST symbols in litellm_client.py and mock_client.py, referenced across FlowRunner and AgenticMerger pipelines.

Evidence

Verified recurring usage across FlowRunner and AgenticMerger modules with established AST symbols in production and mock modules.

Signals

  • recurring implementation across FlowRunner and AgenticMerger modules
  • current architecture state AST support in src/cas_runner/llm/litellm_client.py and src/cas_runner/llm/mock_client.py
  • cross-project applicability for LLM-driven orchestration runtimes
  • governance model alignment with deterministic CI test double capabilities

Confidence Basis

Empirically validated through concrete AST symbols, production multi-provider test coverage, and accepted architecture decision record.

Consequences

Provides architectural decoupling, cost control, and deterministic testing while introducing contract maintenance overhead.

Benefits

  • Eliminates direct vendor SDK dependencies in workflow and domain logic
  • Enables fast, zero-overhead, deterministic CI/CD execution without API keys or token costs
  • Provides standardized prompt and response caching, token budgeting, and centralized observability

Risks

  • Vendor abstraction leakage if vendor-specific parameters or error structures are not normalized
  • Drift between mock client canned responses and real provider completion schemas

Governance Implications

  • Enforces token budgeting, distributed tracing, and centralized review of model integrations
  • Mandates that new model capabilities undergo interface normalization before adoption

Reuse Notes

Universally applicable across all CAS components, agent steps, and workflow runners invoking language models. Mandatory for local development, offline CI/CD, and production multi-cloud deployments.

Confidence

High confidence supported by complete codebase implementation, robust AST evidence, and Architecture Forum ratification.

Level: high

Rationale

High confidence based on verified AST implementations across LiteLLMClient and MockLLMClient, proven multi-provider test coverage, and formal ratification by the Architecture Forum.