Skip to content

ADR: Adopt Hybrid LLM Gateway Integration

Context

The CAS system currently uses a stubbed LLM provider. To support real-world architectural flows, we need to integrate with multiple providers (Gemini, OpenAI, etc.) while ensuring that outputs are machine-readable and system stability is maintained.

Decision

We will implement a unified LLM Gateway in src/cas_runner/llm/ that: 1. Defines a provider-agnostic LLMClient protocol. 2. Uses LiteLLM for normalized multi-provider execution. 3. Mandates Pydantic schemas for all reasoning steps. 4. Implements robust JSON extraction to handle model "chatter".

Consequences

  • Pros: Provider agnosticism, strict type safety, reduced boilerplate for adding new models.
  • Cons: Adds litellm and pydantic as core dependencies.
  • Risks: Potential for validation failures if models deviate from complex schemas.