Pattern Proposal: Deterministic Mock Evaluation Harness

Summary

Provides a drop-in mock LLM client to execute agent pipelines deterministically in CI/CD without network or model flakiness.

Candidate Pattern

• ID: deterministic-mock-evaluation-harness
• Classification: intelligence
• Maturity: established
• Type: implemented

Observed Signals

No specific signals listed.

Evidence

Supporting Evidence

• src/cas_runner/llm/mock_client.py (MockLLMClient) - Implements deterministic mock LLM interface for testing runner flows.

Scope Assessment

• Evidence Strength: high
• Confidence: high
• Applies to: ideas-to-life-cas

Rationale

• Intent: Enable fast, zero-cost, and deterministic end-to-end testing of agentic workflows and governance decision flows.
• Context: Implemented alongside the production LLM client to support offline unit tests, regression suites, and CI pipelines.
• Force: Live LLM calls in CI are nondeterministic, slow, and incur cost
• Force: Agent pipelines must be validated end-to-end including state transitions and schema validation

Traceability

Artifact source: proposed-patterns