Skip to content

Deterministic Mock Evaluation Harness

Intent

Isolate agentic workflows, evaluation harnesses, and automated governance decision gates from external model providers during testing by using a drop-in deterministic mock client, ensuring zero-cost, hermetic, and reproducible execution.

Context

Automated agent pipelines, continuous architecture verification gates, and evaluation suites depend on non-deterministic external LLM responses, introducing network latency, unpredictable failure modes, API quota limits, and financial costs into continuous delivery workflows.

Agentic Profile

Applies to evaluation agents, test harnesses, and continuous verification runners that require deterministic input-output simulation to assert lifecycle transitions, schema compliance, and decision flows without live model calls.

Forces

Continuous automated verification demands speed, idempotency, and zero operational friction, which conflicts with the cost, latency, rate limits, and variance of live external model APIs.

Drivers

  • Requirement for hermetic, fast, and repeatable CI/CD test gates
  • Elimination of token consumption costs and API quotas during routine testing
  • Idempotent verification of agent state transitions and governance rules

Constraints

  • Mock responses must conform strictly to production schema and domain contracts
  • Test doubles must maintain interface parity with production client adapters

Tradeoffs

  • Mock harness does not evaluate live LLM semantic drift or novel prompt nuances
  • Requires maintaining synthetic response fixtures synchronized with schema evolutions

Solution

Implement a drop-in mock LLM client adapter that mirrors the production interface contract, serving canned, deterministic, and schema-valid responses to enable offline, zero-latency validation of orchestration and governance pipelines.

Operationalisation

Standardize client adapter dependency injection across test doubles and CI pipelines to automatically substitute live LLM calls with the deterministic evaluation harness.

Workflow Implications

  • CI/CD regression and contract validation suites default to the deterministic mock client
  • Developer local testing environments execute offline without requiring cloud provider credentials
  • Pre-commit and pull-request checks execute reliably without network flakiness

Automation Opportunities

  • Automate generation of mock response fixtures from verified production execution traces
  • Integrate mock harness into continuous architecture gates to block PRs violating domain schemas

Implementation Signals

Adoption is signaled by the presence of test-double client implementations adhering to standardized LLM interfaces, such as MockLLMClient in src/cas_runner/llm/mock_client.py, paired with CI configurations executing hermetic verification runs.

Evidence

Validated through reference architecture ratification in ArchGov and active implementation of MockLLMClient and LiteLLMClient in the continuous architecture steering codebase.

Signals

  • Ratified Reference Architecture in ArchGov
  • Verified test double implementation in src/cas_runner/llm/mock_client.py
  • Zero external API dependencies in CI test suite execution

Confidence Basis

High confidence backed by ratified reference architecture in ArchGov, production code AST validation, and passing test double suites.

Consequences

Adopting the deterministic mock evaluation harness decouples testing and verification from third-party model providers, enabling reliable continuous architecture pipelines while shifting prompt variation testing to specialized evaluation stages.

Benefits

  • Eliminates network latency, provider downtime, and rate-limiting failures in CI/CD
  • Zero financial cost for routine continuous architecture pipeline executions
  • Guarantees deterministic, reproducible agent evaluation runs and lifecycle verification

Risks

  • Risk of false positives if mock fixtures drift out of sync with real-world model outputs
  • Underlying prompt efficacy and model reasoning nuances are not exercised by static stubs

Governance Implications

  • Offline deterministic verification becomes a required governance gate for continuous architecture pipelines
  • Mock fixture updates must be reviewed alongside schema contract changes

Reuse Notes

Applicable enterprise-wide across all agent orchestration pipelines, offline evaluation suites, and continuous architecture verification gates where deterministic assertions of pipeline control flow and state management are required.

Confidence

High confidence derived from ratified reference architecture and verified production implementation.

Level: high

Rationale

Ratified Reference Architecture in ArchGov, complete implementation in cas_runner codebase, and verified integration across continuous testing workflows.