ADR: Testing Framework for LLM Applications with Tools and Orchestration
[YAML scenarios]
│
▼
Test Runner (Vitest / Jest / Pytest)
│
├──> Call Application Gateway (LLM + Tools + Orchestration)
│
├──> Apply deterministic checks (regex, string, tool call)
│ └── Pass/Fail (CI)
│
└──> Log trace to Evaluation Framework
└── Run LLM-as-Judge / custom evals
└── Advisory scores (warnings, trends)
- id: "critical-events"
description: "Ask for recent critical events"
steps:
- user: "What are recent critical events?"
expect:
checks:
- label: "must-contain-critical"
mechanism: regex
pattern: "critical"
on: output
- label: "must-mention-event"
mechanism: hard
value: "event"
on: output
- label: "usefulness-score"
mechanism: llm
criteria:
usefulness: 0.7