Agentic AI QA Pipeline & LLM Evaluation
View MoreAgentic AI QA Pipeline & LLM Evaluation
Work /
- Project
- Agentic AI QA Pipeline
- Delivered
- 2026
- Impact
- 4-8x faster delivery
Designed and shipped a 4-agent AI pipeline (Claude Code agents integrated with Azure DevOps via MCP) that generates test cases from tickets, converts them into automation, opens pull requests, and self-serves developer environment setup, with human approval gates at every step. Cut test case authoring from ~2 hours to minutes and automation time from ~4 hours to ~30 minutes per test case.
Also built an LLM-as-judge evaluation system for an AI-powered natural-language business assistant, using an LLM as an automated judge to validate multi-turn replies against structured business data and verify rendered output, replacing fixed-string assertions with semantic validation of inherently non-deterministic AI behavior.