About Me

Hello, I’m Umer Niaz, an AI Automation Architect specializing in agentic AI engineering and LLM evaluation systems. I design and ship production agentic AI pipelines (Claude Code agents) that cut delivery time 4-8x, and build LLM-as-judge evaluation systems that validate non-deterministic AI behavior against real business data — doing both AI-driven automation and AI evaluation design, not just one. These systems drive measurable outcomes: automation coverage from 30% to 80%+, defect leakage cut from 19% to 8%, and pass rate up from 60% to 90%. With 16+ years in software quality engineering, I own technical direction for a 25+ engineer automation organization at Altus Group, with deep AWS/Azure/Databricks integration underneath the AI layer.

personal information

Full Name
Umer Niaz
D.o.b.
16 February
address
Toronto, Canada
e-mail
me@umer.live

languages

Java expert
Python pro
TypeScript expert

Professional Skills

AI & Agentic Engineering
Cloud & Integration
LLM Evaluation & Testing
API & Event-Driven Architecture
CI/CD & DevOps
Agile

Automation Tools

Playwright
TestComplete
Jenkins CI
IntelliJ
Visual Studio Code
POSTMAN

Soft Skills

  • Customer oriented
  • Handling Multi Tasks
  • Creative Thinking
  • Self Motivated
  • Managerial Skills

Industries Served

  • IEC-61508 – Safety Critical Softwares
  • Finance & Leasing
  • Real Estate
  • Industrial Automation (Rails)

References

Ali Khalid Test Architect | Helping teams with Automation & DevOps Emirates

Umer is one of the few exceptionally bright and technically strong people I have met. Having him on the team was a big advantage. He’s a self starter, intuitive, great problem solver and extremely hard working. I have no doubt on his success. Truly the smart worker needed in this age of technology.

Ehsen Raza | Director SQA Contour Software Pvt Ltd

Umer is one of the brightest and most talented people I’ve had the pleasure of working with and has an amazing ability to work virtually. He is an accomplished test engineer, a talented analyzer and has great depth of understanding of innovation and what it takes for organizations to create an environment that supports innovation. He is truly an amazing resource.

Portfolio

Agentic AI QA Pipeline & LLM Evaluation

View More

Agentic AI QA Pipeline & LLM Evaluation

Work /

Project
Agentic AI QA Pipeline
Delivered
2026
Impact
4-8x faster delivery

Designed and shipped a 4-agent AI pipeline (Claude Code agents integrated with Azure DevOps via MCP) that generates test cases from tickets, converts them into automation, opens pull requests, and self-serves developer environment setup, with human approval gates at every step. Cut test case authoring from ~2 hours to minutes and automation time from ~4 hours to ~30 minutes per test case.

Also built an LLM-as-judge evaluation system for an AI-powered natural-language business assistant, using an LLM as an automated judge to validate multi-turn replies against structured business data and verify rendered output, replacing fixed-string assertions with semantic validation of inherently non-deterministic AI behavior.

Interests

  • Animals
  • Travel
  • Bicycle
  • Parties
  • Harry Potter
  • Coffee

FAQ

  • What is the hardest part of building AI-driven automation?

    Non-determinism. A traditional automation script asserts an exact expected value; an agentic AI pipeline or LLM-powered feature can produce a dozen valid phrasings of the same correct answer. I build LLM-as-judge evaluation layers that validate semantic correctness against structured business data instead of fixed-string assertions, so the automation stays reliable even when AI output legitimately varies.
  • What tools power your AI-driven automation?

    Claude Code agents for agentic pipelines, TypeScript and Playwright for the automation layer underneath them, and deep AWS/Azure/Databricks integration for the systems those agents actually operate on. I care less about any single tool and more about whether the stack lets AI agents and automation work together reliably in production.
  • What is the most common mistake teams make when adding AI to automation?

    Treating AI output like deterministic code. Teams bolt an LLM onto a pipeline and keep testing it with the same fixed-string assertions they used before, then call the results flaky when the AI is just behaving the way AI does: non-deterministically. The fix is not more retries, it is building real evaluation layers, like LLM-as-judge, around the automation from the start.
  • What advice would you give engineers moving into AI-driven automation?

    Learn the business logic before the AI tooling. The automation engineers who make the agentic-AI transition well are not the ones chasing every new framework; they are the ones who already understand what a pipeline is protecting, and can tell when an AI agent’s output is actually right versus just plausible-sounding.

Blog

Contact Me

Fell free to contact me in case of any question