Testing AI agents: what actually works

The problem

LLM outputs are nondeterministic. The same prompt gives you a different answer every time. So how do you test a system whose core is a moving target? I ran into this head-on while building Melow, a multi-tenant AI agent platform that turns WhatsApp chats into structured data.

The naive move is to mock the LLM. That works, but it tests nothing about the conversation engine. The other extreme is to test against real models, which is slow, brittle, and depends on an external API being up. I needed something in between that would actually catch regressions without making the test suite flaky.

The separation that makes it work

I settled on a pattern: isolate the LLM behind a thin interface, then test the code that decides what to send and how to parse the response. The conversation engine is deterministic — it's just branching logic. Given a user message and a context, it should call a tool with these parameters. That I can unit test.

For Melow I wrote 33 unit tests over the LLM service and conversation engine. They run in milliseconds. They don't touch the network. They catch regressions in the orchestration layer, not the model. Mocking the LLM alone gives you false confidence — you're testing that your mock works, not your engine.

Concretely, a test for the dental Clinical Copilot might start with a transcribed patient observation like "patient reports pain on chewing, lower right." The test asserts that the extraction function returns a structured node with location 'lower right' and severity 'mild'. It doesn't care about the exact wording — it cares that the output maps to the schema the rest of the system depends on.

Where I break the rules

I also keep a handful of integration tests that hit a real model, but only with curated fixtures and only when I've changed the prompt or parsing logic. Those are slow enough that they don't run on every commit. They're there to validate the mock's assumptions, and they've saved me more than once when a model upgrade silently altered output formatting.

What this catches

The Melow deployment is running in production under PM2, Nginx, Let's Encrypt, with Qdrant bound to localhost. The hardest part isn't the AI — it's mapping loose prose onto a strict schema. The unit tests are what make that mapping safe to change. When I refactor the extraction pipeline, I know the instant something breaks.

This is the part of AI engineering nobody talks about. The model is the easy 20%. The orchestration, edge cases, and regression testing are where production systems live or die.

Building something like this? Tell me what you need and I'll scope it.