Hires from a pool of 100000
By Dr. Vaisagh Viswanathan (VT), CEO & Co-founder, impress.ai
We were recently asked for input to an industry consultation on testing agentic systems. That work has since been published as IMDA’s Starter Kit for Testing Agentic Systems for Safety and Reliability, which cites impress.ai among its sources. The consultation deck separated outcome testing from trajectory testing and laid out a three-tier autonomy spectrum, and we found both framings useful.
Answering the questions meant writing down how we actually test our hiring and assessment agents at impress.ai, including the parts that are ad hoc. This post is the longer version of that, told as a set of tales from the wild. They’re independent observations, not a sequence, and I won’t pretend we ended up with clean answers.
If there’s a thread through them, it’s that a lot of testing guidance reads as if one party owns the whole stack: builds it, runs it and sees everything it does. That isn’t our situation. We ship a platform that clients configure with their own prompts and run on their own data, on top of models we don’t control. Most of what follows is what that looks like in practice.
We run dev, pre-prod and prod as separate layers. Dev is fully sandboxed, prod is live, and pre-prod is where things get messy. We give clients a pre-prod system to test in, and for a mix of reasons we can’t completely block outgoing emails there, because sending email is one of the things that traditionally gets tested. We’ve had a case where an unsuspecting user on the client side put the wrong resume through, and the wrong email went out to a real candidate. A real action had reached a real person, and we had to clean it up afterwards.
The obvious fix is “nothing leaves the sandbox.” But then pre-prod is useless for its actual purpose, which is validating real workflow behaviour before it goes live. You can add domain restrictions and similar guardrails, but every layer of containment slows down how fast you can iterate, and speed matters a lot in this space. We don’t think we’ve solved this well yet.
Data has the same tension. We can’t use real candidate data for compliance reasons, so we rely heavily on synthetic resumes and profiles, both in-house and through our testing provider during the AI Verify agentic sandbox. People don’t always appreciate how much harder this is in Singapore and Southeast Asia than in the US. A synthetic profile has to reflect local hiring norms and our specific enterprise use cases, and there’s much less reference data to build from. Generating data that’s actually realistic, not just structurally correct, is a genuinely hard problem in a data-scarce market. There are vendors who specialise in this. We haven’t gone that route yet, given the cost.
Our clients can modify agent behaviour themselves, through their own prompts on top of what we’ve built. That creates a shared-responsibility problem the moment something breaks. We need to tell prompt drift (client-caused) apart from model drift (provider-caused), because the fix, and who is responsible for it, depends on which one it was.
It gets more tangled because an agentic system is several layers: the model, the scaffold or orchestration logic, the harness and the external integrations. Each can fail on its own, and each needs different testing. A regular LLM app is closer to one piece: prompt goes in, output comes out. There’s no separate orchestration layer sitting in between that needs its own testing.
This barely comes up in normal software testing, where you own your code, and it’s less of an issue for a simple LLM app that clients can’t reconfigure. We also don’t see it addressed much in agentic testing guidance, which tends to assume one party controls the whole stack.
The real constraint on post-deployment monitoring is access, not method. Once something is deployed at a client, we mostly don’t see what’s happening in production. We don’t have the data, and in most cases we’re not allowed to look even if we wanted to, given how sensitive candidate data is and what our contracts allow. So we end up relying a lot on clients telling us when something’s wrong.
What we do control is our own testing cadence. Roughly every two to four weeks we run checks for model drift ourselves, whether or not a client has raised anything. When a client does report something, we have an SOP for triaging and escalating it.
The case worth flagging for the guidelines is gradual demographic drift. Catching it needs someone watching outcomes at an aggregate level over time, against some agreed sense of what’s normal. We’re not positioned to do that continuously, since we don’t have the data. Clients often aren’t set up to do it either, even when they are watching, because they don’t have the tooling or the shared definitions. So the risk isn’t that drift can’t be detected in theory. In practice it can fall through the gap between vendor and customer, where neither side has the access, the tooling and an agreed standard all at once.
When a client does flag something, there’s often disagreement about whether it’s actually an issue. Clients want the benefits of AI matching, but there isn’t a shared vocabulary yet for telling normal model variability apart from actual drift. Some fluctuation week to week is expected and not a sign anything’s wrong. So a lot of what should be a quick technical check turns into a longer back-and-forth about what’s even acceptable. That education gap is honestly about as big a problem for us as the access gap.
Part of the difficulty is that acceptable variance in agentic systems is behavioural, not just a matter of output quality, and often it isn’t well defined yet. Regular software testing is pass/fail. LLM app testing usually scores output against a rubric. Agentic testing needs some notion of what counts as normal variation in actions and outcomes over time, and a lot of the time nobody has agreed where that line is. That makes the testing itself harder, and the conversation with stakeholders harder than it is for either of the other two.
For us, logging isn’t just for debugging. On a hiring platform it’s also the audit trail behind scoring and screening decisions, so the bar is higher than usual. Honestly, our current approach is more ad hoc than we’d like, with different logging setups built module by module rather than as one system.
For the simpler workflow-automation agents, we use standard tools like Datadog with reasonable observability on top. It captures what happened well enough, but interpreting it automatically at scale is still hard, so in practice a lot of it comes down to AI-assisted human review, which has obvious limits.
For the more goal-oriented agents, we’ve built custom log-analyser agents: a small multi-agent setup that reviews telemetry, flags performance issues, drift and areas to improve, and feeds that into our own dev cycle. It gives us better signal, but it’s genuinely a lot of work to build and to keep current as the underlying agents change. We’ve talked to a few early-stage companies who claim to have a standardised solution for this. Nothing we’ve seen actually holds up yet, so we’re still debating internally whether to keep investing in the custom system or wait for the market to mature.
Agentic systems get expensive to test fast, so we don’t test everything at the same model fidelity. We use lighter models (AWS Nova, for example) to run large batches of evals on synthetic data and to iterate on things like UI/UX, and we save the frontier models for the deeper evaluation once we’ve narrowed things down. Because the layers of an agentic system fail differently, in practice each one also ends up needing a different fidelity.
A pro-tip from our AI Verify sandbox experience: the tester recommended building early-failure detection into test suites, so you’re not running the full suite on every push. Good advice. It’s also harder to build in-house than it sounds, and there’s a real gap between what’s recommended and what teams like ours have managed to build.
One more thing, which is less a testing lesson than an operating risk of building on this stack right now. Some of the vendor tooling in this space is still working out its own business model, and that can change your cost assumptions after the fact. One of our providers switched to on-demand pricing after we’d already built our workflow (per-PR code review, for example) around a different pricing structure, and we had to rethink how we operate.
Looking across them, what they have in common, I think, is that responsibility and visibility are split. Between the model provider, us, the client who sets prompts and holds the data, and the vendors whose tooling we lean on, nobody sees or controls all of it. Some of the hardest problems above are less about technique than about who can see what, and whether anyone has agreed what “normal” looks like. Cost is the exception.
What we haven’t worked out:
If you’re testing agents and running into the same things, I’d be glad to compare notes: vaisagh@impress.ai.
