A human in the loop is not a control

The check that checked nothing

The agent had already sent the money by the time anyone looked. It read the invoice, matched it to the order, coded it, and queued the payment. A manager clicked approve. 3 weeks later, finance found it in reconciliation (I wrote about why finance, not the dashboard, is where these surface: Measuring an agent that can act): a payment to a bank account that wasn't the vendor's, for an invoice that had already been paid once.

Someone had approved it. That was the problem, not the safeguard.

I've shipped a lot of agents, and been called in to fix more. On every one that can actually act (reschedule a payment, change an account, flag a risk, etc.) the same question comes back: do you put a human in the loop as the safety net… or not?

I've been through it enough times to settle on a method. It isn't the only one, but it's held across very different systems. The 3 examples below are all from real work.

"Keep a human in the loop" is the reflex answer to agent risk. It feels like a control. For the actions that actually matter, it usually isn't. So the real question isn't whether to keep a human. It's when a human is a control, and when they're a signature.

A human in the loop feels like a control. For the decisions that matter, it's often just a signature.

2 questions decide it

Here's the whole decision, and you can run it on any agent action in about 10 seconds.

Ask 2 things.

1st: is the call provable, or a judgement?

Provable means you can check it against a rule: is this a duplicate? is it over the limit? does this cited case actually exist?

Judgement means it needs interpretation: is this argument sound, is this the right exception to grant, does this flagged risk matter given what the product is meant to do.

2nd: how bad is it if it's wrong? Cheap and reversible, or expensive and irreversible?

3 answers fall out. If the call is provable, you don't need a human, you need a deterministic gate: a rule that refuses the wrong action whether or not anyone is watching. If it's a judgement and the cost is low, let it run and sample it later. If it's a judgement and the cost is high, that's the 1 place a human earns their seat, and even then only on the exceptions, with the context that makes the call judgeable.

That's the tool. The rest of this is why each answer is what it is, and where teams get it wrong.

When a gate beats a human

The popular version of this advice goes like this: if you can specify the decision, let the agent commit; if you can't, keep a human in the loop.

Half right.

Specifying a decision is real work: it means writing down the steps to reach it, the evidence that counts, the check that tells you you're confident enough to act, and what would change the answer. Do all 4 and you've genuinely clarified the decision. But you've specified it, not enforced it. A well-specified decision handed to a model that "usually" follows it is still a probability engine (I wrote a piece on exactly this: Stop shipping probabilistic agents) wearing a rulebook. Specifying what's allowed doesn't make the agent do only what's allowed. That's what the gate is for.

Take payments. What makes a payment wrong, a duplicate, an altered payee, a price above contract, isn't generally visible on the approve screen. The approver sees a vendor, an amount, and the agent's tidy summary. To catch the bad one they'd have to leave the screen and reconcile against the ledger, the contract, and the bank feed… for every invoice (pfff). Most don't. So the approval collects a name and enforces nothing. Encode those same checks as a gate on the payment instead, refuse a duplicate, hold any bank-detail change until it's verified out of band, block anything above the contract price, cap anything going to a brand-new account, and the control now runs on all of them, at machine speed, with no one watching. Money that leaves the wrong way doesn't come back. That's exactly the corner where a signature isn't enough.

The same move works in law, on the part that's checkable. Whether a cited case exists, whether an article says what it's quoted as saying, is provable. So verify every citation against the source before the document leaves, and no fabricated citation survives. That matters, because grounding the model in a real database doesn't save you: the leading legal-research tools still hallucinate between 17% and 33% of the time¹. A lawyer skimming fluent output can't be the control on that. A citation check can.

And the gate isn't a whiteboard idea. Policy engines that sit between the agent and the action and refuse anything outside the rules ship today².

When the human is the point

Now the other side, because this isn't an argument against humans :-)

Picture a security agent that reads code and flags risks. It never fixes anything, it flags. Here the human isn't theatre, the human is the point. Whether a piece of code is a real vulnerability often can't be judged from the code alone = it depends on what the system is meant to do, the intent, which lives in the spec, the kickoff, the design docs, not in the file. Hand the agent the code and it buries you in false positives. Hand a person the flagged exceptions plus the intent, and they make the call the model can't. The cost of a miss here is a hole that ships to production, so the judgement is worth a human's time.

Law crosses the same line the moment you move from the citation to the argument. Pulling the exact article and the real case is deterministic, and the agent should never get it wrong. But the document built on top, the argument shaped by the client's situation, is judgement. The lawyer best knows the context, not the model, so the lawyer leads. The agent can suggest a direction that's wrong; it should never cite an article that's wrong. Gate the citations, keep the human on the reasoning. One example, both halves of the rule.

That's the recipe for keeping a human: low volume, high stakes, and context the model can't see. Miss any one and you're back to theatre. Volume most of all. Put a person on every decision and they stop deciding, they rubber-stamp, and it gets worse the more reliable the agent looks. It's not a character flaw. When researchers gave radiologists a computer aid, it made the best readers worse on the hardest cases³, the aid was trusted most exactly where it was least right. A human placed to rubber-stamp a plausible machine is not a safeguard. They're a witness with a mouse.

When you need neither

One more, so the rule stays honest. Plenty of agent decisions are cheap and reversible: a draft's tone, a first-pass summary, a low-value tag. Gating those is over-engineering; reviewing them burns a salary on nothing. Let them run, watch the aggregate, fix the misses. Not every agent needs any of this. The whole method is aimed at 1 corner: the expensive, irreversible calls where being wrong is costly and the right answer depends on state a reviewer can't see.

Where the human actually belongs

So the shape of a reliable acting agent isn't "the agent, plus a human approving everything." It's a deterministic gate on everything provable, a human on the rare high-stakes exception with the context in hand, and free rein on the cheap stuff. Draw those 3 lines well and most of the "should we keep a human" argument dissolves, because you've stopped asking it once about the whole agent and started asking it per action.

It's also the only version that scales. A human on every action caps the agent at human speed, the exact throughput you were trying to escape, and it can't grow without hiring reviewers in lockstep with volume. Move the human to the exception, and the gate does the scaling.

If your safeguard is a person clicking approve, you don't have a control. You have a witness.

--------------------

If your agent can act, and you can't say which of your safeguards actually enforce anything, let's talk.

Sources

(1): Magesh et al., Stanford RegLab (2024), leading AI legal-research tools hallucinate 17-33% — https://reglab.stanford.edu/publications/hallucination-free-assessing-the-reliability-of-leading-ai-legal-research-tools/

(2): Amazon Bedrock AgentCore Policy: deterministic, default-deny enforcement of agent actions — https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/policy.html

(3): Povyakalo, Alberdi, Strigini & Ayton (2013), computer aid degraded the best readers on hard cases, Medical Decision Making — https://pubmed.ncbi.nlm.nih.gov/23300205/