Post-Mortem: Agent Misclassified a Campaign Setup — and How It Got Caught and Fixed
A short case study on human-in-the-loop review of an AI monitoring agent. Anonymized.
Summary
An automated, read-only advertising-monitoring agent produced its daily brief and flagged a top-priority “campaign setup error” across a whole family of campaigns. On review, the operator’s instinct said something was off. A two-minute follow-up confirmed the agent had drawn the right data but the wrong conclusion. The finding was corrected, the saved report and its structured data were updated, and a durable note was added so the agent won’t repeat the mistake.
Net impact: no wrong action was taken (the agent is recommend-only), and the error was caught before it influenced any decision.
What the agent reported
The agent flagged that an entire family of “cross-target” campaigns was targeting their own advertised product — i.e. self-targeting — and raised it as the #1 priority, implying a systemic setup bug to unwind.
What was actually true
The campaigns were set up in a legitimate but opposite direction from what the agent assumed:
The agent assumed the product being advertised matched the product named in the campaign title (the “hub”).
In reality, the campaigns advertised the secondary products (”spokes”) and used the hub product only as the target — so the secondary products’ ads appeared on the hub’s high-traffic page.
Advertised product and targeted product were therefore different items. No campaign was targeting its own advertised product. It was not self-targeting at all.
One of these campaigns was actually performing well; one was simply inefficient. “Inefficient” is a performance note, not a structural defect.
Root cause
The agent inferred a key attribute (the advertised product) from a proxy (the campaign’s name) instead of pulling it from the source field. The data it retrieved was correct; the interpretation layer over-trusted a naming convention. Because the conclusion was stated with high confidence and ranked #1, it looked authoritative.
How it was caught
The operator read the finding against their own mental model of how they’d built the campaigns and asked a clarifying question — essentially, “Wait, did I set this up the other way around?” That question, not a system alert, is what surfaced the error. The operator’s domain intuition was the real detector.
How it was corrected
Verified from the source, not the proxy. Pulled the advertised product field and the targeted product field separately and compared them. They differed — disproving the self-target claim and revealing the true direction.
Fixed the deliverables. Removed the incorrect top-priority finding, re-ranked the remaining items, and reframed the one genuinely weak campaign as an efficiency issue rather than a setup issue. Updated both the human-readable report and the machine-readable data file, and re-synced the “latest” pointer.
Prevented recurrence. Added a persistent note to the agent’s memory: never infer the advertised product from a campaign name; always confirm self-targeting by comparing the advertised-product field against the targeted-product field, and only flag it when the same item appears as both.
Why it worked out well
The agent was recommend-only. It surfaces findings for human review and cannot change anything on its own, so a wrong conclusion could not become a wrong action.
The output was specific and evidence-backed. Because each finding carried its underlying numbers and identifiers, the claim was cheap to audit and disprove.
The human stayed in the loop. The operator treated a confident AI conclusion as a hypothesis to test, not a fact to accept.
The fix was systemic. The correction didn’t stop at this one report; it changed how the agent will interpret this pattern in every future run.
Takeaways
Confidence is not correctness. Rank and certainty in an AI report should invite scrutiny on high-impact items, not replace it.
Distinguish “the data is wrong” from “the interpretation is wrong.” Here the data was right; the inference was not. They have different fixes.
Pull attributes from source fields, not from names or labels that merely imply them.
Domain experts are the best detectors of subtle model errors — design the workflow so their instinct has an easy path to challenge a finding.
When you correct an agent, correct the deliverable and the behavior, so the lesson persists.