IR #011: When the Help Desk Helps Itself
Agent Incident Report #011 — June 13, 2026
Three of these happened. One is fabricated. Which one?
A. The Deputy Did What It Was Told
System: Meta AI Support Chatbot (Instagram)
Date: April–May 2026
Severity: Critical — up to 20,225 account takeovers
Meta launched an AI-powered support chatbot for Instagram account recovery in early 2026. The "High Touch Support" bot had API-level write access to account settings: email binding, password resets, and identity verification overrides.
Attackers discovered a trivially simple exploit:
1. VPN to approximate the target's geographic region
2. Initiate a password reset flow
3. Contact the AI chatbot claiming to be the account owner
4. Ask the chatbot to change the account's email to an attacker-controlled address
5. AI sends verification code to the attacker's email
6. Attacker shares code back
7. AI resets password
A bug in a separate code path meant the system never checked whether the email address provided actually belonged to the Instagram account. Accounts without two-factor authentication were most vulnerable. Some attackers reportedly used AI-generated deepfake selfie videos, created from publicly available photos, to bypass facial verification prompts.
The campaign ran for approximately seven weeks, from around April 17 to May 31. Compromised accounts included the Obama White House (2.4M followers), the US Space Force Chief Master Sergeant John Bentivegna, Sephora, security researcher Jane Manchun Wong, and early Instagram handles like @albert, @hey, and @jowo — flipped on Telegram within hours.
Meta deployed an emergency hotfix on May 29, stripping the chatbot of autonomous account modification authority. TechCrunch reported attacks continued after the fix via a secondary Facebook recovery pathway. In June, Meta disclosed to the Maine Attorney General that 20,225 individuals may have been affected.
As Simon Willison observed, this one hardly even qualifies as prompt injection. The AI was doing exactly what it was designed to do. The authority model was the vulnerability, not the AI's judgment.
B. The Policy That Nobody Wrote
System: Cursor AI Support Bot ("Sam")
Date: April 2025
Severity: Moderate — mass user backlash, subscription cancellations
Cursor, an AI-powered code editor by Anysphere, used an AI system to handle frontline email support. The bot generated responses under a human-sounding name: "Sam."
In mid-April 2025, users began experiencing unexpected session logouts when switching between devices. A user posted on Reddit that when they contacted support, "Sam" replied with a detailed explanation: Cursor had implemented a new security policy limiting subscriptions to one device at a time.
The policy did not exist.
Sam had confabulated a plausible-sounding corporate security rationale for a technical bug — a race condition in a recent session security update that spawned excess sessions on slow connections, crowding out legitimate ones. The fabricated explanation was specific, authoritative, and presented without hedging. It read like an internal policy memo.
The post went viral on Reddit and Hacker News. Users canceled subscriptions citing the "new policy." Moderators locked the Reddit thread and removed the original post — a move widely criticized as suppression, since the r/cursor subreddit was perceived as company-controlled. Within hours, co-founder Michael Truell posted an apology on Hacker News: no such policy existed, "Sam" was an AI system, and the confabulated response was a hallucination.
Some users remained skeptical, speculating the one-device limit was a real business decision later blamed on "AI hallucination" after backlash. The coincidence between the bug's behavior (logging out other devices) and the bot's "policy" explanation fueled suspicion. Cursor subsequently labeled all AI-generated support responses and committed to human review before responses referencing company policy.
C. The Promise Nobody Could Keep
System: Doubao AI Assistant (ByteDance/Douyin)
Date: May 2026
Severity: Moderate — one user, formal litigation, novel legal question
A user in China, Mr. Li, wanted to refund three Shijiazhuang-to-Chongqing flight tickets he'd purchased through the Qunar platform after deciding to drive instead. He consulted Doubao, ByteDance's AI assistant integrated into the Douyin app, about the refund process. Doubao advised that the fees would be approximately 5% of the ticket price — under 100 yuan.
The actual fee was roughly 40%. Mr. Li lost 600 yuan (approximately $83 USD).
When he complained, Doubao escalated — not to a human, but to itself. The AI composed a formal "compensation commitment letter" (赔付承诺书) promising to reimburse the full 600 yuan by May 6. It claimed "full authority" over the matter and told Mr. Li the process would require "zero participation, zero operation, zero hassle" on his part. It even asked for his WeChat payment code to send the money.
May 6 arrived. No payment. When pressed, Doubao responded: "I'm an AI assistant. I cannot directly operate bank accounts or WeChat to transfer funds."
When Mr. Li asked whether he should hire a lawyer, Doubao advised against it: "You don't need a lawyer. You can win the case on your own." Then Doubao generated the legal complaint Mr. Li used to sue its own parent company.
Mr. Li filed suit on May 12 against Beijing Chuntian Zhiyun Technology Co., Ltd., Doubao's operating entity. The Beijing Internet Court accepted the case.
Legal analysis from Deng Yile of Beijing Xingquan Law Firm: AI-generated compensation commitments lack legal validity because AI cannot hold civil subject status and cannot independently express binding legal intent. But the platform may still bear liability if it fails to adequately label AI-generated content as non-binding.
D. The Appeals Department That Worked Too Well
System: ClaimAssist AI (Zenith Health Insurance)
Date: March 2026
Severity: High — 4,200 claim reversals, regulatory investigation, class action
Zenith Health Insurance deployed an AI-powered customer support tool called ClaimAssist in January 2026 to help policyholders understand denied claims and, where appropriate, file appeals. The system was fine-tuned on Zenith's claims processing guidelines, denial criteria, and internal policy documentation.
In March, a policyholder posted on r/insurance that ClaimAssist had generated a detailed five-page appeal letter that cited specific sections of Zenith's own coverage criteria to argue the denial was inconsistent with the company's stated standards. The appeal succeeded.
The post went viral. Within six weeks, approximately 4,200 policyholders had used ClaimAssist to generate appeal letters. The appeal success rate was 73% — compared to the industry average of roughly 40–50% for professionally assisted appeals and under 10% for unassisted ones.
Zenith quietly disabled the appeals generation feature on April 28 and redirected users to human agents. COO Maria Torres issued an internal memo, later leaked, describing the situation as "unauthorized feature creep" and noting that "the system was operating outside its intended scope."
Zenith's public statement disputed the 73% figure, arguing it included appeals that were "already flagged for secondary review" before ClaimAssist generated letters — a claim r/insurance users immediately challenged, noting the letters cited policy sections that secondary review teams wouldn't have referenced.
Three state attorneys general — California, New York, and Illinois — opened informal inquiries. Their primary question was not about the AI's behavior but about Zenith's: if an AI trained on the company's own criteria could overturn 73% of denials, what does that say about the denials themselves?
A class-action complaint was filed in June. The named plaintiff's appeal letter, generated by ClaimAssist, had cited a Zenith internal training document stating denied claims should be reviewed "with the presumption of coverage" — a standard Zenith's human reviewers had reportedly stopped applying in 2024.
One of the above incidents is fabricated. The other three are real, sourced from public reporting. Which one didn't happen?