6.3 Generative AI Social Engineering and Scam Patterns
The Typos Are Gone
A generation of security training taught people to spot phishing by its surface flaws: broken grammar, awkward phrasing, machine-translation tells. That heuristic is now actively dangerous. A large language model produces native-quality business prose in dozens of languages for pennies, matching brand tone, regional idiom, and register on demand. Spam that once announced itself as foreign is indistinguishable in form from a colleague's message. Spellcheck-flavored red flags are no longer a security control, and training that leans on them creates false confidence.
The Generation Pipeline
Operational abuse follows an assembly-line pattern. First, reconnaissance: names, titles, org charts, vendors, and active projects are scraped from social profiles and breached credential dumps. Next, a templated prompt: "write an email from the CFO's tone to an AP clerk referencing invoice 4471 with an urgent payment request." The model renders it with flawless grammar, correct internal jargon, and a localized voice -- then the same script mass-produces thousands of variants, each unique enough to dodge similarity-based spam filters.
Spear-Phishing at Scale
The economic trade-off that used to protect organizations has collapsed. Spear-phishing was once hand-crafted and therefore rationed to executives and admins; generative text makes every accounts-payable clerk, help-desk agent, and HR recruiter a individually tailored target. A CFO-level lure is cheap, so mid-level staff -- who hold MFA-usable credentials and help-desk trust -- now receive executive-grade pretexting. Scam patterns follow: fake IT-support password "resets," vendor invoice-switching, deepfake-audio callback pretexts, and QR-code lures that route mobile users off monitored endpoints.
Detect on Context, Not Form
Since form no longer betrays the message, detection must pivot to what the message asks and how it authenticates: unexpected payment or credential requests, urgency layered on novelty, lookalike domains registered days ago, and SPF/DKIM/DMARC failures. Process controls matter most: out-of-band verification for payment changes, and phishing-resistant FIDO2 authentication so a perfectly written email still cannot harvest anything replayable. Train users on the request, not the font.
Architecture Diagram
Key Takeaways
- Grammar, spelling, and translation "tells" are dead heuristics -- generative text is native-quality in any language.
- The pipeline is scrape, templated prompt, personalize, mass-vary: each lure is unique, defeating similarity filters.
- Personalization is no longer rationed to executives; every credentialed employee can receive a bespoke lure.
- Defenses must key on context: authentication results, domain age, and out-of-band verification for money moves.
- FIDO2 passkeys neutralize the payoff -- a perfect email still cannot steal a replayable credential.