Social platforms are the default layer for news, entertainment, and private communication. Consumer-grade models now synthesize photorealistic faces, lip-synced video, and human-sounding voices from seconds of source material. Injected into ranking systems, falsehoods routinely outpace corrections, overwhelming users’ ability to separate fact from fabrication. The harms are concrete for everyday people: finance staff authorize fraudulent wires after deepfaked meetings; parents receive cloned-voice ransom calls; individuals discover viral non-consensual intimate imagery; AI-styled memorial graphics falsely announce celebrity deaths; and highly realistic “breaking-news” images briefly move markets. These incidents cause financial loss, reputational injury, and psychological distress. Evidence links online victimization to anxiety/depression and links unemployment/financial stress to elevated suicide risk. Platforms therefore need safeguards that address both authenticity and human impact.

This paper: 1) Maps the threat landscape; 2) Formalizes a risk taxonomy; 3) Proposes authenticate-then-distribute with PPAV screening; 4) Details governance (policy/ops/metrics); 5) Provides an implementation roadmap; 6) Adds evaluation, legal, and consent frameworks.

Taxonomy provenance. Our R1-R5 harm taxonomy (identity/consent, financial, information, psychological, systemic) diverges from “process-first” frameworks (e.g., NIST AI RMF’s cross-cutting risk characteristics and the EU DSA’s “systemic risks”) by prioritizing harm class at content level to drive class-specific gates, SLAs, and metrics. NIST AI RMF organizes risk management functions and characteristics (e.g., safety, privacy, explainability) ( Raimondo et al., 2023 ), while the DSA mandates platform-wide assessment/mitigation of systemic risks (e.g., illegal content, disinformation, minors’ safety, fundamental rights). Our taxonomy complements these by mapping concrete synthetic-media incidents to controls that can be executed at upload time.

Creation costs for convincing fakes are near zero: seconds of audio can clone a voice; a handful of photos can synthesize a face; off-the-shelf apps lip-sync or restyle video. Distribution is frictionless—upload flows accept realistic media; recommenders can propel a post to mass reach in minutes. Because virality precedes verification, time-to-harm is short; even fast removals cannot retract screenshots, downloads, and mirrors. Users are exposed because traditional trust cues are forgeable (familiar face/voice), context collapse places high-risk claims amid entertainment, and incentives are asymmetric (attackers need one win; targets must detect every attempt). Authenticity checks must therefore shift before distribution.

Across platforms, synthetic media has already produced concrete harms for ordinary people and organizations. In one widely reported case, a finance employee joined what appeared to be a routine multi-participant video conference with senior executives and, acting on urgent instructions, authorized wire transfers totaling roughly US$25 million; only later did the firm discover the entire “boardroom” was a deepfake ( Internet Crime Complaint Center, 2024 ). Similarly, an accounts-payable controller at another company sent about US$243,000 after receiving a phone call that flawlessly mimicked the CEO’s voice, accent, and cadence ( Internet Crime Complaint Center, 2024 ; Reshef, 2023 ). Beyond corporate fraud, non-consensual sexual deepfakes have targeted celebrities and private citizens—including schoolgirls—causing acute distress, bullying, and long-tail reputational damage ( Internet Crime Complaint Center, 2024 ; Ortutay, 2023 ). Synthetic “breaking-news” images have briefly moved markets, as when a photorealistic picture of an explosion near a federal building spread widely before being debunked. ( Wang et al., 2024 ). Deepfake advertising has also co-opted public figures’ likenesses to pitch dubious products, such as fake “$2 phone” giveaways, deceiving consumers and tarnishing reputations ( Reporter, 2023 ; Pringle, 2023 ). Families have been hit by kidnapping scams in which parents receive calls using a cloned voice of a loved one demanding immediate payment. Even when no money changes hands, AI-generated memorial graphics and auto-written obituaries have falsely announced the deaths of popular actors and TV hosts, spreading panic while driving monetizable clicks.

These incidents point to specific controls leaders should institute. High-value payments must never be approved within a single channel; require out-of-band verification (e.g., a call back to a known number or a second approver on a different medium) and liveness checks for urgent requests ( Qi et al., 2020 ). Treat any likeness-based advertising as high risk and demand verifiable consent and asset provenance before an ad can run. Classify celebrity-death claims as high-risk information harm and hold such uploads until authenticity can be corroborated by trusted sources; suppress recommendations when provenance is absent. For crisis-language P2P transfers (e.g., ransom or emergency pleas), add friction such as cool-off timers and secondary confirmations. Finally, adopt one-click privacy takedowns for intimate imagery and hash-blocking to prevent re-uploads, coupled with fast SLAs so victims receive timely relief ( Farid, 2021 ).

Where possible, we anchor impersonation, voice-cloning, and deceptive-ad risks in law-review and government material (keeping the original news reports as supplementary footnotes). The FBI has issued formal PSAs documenting AI-enabled impersonation and fraud trends and operational mitigations; recent law-review work examines deepfake exploitation and the right of publicity in ads and endorsements ( Preminger & Kugler, 2024 ; Murray, 2025 ).

High-value payments require out-of-band verification and liveness; likeness-based advertising requires verifiable consent and provenance; celebrity-death claims are gated until corroborated; crisis-language P2P transfers gain cool-off timers; intimate imagery routes to one-click privacy takedown and hash-blocking (Controls detailed in Table 1 ).

We group synthetic-media harms into five classes:

R1—Identity & consent harm covers any simulation of a real person without permission—most painfully, non-consensual sexual imagery and impersonations used in ads or outreach. The injury here is personal and immediate: dignity, privacy, safety, and livelihood can be damaged the moment a look-alike face or voice circulates.

R2—Financial harm captures direct monetary losses when synthetic media is used to deceive—wire-transfer fraud after a deepfaked “executive” meeting, voice-clone phone calls authorizing payments, or fake celebrity endorsements that trick users into buying scams or handing over credentials.

R3—Information harm arises when realistic fakes mislead at scale: fabricated health or finance claims, market-moving “breaking-news” images, or celebrity-death hoaxes that cause panic and poor decisions before corrections can catch up.

R4—Psychological harm focuses on the human aftermath—anxiety, depression, and in some cases suicidal ideation—following reputational attacks, image-based abuse, or mass harassment, with adolescents and other vulnerable groups at higher risk.

R5—Systemic harm reflects the broader erosion of trust in authentic user-generated content and the creator economy; when users cannot tell real from fake, engagement quality drops, advertisers pull back, and the platform’s legitimacy suffers.

R1 - R5 is intentionally incident-centric: it optimizes for fast, class-specific actions at upload time, whereas NIST and the DSA are system-centric (governance, process, and systemic risk). The two layers interlock: platform-level duties (DSA/NIST) require class-level controls (R1 - R5) to be measurable and auditable.

This taxonomy matters because each risk prompts a different response: R1 needs consent gates, one-click takedowns, and hash-blocking; R2 requires out-of-band payment verification and fraud-aware friction; R3 calls for authenticate-then-distribute policies and provenance-aware ranking; R4 adds victim-first operations and mental-health signposting; and R5 demands transparent reporting and independent audits to rebuild trust. Controls should be class-specific rather than treating all synthetic media as uniform risk.

Under this model, distribution is earned—not assumed.

The baseline rule is simple: no provenance, no promotion. If an upload has no verifiable origin, the platform may still allow it to exist (e.g., on the uploader’s profile or via direct link), but it does not enter recommendation systems and it must carry a clear context panel stating that its origin or authenticity is unverified.

For high-risk classes—identity/consent harms, financial harms, and information harms (R1 - R3)—the bar is higher: no provenance, no posting to public feeds. Before such content can be publicly distributed, the uploader must satisfy one of two conditions.

Either the asset includes cryptographic provenance (e.g., C2PA/Content Credentials) that attests to its capture and edit chain, or the uploader passes the verified identity checks and marks the upload as synthetic with an explicit disclosure and prominent label.

Even then, the item is quarantined for review or reach-constrained until checks clear.

In practice, this shifts platforms from a publish-then-moderate posture to an assure-then-amplify posture, reducing the chance that high-risk fakes go viral before anyone can intervene. Items may be reach-limited or queued for rapid review. Rejections must be “reject with rationale + appeal link”.

For contexts where pre-publication screening is technically limited (E2E messaging, low-latency live, or ephemeral “Stories”), apply recipient-side warnings, share-rate limits, forward limits, and contextual interstitials; allow optional “verify before send” for business accounts; and run PPAV asynchronously with retroactive throttling or takedown if risk thresholds are exceeded.

Goal. The PPAV model is designed to stop high-risk synthetic media from entering public feeds before authenticity and consent are established, while letting ordinary creative posts flow with minimal friction. It inserts a short, automated screening pipeline at upload, then routes only uncertain or sensitive items for fast human review.

Watermarking and forensics are adversarially brittle; C2PA coverage is uneven and manifests can be stripped. PPAV mitigates these with ensembles and provenance-aware ranking, but new generative methods may outpace detectors. Future work includes open benchmarks for risk-class detection, standardized creator-impact metrics, and multi-platform shared hashing for synthetic harms.

Watermarking and forensics are adversarially brittle; C2PA coverage is uneven and manifests can be stripped. PPAV mitigates with ensembles and provenance-aware ranking, but new generative methods may outpace detectors. We also note open risks around cross-platform interoperability and provenance stripping; future work should include open benchmarks for risk-class detection, standardized creator-impact metrics, and shared hashing for synthetic-harm signatures across platforms.

The conclusion argues that the core problem isn’t “AI itself”, but how easily AI now lets anyone forge the signals people rely on—sight, sound, and familiar voices—and how quickly social platforms can amplify those forgeries. We are already seeing the costs in three dimensions: money lost to scams, reputations damaged by impersonations and deepfakes, and real mental-health harms to victims. The remedy is not to slow or ban creative uses of AI; it’s to raise assurance exactly where harm is likely. That means switching from publish-then-moderate to authenticate-then-distribute for high-risk content: before a post can reach public feeds or recommendations, it passes a Pre-Publication Authenticity Verification (PPAV) gate that combines provenance (e.g., Content Credentials), forensic signals (media/watermark checks), and semantic checks (what the content claims) to decide whether to publish normally, publish with limited reach and context, or quarantine for review. It also means running victim-first operations—fast takedowns, hash-blocking of reuploads, and clear survivor support—so people get relief quickly when harm occurs. Finally, platforms must report transparent, auditable metrics (time-to-label, time-to-takedown, provenance coverage, reupload recidivism) so progress is visible and accountable. Taken together, these steps protect users without stifling creativity, reassure advertisers that the environment is trustworthy, and preserve the long-term legitimacy of the social web.