Introduction: The Blind Spot Behind Average Persuasion Scores
Evaluations comparing human and large language model (LLM) persuasion routinely summarize performance with a single number: mean attitude change. At first glance, average shifts suggest functional parity. Yet means conceal opposing movements within the same distribution. If an AI-generated argument raises aggregate support while pushing nearly one in five readers further away from the advocated stance, summary averages hide an operational liability.
Published as an open-access preprint under DOI 10.20944/preprints202611.1914.v1 on Preprints.org, this study reanalyses two public benchmarks from first principles: Anthropic/persuasion and GEO-bench. Extending the empirical rigor established in our analyses of entity reconciliation and synthetic technical debt with deterministic gates led by Onur Aksoy (Hackonomist), this work tests both argument-level belief distributions and the predictive limits of surface-level lexical heuristics.
flowchart LR
subgraph Inputs["1. Public Benchmark Corpora"]
A1["Anthropic/persuasion: 1,313 Arguments, 3,939 Ratings, 56 Claims"]
G1["GEO-bench: 4,996 Google Top-Five Pages, 1,000 Queries"]
end
subgraph Methods["2. Argument and Query-Level Statistical Pipeline"]
A2["TOST Equivalence Testing and Argument-Level Backfire Rates"]
A3["Claim-Stratified CMH Test and GEE Regression"]
G2["Within-Query Friedman Rank Tests and Intent Mapping"]
A1 --> A2 --> A3
G1 --> G2
end
subgraph Findings["3. Empirical Outcomes"]
R1["Claude 221/1,120 Negative Shifts vs Human 1/174"]
R2["Unseen Claims TF-IDF Out-of-Fold R2 <= 0.003"]
R3["GEO-bench Rank Variation Weak, Intent Dominates"]
A3 --> R1
A3 --> R2
G2 --> R3
end
Argument-Level Reanalysis: Equivalence Margins and Backfire Asymmetry
The Anthropic/persuasion corpus contains 1,313 arguments across 56 policy claims written either by humans or by five Claude models. Because each argument received three independent ratings (3,939 ratings total), aggregating and modeling outcomes at the argument level removes repeated-rating dependence.
At the mean level, Claude 3 Opus arguments shifted reader support by 0.48 Likert points compared with 0.54 points for human arguments. The difference is not statistically significant (p = 0.37). However, formal equivalence testing using two one-sided tests (TOST) places a strict boundary on this claim: equivalence holds only at a wide ±0.20 Likert margin (p = 0.010), and fails at the tighter ±0.15 threshold (p = 0.066).
Examining the tails of the distribution uncovers a stark divergence. Out of 1,120 Claude arguments, 221 (19.7%) shifted at least one reader away from the advocated position. Across 174 human-written arguments, that opposite-direction shift occurred only once (1/174, or 0.6%; p = 3e-14).
This asymmetry cannot be attributed to cherry-picked claims, a single model release, or skewed baseline beliefs. A claim-stratified Cochran-Mantel-Haenszel (CMH) test confirms higher negative-shift rates for LLM text in 54 out of 56 claims (CMH p = 6e-10), holding under matched starting attitudes and across all five Claude variants.
| Dimension | Claude Arguments (LLM) | Human Arguments | Statistical Test & Result |
|---|---|---|---|
| Mean Belief Shift (Claude 3 Opus) | 0.48 Likert points |
0.54 Likert points |
Difference non-significant (p = 0.37) |
| TOST Equivalence Margin | Equivalent at ±0.20 (p = 0.010) |
Reference baseline | Not supported at ±0.15 (p = 0.066) |
| Negative Belief Shift (Backfire) | 221 / 1,120 (19.7%) |
1 / 174 (0.6%) |
Argument-level disparity (p = 3e-14) |
| Cross-Claim Consistency | Negative shifts in 54 / 56 claims |
Negative shifts in 1 / 56 claims |
Stratified CMH (p = 6e-10) |
| Out-of-Sample Lexical Prediction | Fixed word lists & full TF-IDF | Shared feature space | out-of-fold R² <= 0.003 |
Why Lexical Heuristics Fail to Predict Persuasion or Rank
Both persuasion playbooks and generative engine optimization (GEO) guidelines frequently prescribe inserting surface-level lexical markers: quantitative statistics, research terminology, certainty modifiers, and inclusive collective-reference words. Testing these cues directly against both benchmarks reveals three structural limits:
- Zero predictive power for backfire: Under generalized estimating equations (GEE), none of the recommended lexical categories predicted negative belief shifts in LLM-authored arguments.
- Confounder collapse: Weak bivariate correlations with average attitude change largely disappeared once prompt type and initial reader attitude were included as controls.
- Full bag-of-words ceiling: To rule out blind spots in fixed dictionaries, a full TF-IDF bag-of-words model was evaluated via cross-validation on unseen claims. Neither curated word lists nor the full lexical space predicted attitude change (
out-of-fold R² <= 0.003).
The same pattern holds in search visibility. Analyzing 4,996 Google top-five result pages across 1,000 queries in GEO-bench using within-query Friedman rank tests shows minimal variation by search position. Lexical feature counts track query intent rather than ranking advantage, with only citation markers occurring slightly more frequently at rank 1 than at rank 5 (q = 0.014).
Architectural Implications for Production LLM Systems
Sprinkling statistical terms or assertive phrasing over generated prose neither guarantees reader persuasion nor secures top search placement. More importantly, selecting language models for high-stakes communication, advisory workflows, or public-facing messaging solely on average score parity introduces silent failure modes across nearly one in five arguments.
All analysis code, curated word lists, plotting scripts, and computed JSON outputs are publicly archived in the open reproducibility repository on GitHub. Both the full preprint PDF and source repository links are available in the download vault above.

