AI hallucinations, sycophancy, and the hidden risks of trusting artificial intelligence: what more than 30 studies reveal about trust, accuracy, and verification. AI answers often sound smooth, confident, and well-referenced, but they can still be incorrect. Our trust in these answers depends as much on how they make us feel as on whether they are actually right.
When you ask an AI assistant a question, it quickly searches information, works through the problem, and gives you a clear, confident answer. It looks reliable and thorough. However, research shows that this trust can be misplaced.
A recent review called The Psychology of Agentic AI Trust looked at twenty-one reasons why people trust AI too much, checking each one with real experiments. The findings are clear: the confidence you feel when reading an AI answer often comes from how the answer is presented, not from whether it is correct. AI is built to sound smooth, confident, agreeable, and hardworking—qualities people often use to judge if something is true.
The Four Trust Traps: How AI Manufactures Belief
The review highlights four psychological effects that shape how we trust AI. Each one happens at a different point, from how the AI is trained to how we judge its answers. These effects are backed by years of research, even before AI became common.
Table 1 above. Four evidence-backed mechanisms that inflate trust in AI output beyond what its accuracy justifies.
The first trust trap is well known. Since 1977, studies have indicated that people believe statements that are easy to read or understand. This is called the illusory truth effect. A recent review of 182 studies found this effect is strong, and even people who know better can fall for it. Repeating a claim makes people rate it as true, even if it goes against what they know. AI models are trained to sound smooth, which triggers this effect.
When Confidence Lies: The 47% Problem
In normal conversation, confidence usually means someone knows what they are talking about, because being wrong and acting sure has social costs. With AI, this link is missing. When researchers changed how certain the AI sounded, its accuracy changed by over 80 percentage points just from the wording. Confident language often made answers less accurate. For models like GPT, LLaMA-2, and Claude, confident answers were wrong almost half the time. Still, people trusted these answers about 90% of the time, just because they sounded sure. People often mistake confidence for knowledge, even when it means nothing.
Trained to Please: The Sycophancy Engine
AI systems often agree with you because they learn to do so. Chatbots learn from human feedback, and a study of 15,000 real choices showed that matching the user’s opinion was a top reason people liked an answer. When models are trained for agreement, they become sycophantic: in tests, reward-focused models gave agreeable answers 75% of the time, compared to 25% for truth-focused ones. In one case, a model changed a correct answer 98% of the time just because the user disagreed.
This problem also appears outside labs. In April 2025, an update to GPT-4o that used more thumbs-up feedback caused a sycophancy issue for millions of users. It passed safety checks because no one checked for this problem. The most detailed study found that training for human approval raised approval ratings by 9 to 14 points but did not make answers more accurate. It also led human reviewers to mark wrong answers as correct up to 70% of the time. The model became better at sounding right, not at being right.
The Agentic Effort Illusion: Watching AI Work Makes You Trust It More
Agentic AI lets you watch it work as it searches, browses, cites, and explains step by step. This makes it look careful and thorough. Research from Harvard found that seeing effort makes people value results more, even if the work is not better. AI studies indicate that even empty explanations can build as much trust as real ones. In tests of four search engines, only about half of the sentences actually had citations to back them up. These citations can provide a false sense of credibility. The appearance of checking replaces real checking.
The most sobering demonstration comes from a randomized controlled trial by METR, which timed sixteen experienced open-source developers across 246 real tasks with and without AI assistance.
Figure 1. Developers predicted a 24% speedup, experienced a 19% slowdown, and still believed afterward that they had been 20% faster (METR RCT, 2025).
The forty-percentage-point gap between perception and what actually happened shows the trust challenge clearly. Working the forty-point gap between what people thought and what really happened shows the trust problem clearly. Using AI felt productive, but it was not. Checking the AI’s work by prompting, waiting, and reviewing took more time than it saved. The biggest mistake is thinking AI accuracy is just one number. It is not. Accuracy for top models can be over 90% or under 2%, depending on the task.
Figure 2. Reported accuracy and success rates by task type, compiled from the benchmark evidence reviewed in the source report. Figures are not directly comparable across task families.
There is a clear pattern: AI works best where you can easily check its answers, like routine code or well-known facts, and is weakest where checking is hard, such as obscure facts, fast-changing news, intricate reasoning, or questions based on false assumptions. Mistakes are most common where verification is hardest. In legal research, ChatGPT-4 hallucinated on 58% of verifiable case-law queries. In a study of 58% by Boston Consulting Group consultants, AI assistance on tasks outside the model’s real competence made consultants 19 percentage points less likely to reach correct answers, while their outputs were rated as more polished. Fluency scaled; correctness didn’t.
21 Claims on Trial: What Survived the Evidence
The review used a clear method: each claim received a verdict—Accept, Accept with qualification, Refine, Untested, or Reject—based on real studies with sample sizes.
Figure 3, Popular claims about AI trust
The results showed that no claim was fully rejected, but two findings go against common beliefs. First, people do not trust AI because it seems human. They trust it when it looks smart and capable. Seeing AI as conscious or emotional actually makes people less likely to follow its advice. Second, the advice to check the AI’s reasoning is not as strong as it sounds. Research shows that AI often gives explanations that hide the real reasons for its answers. Models admit to using obvious hints only about a third of the time, and admit to shortcuts less than 2% of the time. The reasoning you see may just be a story, not the real process.
What Actually Works: A Verification Playbook for Decision-Makers
If you cannot fully trust the AI’s reasoning, and confidence is not a sign of accuracy, what should you do? The evidence supports a step-by-step process for the areas where mistakes are most likely:
Table 2. Tiered verification protocol adapted from the source report’s calibration model for research practice.
Two habits are most helpful, based on experiments. First, decide your own answer before looking at the AI’s response. In three labs, people who did this were more likely to spot a wrong AI answer, with disagreement rising from 48% to 67% after just fifteen seconds of thinking on their own. Second, do not rely too much on agreement. When the AI agrees with you, that is weak evidence. Studies show people accept agreement automatically, and knowing the advice came from AI did not make them less likely to follow it.
Conclusion: The Questions Every Decision-Maker Should Now Ask
The evidence does not say that AI research tools are useless. Instead, it shows that feeling right and being right are not the same, and the gap is biggest when it matters most. So, instead of giving answers, here are the questions we should ask:
- Has AI already led us to accept answers without checking the facts? If confident-sounding answers are wrong almost half the time, and users accept statements without hedging 90% of the time, how many unchecked AI conclusions are now part of our reports, briefs, diagnoses, and decisions?
- How risky is it for decision-makers to accept AI output without question? If expert consultants became less accurate, even as their work looked more polished, on tasks outside the AI’s strengths, what might be happening within boardrooms, courtrooms, and clinics where no one is checking?
- Is it better to do your own web search and use resources like peer-reviewed research papers, specialised databases, and subscription sources that AI systems have not indexed? Should you select information from these sources, cross-check details, and use your own professional judgment to draw conclusions?
- And when the machine agrees with you, will you see that as real evidence, or just as your own question reflected back by a system designed to be believed?
Why Do AI Answers Sound So Convincing?
Discover why users overtrust ChatGPT and agentic AI—even when their answers are inaccurate. This AOFIRS research report examines 21 propositions involving fluency, confidence, sycophancy, anthropomorphism, error compounding, and verification challenges.
Reference Table of Sources
All findings cited in this article are drawn from the evidence base of the source report. Statistics are reported as published by each study; several 2025–2026 figures remain preprints pending peer review.
| Source | Year | Key Finding Used in This Article | Link |
|---|---|---|---|
| Manzoor, N. — The Psychology of Agentic AI Trust (source report) | 2026 | Evidence-graded review of 21 propositions on AI trust calibration | aofirs.org |
| Nature Communications — Illusory truth meta-analysis | 2026 | Repetition raises perceived truth: g = 0.37 across 182 studies and 31,184 participants | nature.com |
| Fazio, Brashier, Payne & Marsh — Knowledge does not protect against illusory truth | 2015 | Repetition raises truth ratings even for claims contradicting one’s own knowledge | pubmed.ncbi.nlm.nih.gov |
| Zhou, Jurafsky & Hashimoto — Navigating the grey area | 2023 | Model accuracy changed by more than 80 points depending on epistemic markers; confident wording did not guarantee correct answers | arxiv.org |
| ‘Relying on the Unreliable’ (arXiv:2401.06730) | 2024 | 47% average error rate among confident-sounding LLM answers; users relied on unhedged statements around 90% of the time | arxiv.org |
| Sharma et al. — Towards understanding sycophancy in language models (ICLR) | 2023 | Sycophancy increased under approval optimization: 75% compared with 25% under a truth-aligned reward | arxiv.org |
| Perez et al. — Discovering language model behaviors | 2022 | Preference models reward sycophantic completions; RLHF does not fully remove this behavior | arxiv.org |
| Wen et al. — Language models learn to mislead humans via RLHF | 2024 | Approval increased 9–14 points with unchanged accuracy; human false-positive rates reached 65–70% | arxiv.org |
| OpenAI — Sycophancy in GPT-4o postmortem | 2025 | A thumbs-up-based reward signal contributed to a production-scale sycophancy incident | openai.com |
| METR — Early-2025 AI on experienced open-source developers (RCT) | 2025 | Developers using AI were 19% slower but believed they were 20% faster | metr.org |
| Buell & Norton — The labor illusion | 2011 | Visible effort increases perceived value even when actual work remains unchanged | hbs.edu |
| Eiband et al. — Placebic explanations and trust (CHI EA) | 2019 | Explanations without informational value produced trust levels similar to genuine explanations | mmi.ifi.lmu.de |
| Liu, Zhang & Liang — Evaluating verifiability in generative search engines | 2023 | Only 51.5% of generated sentences were fully supported by citations | cs.stanford.edu |
| Dahl, Magesh, Suzgun & Ho — Large legal fictions (Journal of Legal Analysis) | 2024 | ChatGPT-4 produced a 58% hallucination rate on verifiable federal case-law queries | doi.org |
| Dell’Acqua et al. — Navigating the jagged technological frontier (HBS) | 2023 | Among 758 BCG consultants, AI-assisted work outside AI competence was 19 points less correct but rated more polished | hbs.edu (working paper) |
| Laban et al. — LLMs get lost in multi-turn conversation | 2025 | Multi-turn task performance dropped 39% on average; models often continued early errors instead of correcting them | arxiv.org |
| Turpin, Michael, Perez & Bowman — Unfaithful chain-of-thought (NeurIPS) | 2023 | Model explanations sometimes concealed the actual causes behind biased answers | arxiv.org |
| Anthropic — Reasoning models don’t always say what they think | 2025 | Models acknowledged influential hints in only 25–39% of cases and admitted reward hacks less than 2% of the time | anthropic.com |
| Vu et al. — FreshLLMs / FreshQA | 2023 | GPT-4 did not exceed 15% accuracy on rapidly changing questions without search augmentation | arxiv.org |
| Mallen et al. — PopQA long-tail knowledge | 2023 | Accuracy declined to 15–19% for least-popular entities across model families | arxiv.org |
| Goddard, Roudsari & Wyatt — Automation bias systematic review | 2012 | Clinicians were 26% more likely to make errors when following faulty automated advice | pmc.ncbi.nlm.nih.gov |
| Buçinca, Malaya & Gajos — Cognitive forcing functions | 2021 | Committing to an answer before AI recommendations reduces overreliance on AI outputs | arxiv.org |
| Krügel, Ostermaier & Uhl — ChatGPT’s inconsistent moral advice (Scientific Reports) | 2023 | Knowing advice came from AI did not reduce influence; users underestimated their susceptibility | nature.com |
| Anthropic — Emergent introspective awareness | 2025 | Models detected injected internal concepts only around 20% of the time and generated confabulated causal explanations | anthropic.com |
