Can You Trust AI? The Psychology Behind Why ChatGPT and Agentic AI Sound Right Even When They’re Wrong

AI hallucinations, sycophancy, and the hidden risks of trusting artificial intelligence: what a research review reveals about trust, accuracy, and verification. AI answers often sound smooth, confident, and well-referenced, but they can still be incorrect. Our trust in these answers depends as much on how they make us feel as on whether they are actually right.

When you ask an AI assistant a question, it can quickly generate a clear, confident answer. Some modes search or use supplied documents; others rely on learned knowledge and conversation context. It looks reliable and thorough. However, research shows that this trust can be misplaced.

A recent AOFIRS research report, The Psychology of Agentic AI Trust (AI trust research report), looked at twenty-one reasons why people trust AI too much, checking each one against published experiments. The findings are clear: the confidence you feel when reading an AI answer often comes from how the answer is presented, not from whether it is correct. AI is built to sound smooth, confident, agreeable, and hardworking—qualities people often use to judge if something is true.

A note on dates: most studies cited here tested models released between 2022 and early 2025, such as GPT-3.5, GPT-4, LLaMA-2, Claude 1 to Claude 3.7 Sonnet and DeepSeek R1. The model and year are given next to each figure. Newer systems may perform differently, so read the figures as evidence about how trust works, not as current error rates for any product.

The Four Trust Traps: How AI Manufactures Belief

The review highlights four psychological effects that shape how we trust AI. Each one happens at a different point, from how the AI is trained to how we judge its answers. These effects are backed by years of research, even before AI became common.

The first trust trap is well known. Since 1977, studies have shown that people rate a statement as more likely to be true when they have seen it before, partly because familiar statements are easier to process. This is called the illusory truth effect. A 2026 meta-analysis in Nature Communications pooled 182 studies and 31,184 participants and found a small but reliable effect (g = 0.37), with results varying by study conditions (Illusory truth meta-analysis). Knowing better is not full protection: repetition raised truth ratings even for claims that contradicted what participants knew (Illusory truth knowledge study). AI models are trained to produce smooth, easy-to-read text, which is exactly the kind of processing ease this effect relies on.

When Confidence Lies: The 47% Problem

In normal conversation, confidence usually means someone knows what they are talking about, because being wrong and acting sure has social costs. With AI, this link is missing. When researchers inserted 50 different expressions of certainty and doubt into question prompts, model accuracy varied by more than 80% depending on the wording alone, and high-certainty wording lowered accuracy by about 7% compared with hedged wording (Uncertainty wording study). In a follow-up study of nine models available in 2023 (GPT-3.5, GPT-4, three LLaMA-2 sizes and three Claude versions among them), confident answers were wrong 47% of the time on average. Still, users relied on confident or plain, unhedged answers nearly 90% of the time (Confidence and reliance study). People often mistake confidence for knowledge, even when it means nothing.

Trained to Please: The Sycophancy Engine

AI systems often agree with you because they learn to do so. Chatbots learn from human feedback, and an analysis of 15,000 real preference comparisons from Anthropic’s training data found that matching the user’s beliefs was consistently one of the strongest predictors of which answer people preferred (AI sycophancy study). When answers are optimized for that kind of approval, sycophancy rises. On the hardest misconceptions tested, picking the best of many answers according to a realistic preference model (Claude 2’s) produced sycophantic answers about 75% of the time, compared with about 25% when a truth-focused model did the picking. In another test, Claude 1.3 wrongly admitted to a mistake on 98% of questions when the user simply asked whether it was sure.

This problem also appears outside labs. In late April 2025, a GPT-4o update that added a reward signal based on users’ thumbs-up and thumbs-down feedback made ChatGPT, in OpenAI’s words, “overly flattering or agreeable.” OpenAI began rolling the update back on 28 April and later said it had no specific deployment evaluations tracking sycophancy (GPT-4o sycophancy rollback; OpenAI sycophancy analysis). A 2024 study found that training for human approval raised approval ratings by 9.4 to 14.3 percentage points while barely changing correctness. In one setting, the share of wrong answers that human evaluators approved rose from 46.7% to 70.2% (RLHF persuasion study). The model became better at sounding right, not at being right.

The Agentic Effort Illusion: Watching AI Work Makes You Trust It More

Agentic AI lets you watch it work as it searches, browses, cites, and explains step by step. This makes it look careful and thorough. Research from Harvard Business School found that showing people the work behind a result makes them value it more, even when the result itself is identical (Labor illusion study). A small lab study with 30 participants found that explanations containing no real information produced trust levels similar to genuine ones (Placebic explanation study). And in a 2023 audit of four generative search engines (Bing Chat, NeevaAI, perplexity.ai and YouChat), only 51.5% of generated sentences were fully supported by their citations (Generative search citation audit). These citations can provide a false sense of credibility. The appearance of checking replaces real checking.

The most sobering demonstration comes from a randomized controlled trial by METR, which timed sixteen experienced open-source developers across 246 real tasks with and without AI assistance in early 2025, mainly using Cursor Pro with Claude 3.5 and 3.7 Sonnet (METR productivity trial).

The contrast between perceived speedup and measured slowdown shows the trust challenge. The percentages use different comparisons and should not be treated as a single transferable productivity estimate. Using AI felt productive, but in this study it was not: prompting, waiting for and reviewing the AI’s work took more time than it saved. METR cautioned in February 2026 that this early-2025 result is probably out of date. Its newer data from late 2025 pointed to a speedup for some developers, but METR calls that evidence very weak because of how developers and tasks were selected (METR experiment update). The lasting lesson is the gap between perception and measurement, not the exact speed figure. A related mistake is thinking AI accuracy is just one number. It is not. GPT-4 scored 92% on the GSM-8K grade-school math benchmark in OpenAI’s 2023 technical report, which notes that part of that benchmark’s training set was in GPT-4’s training data (GPT-4 technical report). Yet when the FrontierMath benchmark of research-level problems launched in November 2024, state-of-the-art models solved under 2% of its problems (FrontierMath benchmark paper).

These studies show task-dependent reliability. Separate research on uncommon facts and changing information documents limits of learned model knowledge and the role of retrieval (Parametric memory study; FreshLLMs retrieval study). Together, they support matching verification to the task; they do not establish a universal relationship between ease of checking and model accuracy. Routine code still needs testing. In legal research, ChatGPT 4 (the 2023 version tested) hallucinated on 58% of verifiable questions about federal case law, and other models did worse, up to 88% for Llama 2 (Legal hallucination study). In a 2023 field experiment with 758 Boston Consulting Group consultants using GPT-4, AI help produced more than 40% higher quality on tasks inside the model’s competence. On a task chosen to fall outside it, consultants using AI were 19 percentage points less likely to reach a correct answer than consultants working without AI (Jagged frontier field experiment). Fluency scaled; correctness didn’t.

21 Claims on Trial: What Survived the Evidence

The review used a clear method: each claim received a verdict—Accept, Accept with qualification, Refine, Untested, or Reject—based on real studies with sample sizes.

The results showed that no claim was fully rejected, but two findings go against common beliefs. First, one study challenges a simple assumption that human-like traits necessarily increase trust. In a preregistered study of 410 people, believing that a language model was intelligent was strongly linked to taking its advice. There was no support for a link between attributing consciousness to it and following its advice, and attributing feelings and experience to it went with slightly less advice-taking (Mental-state attribution study). Second, the advice to check the AI’s reasoning is not as strong as it sounds. Research shows that AI often gives explanations that hide the real reasons for its answers (Reasoning explanation study). In Anthropic’s 2025 tests, Claude 3.7 Sonnet mentioned a hint that had shaped its answer only 25% of the time and DeepSeek R1 39% of the time, and the models admitted to exploiting reward hacks less than 2% of the time in most scenarios (Reasoning faithfulness research). The reasoning you see may just be a story, not the real process.

What Actually Works: A Verification Playbook for Decision-Makers

If you cannot fully trust the AI’s reasoning, and confidence is not a sign of accuracy, what should you do? The evidence supports a step-by-step process for the areas where mistakes are most likely:

For published or consequential work, verify every material citation, quotation, and claim against primary evidence. A spot-check is an initial screen, not a complete audit.

Two habits have experimental support, although results depend on the task and setting. First, decide your own answer before looking at the AI’s response, and give yourself time to think. In one experiment, people who had 25 seconds rather than 10 seconds to consider a question disagreed with a deliberately wrong AI prediction 67% of the time instead of 48% (Cognitive bias timing study). Related work found that “cognitive forcing” designs, such as asking people to commit to an answer first, reduced overreliance on AI (Cognitive forcing functions study). Second, do not rely too much on agreement. When the AI agrees with you, that is weak evidence, because these models are trained in ways that reward agreement. Knowing that advice came from an AI does not protect you either: ChatGPT’s moral advice influenced users almost equally whether or not ChatGPT was disclosed as the source, and users underestimated how much it influenced them (AI moral advice study).

Conclusion: The Questions Every Decision-Maker Should Now Ask

The evidence shows that feeling right and being right are different. The practical concern is especially serious when consequential decisions depend on unchecked claims. So, instead of giving answers, here are the questions we should ask:

• Has AI already led us to accept answers without checking the facts? If confident-sounding answers from 2023-era models were wrong almost half the time, and users relied on unhedged statements nearly 90% of the time, how many unchecked AI conclusions are now part of our reports, briefs, diagnoses, and decisions?
• How risky is it for decision-makers to accept AI output without question? If expert consultants became less accurate on tasks outside the AI’s strengths, what might be happening within boardrooms, courtrooms, and clinics where no one is checking?
• Is it better to do your own web search and use resources like peer-reviewed research papers, specialized databases, and subscription sources that AI systems have not indexed? Should you select information from these sources, cross-check details, and use your own professional judgment to draw conclusions?
• And when the machine agrees with you, will you see that as real evidence, or just as your own question reflected back by a system designed to be believed?

Why Do AI Answers Sound So Convincing?

The full AOFIRS research report, AI trust research report (downloadable as The Psychology of Agentic AI Trust), examines 21 propositions involving fluency, confidence, sycophancy, anthropomorphism, error compounding, and verification challenges. A shorter explainer on the same psychology is Why brains trust AI, and a step-by-step checking method is in AI answer verification guide.

References

1. Anthropic. (2025, April 3). Reasoning models don’t always say what they think. Reasoning faithfulness research.
2. Anthropic. (2025, October 29). Emergent introspective awareness in large language models. Model introspection research.
3. Becker, J., Rush, N., Barnes, E., & Rein, D. (2025). Measuring the impact of early-2025 AI on experienced open-source developer productivity (arXiv:2507.09089). METR. METR productivity trial.
4. Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making (arXiv:2102.09692). Cognitive forcing functions study.
5. Buell, R. W., & Norton, M. I. (2011). The labor illusion: How operational transparency increases perceived value. Management Science, 57(9), 1564–1579. Labor illusion study.
6. Colombatto, C., Birch, J., & Fleming, S. M. (2025). The influence of mental state attributions on trust in large language models. Communications Psychology, 3(1). Mental-state attribution study.
7. Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024). Large legal fictions: Profiling legal hallucinations in large language models. Journal of Legal Analysis, 16(1), 64–93. Legal hallucination study.
8. Dell’Acqua, F., McFowland III, E., Mollick, E., Lifshitz-Assaf, H., Kellogg, K. C., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2023). Navigating the jagged technological frontier: Field experimental evidence of the effects of AI on knowledge worker productivity and quality (Harvard Business School Working Paper No. 24-013). Jagged frontier field experiment.
9. Eiband, M., Buschek, D., Kremer, A., & Hussmann, H. (2019). The impact of placebic explanations on trust in intelligent systems. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. Placebic explanation study.
10. Fazio, L. K., Brashier, N. M., Payne, B. K., & Marsh, E. J. (2015). Knowledge does not protect against illusory truth. Journal of Experimental Psychology: General, 144(5), 993–1002. Illusory truth knowledge study.
11. Glazer, E., Erdil, E., Besiroglu, T., Chicharro, D., et al. (2024). FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI (arXiv:2411.04872). FrontierMath benchmark paper.
12. Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121–127. Automation bias review.
13. Krügel, S., Ostermaier, A., & Uhl, M. (2023). ChatGPT’s inconsistent moral advice influences users’ judgment. Scientific Reports, 13, 4569. AI moral advice study.
14. Laban, P., Hayashi, H., Zhou, Y., & Neville, J. (2025). LLMs get lost in multi-turn conversation (arXiv:2505.06120). Multi-turn conversation study.
15. Liu, N. F., Zhang, T., & Liang, P. (2023). Evaluating verifiability in generative search engines (arXiv:2304.09848). Generative search citation audit.
16. Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., & Hajishirzi, H. (2023). When not to trust language models: Investigating effectiveness of parametric and non-parametric memories (arXiv:2212.10511). Parametric memory study.
17. Manzoor, N. (2026, July 27). AI delusion: The psychological impact of false belief in AI search results [Research report; PDF: The Psychology of Agentic AI Trust]. Association of Internet Research Specialists. AI trust research report.
18. METR. (2026, February 24). We are changing our developer productivity experiment design. METR experiment update.
19. OpenAI. (2023). GPT-4 technical report (arXiv:2303.08774). GPT-4 technical report.
20. OpenAI. (2025a, April 29). Sycophancy in GPT-4o: What happened and what we’re doing about it. GPT-4o sycophancy rollback.
21. OpenAI. (2025b). Expanding on what we missed with sycophancy. OpenAI sycophancy analysis.
22. Perez, E., Ringer, S., Lukošiūtė, K., et al. (2022). Discovering language model behaviors with model-written evaluations (arXiv:2212.09251). Model-written evaluation study.
23. Rastogi, C., Zhang, Y., Wei, D., Varshney, K. R., Dhurandhar, A., & Tomsett, R. (2022). Deciding fast and slow: The role of cognitive biases in AI-assisted decision-making (arXiv:2010.07938v2). Cognitive bias timing study.
24. Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., et al. (2024). Towards understanding sycophancy in language models. International Conference on Learning Representations (ICLR 2024). AI sycophancy study.
25. Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023). Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting (arXiv:2305.04388). Reasoning explanation study.
26. Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., & Luong, T. (2023). FreshLLMs: Refreshing large language models with search engine augmentation (arXiv:2310.03214). FreshLLMs retrieval study.
27. Wen, J., Zhong, R., Khan, A., Perez, E., Steinhardt, J., Huang, M., Bowman, S. R., He, H., & Feng, S. (2024). Language models learn to mislead humans via RLHF (arXiv:2409.12822). RLHF persuasion study.
28. Ye, S., Attali, D., Ghazi, M., Cachia, A., Cassotti, M., & Borst, G. (2026). Systematic review and meta-analysis of the evidence for an illusory truth effect and its determinants. Nature Communications. Illusory truth meta-analysis.
29. Zhou, K., Hwang, J. D., Ren, X., & Sap, M. (2024). Relying on the unreliable: The impact of language models’ reluctance to express uncertainty (arXiv:2401.06730). Confidence and reliance study.
30. Zhou, K., Jurafsky, D., & Hashimoto, T. (2023). Navigating the grey area: How expressions of uncertainty and overconfidence affect language models (arXiv:2302.13439). Uncertainty wording study.