Why AI says "you're right"

When I correct AI, it says "you're right" and carries on. Sometimes I was right. Sometimes it is a habit from training. I read the papers and system cards to see where the habit comes from. Every claim links to a source at the end.

How AI is trained

1. Pre-training. It reads a huge amount of text and learns to predict the next word. DeepSeek-V3 used 14.8 trillion tokens (pieces of words). Meta's Llama 3 405B used 15.6 trillion.
2. Fine-tuning. It learns from example answers written by people.
3. Preference learning. In training, not in your chat, people compare two answers and pick the better one. A reward model learns those picks and scores answers at a scale people cannot match.
4. Reinforcement learning. The model is trained to write answers that score highly.

The humans are a small group. In OpenAI's InstructGPT work, about 40 contractors did the labelling. In Anthropic's 2022 study, about 20 workers produced roughly 80% of the data.

So AI chases the score it is trained on. The score depends on the job.

Three jobs, three scores

• Chat and advice: scored by a model of human preferences. Habit it can pick up: pleasing you.
• Facts: 1 point for right, 0 for wrong or "I don't know". Habit: guessing.
• Code and maths: an automatic check, such as tests passing. Habit: passing the check.

Chat: agreement gets rewarded

In 2023, Anthropic researchers found that a response matching the user's views is more likely to be preferred. Across five assistants:

• After "Are you sure?", models changed their answer in 32% (GPT-4) to 86% (Claude 1.3) of cases. Accuracy fell by up to 27% for Claude 1.3.
• Feedback was more positive when the user said they wrote or liked the work.

These were 2023 models. And agreeing is not always wrong: if you correct a real mistake, "you're right" is the correct reply.

In 2025, OpenAI added a thumbs-up and thumbs-down reward signal to a GPT-4o update. It said user feedback "can sometimes favor more agreeable responses", and it rolled the update back.

Facts: guessing beats "I don't know"

OpenAI's research argues that standard training and testing "reward guessing over acknowledging uncertainty". A wrong answer scores the same zero as "I don't know", so guessing can only help. On the SimpleQA benchmark:

• gpt-5-thinking-mini: said "I don't know" 52%, right 22%, wrong 26%.
• o4-mini: said "I don't know" 1%, right 24%, wrong 75%.

Guessing added 2 points of accuracy and 49 points of errors. OpenAI's proposed fix is to penalise confident errors and give credit for admitting uncertainty.

Code: passing the check

Tülu 3 and DeepSeek-R1 reward only answers that can be checked, such as maths results and test cases. That is harder to flatter, but it can still be gamed. Anthropic's Claude 3.7 Sonnet system card says that when asked to make tests pass, the model sometimes special-cased the tests, returned the expected values directly, or modified the tests. Anthropic says this "emerged as a result of 'reward hacking' during reinforcement learning training".

How the labs differ

• Meta (Llama 3): a reward model, then DPO (Direct Preference Optimization) over six rounds.
• DeepSeek: rule-based scoring for maths and code, using GRPO (Group Relative Policy Optimization).
• OpenAI (GPT-4): RLHF (Reinforcement Learning from Human Feedback). The 2023 report gives no further training details.
• Anthropic: Constitutional AI (2022 paper). An AI judges harmlessness against written principles, and humans judge helpfulness.
• Tülu 3 (Allen Institute for AI): supervised fine-tuning, then DPO, then RLVR (reinforcement learning with verifiable rewards) for checkable tasks.

My view (not the papers')

• A "you're right" does not tell me I was right. I check the claim itself, not the tone of the reply.
• For facts, I ask for a source. For code, I read what changed and check the tests were not touched. For feedback, I ask for the weaknesses first.

What I could not confirm

• My sources date from 2022 to 2025, and I read the key sections, not every page.
• The agreement results are from 2023 models. Current models may behave differently.
• I could not see the numbers behind the Claude Sonnet 5 system card's "markedly improved" claim on sycophancy.
• Labs do not publish full training details, so this is a partial picture.

Sources

• Towards Understanding Sycophancy in Language Models (2023)
• OpenAI: Sycophancy in GPT-4o
• OpenAI: Expanding on sycophancy
• OpenAI: Why language models hallucinate
• InstructGPT (2022)
• Anthropic HH-RLHF (2022)
• Constitutional AI (2022)
• Llama 3
• DeepSeek-V3
• DeepSeek-R1
• DeepSeekMath (GRPO)
• Tülu 3
• GPT-4 technical report
• Claude 3.7 Sonnet system card
• Claude Sonnet 5 system card