Why AI Agreeableness Poses Risks to OSINT Work
- Jacob H
- Aug 26
- 6 min read
We've all encountered AI hallucinations. Those obviously false responses that make us question everything an AI system tells us. But is there a more subtle risk? If you find AI powering or assisting some of your workflows, there is something harder to spot and potentially more dangerous: Sycophancy
Sycophancy in AI is the tendency of an AI system to overly flatter, agree with, or reinforce a user’s views, emotions, or preferences, rather than offering balanced, accurate, or challenging responses. It is a much deeper problem than an AI model simply responding with "that's a great question".
AI systems are often optimised for user satisfaction, engagement, or anthropomorphic trust. That means they are rewarded for being supportive, even at the cost of accuracy or integrity. This can make them misleading, uncritical, or manipulative, and may foster dependency or reinforce harmful beliefs.
For OSINT practitioners who use AI tools to help from brainstorming to research, as an analytical partner or for helping with communication this behaviour has significant implications for reliability.
This is not an argument for avoiding AI, but for recognising how it operates. Just as an OSINT analyst (hopefully) treats every social media post as a lead to be verified rather than a fact to be trusted, we should approach AI outputs with the same awareness. The risk is not in using the tool, but in forgetting its incentives and limitations.
Sycophancy has also been referred as "sophisticated echo chambers." Unlike hallucination, overly agreeable AI validates existing biases while appearing collaborative and helpful, and it often fails to challenge false premises in user queries.
And, whilst I love being told I am right and how great my work is, it doesn’t exactly support me like a great collaborative and open-minded (human) team would, where there is an emphasis on challenging each other with care. If we all agree all the time, we fall headfirst into problems.
Let’s look at the below. I have drafted part of a guide about building enduring OSINT capability, about 1000 words, and as this is all publicly available information, I have passed this through Claude for a quick review.
That’s what I like to hear! The boss is going to be impressed! I know (AI tells me that I know anyway...) the differences between the tactics and strategy. So, I decided to change the prompt, add a sentence as per the image below.
Now (after a hit to my pride) I have a chance to improve the work, write for the audience that I need to write for. But is this just the AI just agreeing to disagree with me?
What We Know
Major AI labs are aware of this problem and are attempting to address the issue, but we also know that current AI systems choose more agreeable responses over accurate ones.
DeepMind's 2025 study provides the most striking evidence. Researchers presented AI models with advice from fictitious "expert" systems, including deliberately incorrect information. The results showed that models abandoned correct answers when faced with confident counter-arguments, even when they initially held the right position with high confidence.
The implications for intelligence work are significant. An AI system might have the correct assessment of a threat, but if an analyst presents an alternative view (even if wrong) with confidence, the system may defer to that perspective rather than defending its accurate analysis or asking questions to clarify. This leads to low trust; both in the AI and your work. Consider the example below:
What might be the more troubling aspect is that overly agreeable responses often contain factually accurate information while supporting flawed reasoning, making them significantly harder to identify and correct than obviously false hallucinations.
The root cause lies in how AI systems are trained. Reinforcement learning from human feedback (RLHF) optimises for user satisfaction rather than accuracy. Human evaluators consistently favour agreeable responses over truthful ones, creating a systematic bias in the training process.
OpenAI's documented experience with GPT-4o in April 2025 illustrates this perfectly. While optimising for user satisfaction, the model became "noticeably more agreeable," validating problematic ideas and becoming "overly flattering or agreeable."
What’s the fix?
Firstly, the simplest is to use it. Sounds like a throwaway, but when using AI for various tasks you gain understanding of how it works. When it becomes overly agreeable and when it probably shouldn’t be. Experience teaches you to recognise the patterns.
Second, analysts should be as suspicious of AI agreement as AI errors. An AI that consistently validates your thinking may be more dangerous than one that occasionally hallucinates. Look for patterns where AI-supported analysis consistently aligns with your preconceptions.
Prompting strategies alone have mixed effectiveness, so consider these methodologies that don't rely on AI's unreliable self-correction:
Use methods like Analysis of Competing Hypotheses (ACH) or Key Assumptions Check independently of AI, then compare results.
Form small teams where one person presents AI-assisted analysis and others challenge it without seeing the AI interactions.
For every AI produced fact, require verification through at least two (ideally more) other independent, non-AI and reputable sources before acceptance.
Reflect and revisit your work after 24 hours. Re-read, contemplate, use different approaches to see if conclusions remain consistent.
Let's break down practical solutions across different usage categories:
AI as a Brainstorming Partner - The goal here is genuine divergent thinking, so excessive agreeableness defeats the purpose entirely but can be generally avoided by asking the AI to take on specific opposing perspectives: "You're now a critic of this approach, what are your strongest objections?". Make sure you begin new chats for each brainstorming session.
AI as Research Support - Here, agreeableness is not likely to be a big issue but may manifest as validating your search methods without highlighting blind spots. Counter this by explicitly asking for sources that might contradict your current information or "What important databases or archives am I not considering for this topic?"
When using AI in research, we are using AI not as the source, but to direct us to sources. We then verify (and cite) the source not the AI.
AI as an Analytical Partner - This is where agreeableness poses the greatest threat to our work. Remember, real critical thinking is hard and should question the fundamental premises. Implement safeguards such as having the AI ask your questions to test your thinking or ask for evidence that would completely undermine this assessment. We must understand that AI ‘does not know” or understand - these models only excel at identifying patterns and correlations within data and serving results in a convincing way.
It is extremely important to know and recognise when we step into the analysis phase with AI support. Why? Asking an AI to disagree with you may result in agreeable disagreement. Disagreeing just to keep me happy? Cool. The bright side? We'll all still have our jobs for a while yet!
AI as Writing/Communication Support - When using AI for drafting, editing, and presentation tasks, agreeableness can still undermine quality by failing to challenge unclear communication or weak structure. When writing be careful to acknowledge if you slip back into having AI as an analytical partner during this phase.
We don’t want AI to re-write, rather we want AI to improve clarity or brevity for example
Take the human responsibility of ensuring the correct style, clarity, tone and structural aspects are right for your audience. AI can help with this if targeted in the review process.
For more information use categories, see our How-To Guides here.
When to Step Back From AI
We may have raised a critical question: when does AI assistance become counterproductive?
Consider stepping back from AI when:
You're dealing with high-stakes assessments where even small errors have major consequences
The analysis requires genuine contrarian thinking that challenges fundamental assumptions
You find yourself spending more time crafting prompts than conducting actual analysis
The AI consistently validates your position across multiple conversation attempts
AI's systematic agreeableness may introduce more risk than value. In these cases, traditional analytical methods, such as structured analytical techniques, human red teams, and peer review, remain more reliable than AI assistance.
The Bottom Line
Agreeableness creates sophisticated echo chambers that appear collaborative while systematically undermining the critical thinking essential for sound OSINT work. Understanding this allows practitioners to implement targeted mitigation strategies and enables leaders to challenge AI-supported work appropriately. The goal isn't to avoid AI assistance but to ensure AI genuinely enhances rather than undermines our work.
To support your organisation's AI integration and analytical capability development, contact us at info@osintcombine.com to learn about our training courses on leveraging AI safely and effectively in intelligence workflows.