Most of us know ChatGPT isn't perfect. It can hallucinate facts, confidently invent citations and occasionally give completely incorrect answers. But a new study made me wonder whether the bigger problem isn't what AI gets wrong, but how AI might be changing about us as critical thinkers.
Researchers found that when people relied on an AI assistant for difficult questions, they became far less likely to admit they didn't know the answer. Even more surprising, they grew much more confident in their responses, even when the AI led them straight into an error.
That hit a little too close to home. I use ChatGPT all the time as a brainstorming partner, research assistant and sounding board. So, I decided to recreate the experiment myself to see whether I had quietly fallen into the same habit.
The results weren't exactly what I expected.
What the study found
Researchers from Carnegie Mellon University and Microsoft Research designed an experiment around a simple question: What happens when people consult AI instead of relying on their own judgment?
To find out, they intentionally used questions that the AI was likely to answer incorrectly. Participants could choose one of three options:
Answer on their own
Ask the AI for help
Say, "I don't know"
Without AI, participants admitted they didn't know the answer nearly half the time. Once AI entered the picture, that changed dramatically.
Instead of saying "I don't know," most participants accepted the AI's suggestion — even when it was wrong. Their confidence increased sharply, but their actual accuracy stagnated. Humans were simply becoming confidently incorrect.
The researchers argue that this points to automation bias: our tendency to trust computer-generated recommendations simply because they're presented confidently.
This study suggests that a polished, authoritative answer can make us feel more certain than the evidence warrants.
I decided to try it myself
Rather than recreate the exact academic experiment, I came up with my own version using ChatGPT. I chose questions I suspected would be difficult or easy to confuse, including obscure movie trivia, older technology, historical details and image-based questions. I even made up a completely untrue question about greeting cards made out of cockroaches.
For each one, I followed a simple, three-step process:
I guessed the answer first based entirely on my own memory.
I asked ChatGPT the exact same question.
I compared both answers with reliable, verified sources.
Almost immediately, I noticed that whenever ChatGPT answered confidently, I found myself questioning my own instincts — even when my original guess turned out to be correct.
For a fleeting second, I actually questioned my own childhood memory. “Well, ChatGPT probably knows better,” I caught myself thinking. And that's the behavior the researchers described. The confidence of the response mattered almost as much as the information itself.
Confidence can be surprisingly persuasive
One thing I've noticed as a ChatGPT power user is that the AI never sounds unsure or even admits it does not know something. Even when it includes phrases like "I believe" or "it's likely," the overall response is truly convincing.
Humans naturally associate confident communication with expertise. That's usually a useful shortcut in everyday life, but it's a dangerous strategy when interacting with AI.
Unlike a human expert, ChatGPT isn't checking a database or recalling a verified memory. Sure, it can search the web at lighning speed, but even then it can pick up wrong information. It's essentially generating the most statistically likely next words based on patterns in its training.
Sometimes that's incredibly accurate, but a lot of times that is confidently wrong. The challenge is that those two responses look absolutely identical.
How this study changed my workflow
I've always been cautious about accepting AI's first response because I know how wrong it can be. But this study is certainly a wakeup call. ChatGPT is a powerful productivity tool, but just because it delivers confident answers, it's worth doing your own due diligence to verify what it says are facts.
For me, that means:
Checking original sources rather than taking the AI's summary at face value.
Confirming statistics and data points through independent search engines.
Making sure quotations, dates and names match reputable, primary references.
It only takes a minute or two, but it's a habit worth building. If anything, this study reminded me that the smartest way to use AI is to maintain your own crtitical thinking skills because the final decision is still yours.
Follow Tom's Guide on Google News and add us as a preferred source to get our up-to-date news, analysis, and reviews in your feeds. Subscribe to Tom's Guide on YouTube and follow us on TikTok. Finally, you can visit our dedicated Tom's Guide Savings Squad hub for expert help on getting the best products for less.
More from Tom's Guide
