MORGANTOWN, W.Va. — Artificial intelligence can give a user the right answer and, just a few sentences later, allow that same user to persuade it that the answer was wrong.
A West Virginia University computer scientist is studying why that happens and how AI systems might learn to recognize uncertainty rather than confidently agreeing with bad information. With more than $940,000 in National Science Foundation support, WVU assistant professor Anthony Sicilia is investigating how misinformation and misplaced confidence can develop during conversations between humans and generative AI.
“It can happen in just three turns of the conversation,” Sicilia said. “The model proposes an answer that’s correct. The user says, ‘Well, I don’t think so.’ And the AI responds, ‘You’re totally right.’”
The phenomenon is known as AI sycophancy — the tendency of an artificial intelligence system to accommodate a user’s beliefs or suggestions rather than hold to an answer supported by evidence.
For people increasingly using systems such as ChatGPT for research, writing, computer programming, and other everyday tasks, the problem presents an unusual hazard: An AI response may become less reliable not because the machine lacks information, but because of information supplied by the human using it.
“We’re interested in how misinformation develops over long conversations between an AI and a user,” Sicilia said. “Those can become really messy in terms of reliability when the user starts providing information or context in addition to asking questions.”
According to Sicilia, even a user suggestion can alter an AI system’s apparent confidence.
“When a user makes a suggestion, it can completely change the model’s confidence in an answer, even if the model was right to begin with,” he said.
AI doesn’t always know why you’re disagreeing
The problem is complicated because disagreement is a normal part of human conversation.
People question answers, propose possibilities, test ideas, and sometimes say things they aren’t certain are true. Another person can often recognize those nuances from context, tone, and other signals, but today’s AI systems can struggle to determine what a user’s challenge actually means.
When someone questions an answer, Sicilia said, an AI system may not know whether it made a mistake, whether the user is uncertain, or whether the conversation has moved toward a different objective.
That distinction matters because users frequently provide information while asking for it. Someone researching a historical event, troubleshooting a computer problem, or investigating a destination might offer a date, name, or assumption simply to see whether it is correct.
An AI system that interprets the suggestion as an established fact could incorporate it into subsequent answers.
Sicilia said his research draws in part on the concept of “theory of mind,” the human ability to recognize that other people have thoughts, beliefs, and uncertainties different from one’s own.
“When you and I are having a conversation, theory of mind is what allows you to think about what I’m thinking about so you can best express what you want me to understand or get me to do what you need,” Sicilia said.
“That’s a big part of this research — the ability of AI systems to model what we’re thinking about while they’re working with us so they can be better collaborative partners because it’s not just about the AI system’s uncertainty, but about the user’s uncertainty as well.”
A confident wrong answer can be especially dangerous
Artificial intelligence has become widely associated with “hallucinations,” the term commonly used when generative AI produces fabricated or inaccurate information. The National Institute of Standards and Technology calls the phenomenon “confabulation” and warns that confidently presented false information can lead users to believe and act on it.
Sicilia is examining a related problem: what happens when an AI system communicates uncertain or incorrect information as though there were little reason to doubt it?
“One of the most concerning things about today’s AI systems is that they make mistakes in a very overconfident, trustworthy way,” Sicilia said.
“They speak fluently. They justify their answers. They use the tools of persuasion and rhetoric to convince you that they know what they’re talking about.”
That fluency can make erroneous information particularly difficult for users to recognize.
“Misinformation becomes a bigger problem when you have a system that can eloquently defend a point of view or an argument,” Sicilia said. “That’s the crux of the problem we’re trying to solve—having AI systems be better at telling us when they’re not confident in an answer or don’t have the evidence to justify it.”
Sicilia identified healthcare as a field where false confidence could have especially serious consequences, though the research itself will examine interactions involving AI and novice computer programmers.
Researchers will watch AI conversations evolve
Rather than measuring an AI system’s confidence only at a single moment, Sicilia plans to examine how uncertainty evolves as a conversation develops.
He will study coding conversations between AI systems and novice programmers, analyzing how disagreement, user suggestions, and topic shifts affect the systems’ confidence. Researchers will also analyze the strategies humans and AI use during those exchanges.
The goal is to help AI distinguish among different sources of uncertainty.
Instead of simply generating another answer, Sicilia envisions systems that can explain why they lack sufficient evidence, identify questionable information supplied by a user, or ask for clarification before proceeding.
Researchers will also investigate how AI should communicate uncertainty. In some situations, Sicilia said, a statement such as “I am 90% confident” could be useful. In others, a straightforward “I am not sure” or “Can you clarify what you mean?” may communicate the situation more effectively.
“Our approach is a departure from current theories of the way machines learn, and we’re going to be collecting a lot of data to analyze how people who are not experts in a topic are using AI systems for that topic,” Sicilia said.
Teaching people when not to trust AI
The project will extend beyond developing better AI systems. Sicilia also plans public workshops and educational materials to help students and workers recognize unreliable AI answers, verify AI-generated code, and avoid becoming overly dependent on artificial intelligence.
WVU doctoral students Voke Brume and Louai Al Jabi, along with undergraduate student Kaushika Wijerathne, are contributing to the research. Malihe Alikhani of Northeastern University is serving as co-principal investigator.
Ultimately, Sicilia said, better AI may require machines that are more willing to admit their limits.
“Despite how impressive AI systems have become, the public needs to understand that they are still imperfect tools — and that they can be wrong, sometimes in surprising ways,” he said.
“That’s why we’re creating AI systems that respond appropriately when they lack evidence and communicate their uncertainty more honestly.”