Research Focus: Dhaimade

Prita Dhaimade, BDS, MS

Dr. Prita Dhaimade, clinical assistant professor in the Division of Periodontology in the Department of Surgical Sciences, published her research May 29 in PLOS Digital Health.

“Multidimensional evaluation of large language models on the AAP in-service examination: assessing accuracy, calibration and citation reliability” explores a comprehensive assessment of commercially available large language models (LLM) relating to in specialty-specific domains such as dentistry and periodontology.

“As large language models become increasingly embedded in how students learn and how clinicians access information, a critical question emerges: Are these tools actually reliable in specialty medical contexts? My research addresses that question directly, using periodontology as a testing ground,” Dhaimade said.

Her work, published alongside co-author Dr Robin Henderson, evaluates three commercially available AI systems: GPT-4.0, GPT-5.0 and Claude Sonnet 4.0, on the 2024 AAP In-Service Examination, assessing not just accuracy but confidence calibration, citation validity and performance consistency across question formats.

“While overall accuracy was strong (78–87%), the nuances tell a more important story,” Dhaimade said.“Performance was weakest in diagnosis and clinical reasoning, the domains most relevant to patient care, and AI-generated citations were frequently fabricated, regardless of whether the answer itself was correct.”

The research shines a light on how dental students could utilize AI in their dental education in the future.

“These findings have meaningful implications for dental education,” Dhaimade said. “Students increasingly rely on AI as a study and reference tool, often without a framework for evaluating its outputs. This research offers that framework, and makes the case for deliberate, evidence-informed integration of AI into health professions curricula rather than passive adoption.”

PLOS Digital Health is an interdisciplinary Open Access journal publishing boundary-breaking original academic and clinical research that uses open science practices, digital tools, technologies and data science to significantly advance human health and health care.

Dhaimade said her research showcases the school’s mission to prepare graduates who have a strong understanding of resources around them.

“For the ECU School of Dental Medicine, engagement in this kind of work reflects our broader mission,” she said. “Serving a region with significant healthcare disparities means our graduates must be especially well-prepared, critical thinkers who understand both the potential and the limitations of the tools available to them. Contributing to the national conversation on AI reliability in dental education is a natural extension of that commitment.”