Evaluation of Quality of AI Chatbots in Patient Education Guidelines on Dental Implant Maintenance

Abstract

Introduction: The use of Artificial intelligence (AI) chatbots in medical science has been ever increasing. Patients rely on the information provided by these chatbots. The assessment of these chatbots is of absolute necessity for the detection of accuracy of their responses. The patient education guidelines have to be clear and useful in a structured and understandable sentence pattern, along with proper referencing.

Methods: Patient education guideline on dental implant was generated. Two investigators scored the chatbots based on Quality Analysis of Medical Artificial Intelligence (QAMAI) scores of two chatbots: Chat GPT and Claude. The dimensions detected were accuracy, clarity, relevance, completeness, references and usefulness. Statistical analysis were carried out by unpaired t test.

Results: The QAMAI scores for both Chat GPT and Claude were good. There were no statistically different differences among the QAMAI score of the two chatbots. (p=0.73)

Conclusion: The patient education guide provided by Chat GPT and Claude are satisfactory. There is, however, room for improvement in increasing the quality of the AI chatbots.