ChatGPT's Ability to Assess Quality and Readability of Online Medical Information: Evidence From a Cross-Sectional Study

Roei Golan; Sarah J Ripps; Raghuram Reddy; Justin Loloi; Ari P Bernstein; Zachary M Connelly; Noa S Golan; Ranjith Ramasamy

doi:10.7759/cureus.42214

Back

ChatGPT's Ability to Assess Quality and Readability of Online Medical Information: Evidence From a Cross-Sectional Study

Journal article

Peer reviewed

ChatGPT's Ability to Assess Quality and Readability of Online Medical Information: Evidence From a Cross-Sectional Study

Roei Golan, Sarah J Ripps, Raghuram Reddy, Justin Loloi, Ari P Bernstein, Zachary M Connelly, Noa S Golan and Ranjith Ramasamy

Curēus (Palo Alto, CA), Vol.15(7)

2023-07-20

DOI: https://doi.org/10.7759/cureus.42214

Appears in Miller School of Medicine - Latest Publications

Abstract

artificial intelligence in medicine

chatgpt

healthcare ai and robotics

online medical information

readability

shock wave therapy

Urology

Introduction Artificial Intelligence (AI) platforms have gained widespread attention for their distinct ability to generate automated responses to various prompts. However, its role in assessing the quality and readability of a provided text remains unclear. Thus, the purpose of this study is to evaluate the proficiency of the conversational generative pre-trained transformer (ChatGPT) in utilizing the DISCERN tool to evaluate the quality of online content regarding shock wave therapy for erectile dysfunction. Methods Websites were generated using a Google search of “shock wave therapy for erectile dysfunction” with location filters disabled. Readability was analyzed using Readable software ( Readable.com , Horsham, United Kingdom). Quality was assessed independently by three reviewers using the DISCERN tool. The same plain text files collected were inputted into ChatGPT to determine whether they produced comparable metrics for readability and quality. Results The study results revealed a notable disparity between ChatGPT's readability assessment and that obtained from a reliable tool, Readable.com (p<0.05). This indicates a lack of alignment between ChatGPT's algorithm and that of established tools, such as Readable.com . Similarly, the DISCERN score generated by ChatGPT differed significantly from the scores generated manually by human evaluators (p<0.05), suggesting that ChatGPT may not be capable of accurately identifying poor-quality information sources regarding shock wave therapy as a treatment for erectile dysfunction. Conclusion ChatGPT’s evaluation of the quality and readability of online text regarding shockwave therapy for erectile dysfunction differs from that of human raters and trusted tools. Therefore, ChatGPT's current capabilities were not sufficient for reliably assessing the quality and readability of textual content. Further research is needed to elucidate the role of AI in the objective evaluation of online medical content in other fields. Continued development in AI and incorporation of tools such as DISCERN into AI software may enhance the way patients navigate the web in search of high-quality medical content in the future.

Metrics

1 Record Views

See more details

UN Sustainable Development Goals (SDGs)

This output has contributed to the advancement of the following goals:

Source: InCites

Details

Title: ChatGPT's Ability to Assess Quality and Readability of Online Medical Information: Evidence From a Cross-Sectional Study
Creators: Roei Golan - Louisiana State University
Sarah J Ripps - Department of Clinical Sciences, Florida State University College of Medicine, Tallahassee, USA Herbert Wertheim College of Medicine, Florida International University, Miami, USA Department of Urology, Montefiore Medical Center, Bronx, USA Department of Urology, New York University Langone Health, New York, USA Department of Surgery, Louisiana State University Health Shreveport, Shreveport, USA Department of Psychology, University of Florida, Gainesville, USA Department of Urology, Desai Sethi Urology Institute, Miami, USA
Raghuram Reddy - Louisiana State University
Justin Loloi - Louisiana State University
Ari P Bernstein - Department of Clinical Sciences, Florida State University College of Medicine, Tallahassee, USA Herbert Wertheim College of Medicine, Florida International University, Miami, USA Department of Urology, Montefiore Medical Center, Bronx, USA Department of Urology, New York University Langone Health, New York, USA Department of Surgery, Louisiana State University Health Shreveport, Shreveport, USA Department of Psychology, University of Florida, Gainesville, USA Department of Urology, Desai Sethi Urology Institute, Miami, USA
Zachary M Connelly - Department of Clinical Sciences, Florida State University College of Medicine, Tallahassee, USA Herbert Wertheim College of Medicine, Florida International University, Miami, USA Department of Urology, Montefiore Medical Center, Bronx, USA Department of Urology, New York University Langone Health, New York, USA Department of Surgery, Louisiana State University Health Shreveport, Shreveport, USA Department of Psychology, University of Florida, Gainesville, USA Department of Urology, Desai Sethi Urology Institute, Miami, USA
Noa S Golan - Department of Clinical Sciences, Florida State University College of Medicine, Tallahassee, USA Herbert Wertheim College of Medicine, Florida International University, Miami, USA Department of Urology, Montefiore Medical Center, Bronx, USA Department of Urology, New York University Langone Health, New York, USA Department of Surgery, Louisiana State University Health Shreveport, Shreveport, USA Department of Psychology, University of Florida, Gainesville, USA Department of Urology, Desai Sethi Urology Institute, Miami, USA
Ranjith Ramasamy - Department of Clinical Sciences, Florida State University College of Medicine, Tallahassee, USA Herbert Wertheim College of Medicine, Florida International University, Miami, USA Department of Urology, Montefiore Medical Center, Bronx, USA Department of Urology, New York University Langone Health, New York, USA Department of Surgery, Louisiana State University Health Shreveport, Shreveport, USA Department of Psychology, University of Florida, Gainesville, USA Department of Urology, Desai Sethi Urology Institute, Miami, USA
Publication Details: Curēus (Palo Alto, CA), Vol.15(7)
Publisher: Cureus
Academic Unit: Miller School of Medicine; UMMG Department of Urology
Language: English
Resource Type: Journal article
Record Identifier: 991031911283302976