Search Results

Abstract

PURPOSE: Natural language models and chatbots, particularly OpenAI’s Generative Pre-Trained Transformer architecture, have transformed human interaction with digital interfaces. The latest versions, including ChatGPT-4o, offer enhanced functionalities compared to their predecessors. This study evaluates the accuracy of ChatGPT-4, ChatGPT-4o, and Claude 3.5 Sonnet in answering questions from the Brazilian Retina and Vitreous Society certification exam.
METHODS: We compiled 200 multiple-choice questions from the Brazilian Retina and Vitreous Society 2018 and 2019 exams. Questions were categorized into three domains: Anatomy and Physiology of the Retina, Retinal Pathology, and Diagnosis and Treatment. Using a standardized prompt developed according to prompt design guidelines, we tested ChatGPT-4, ChatGPT-4o, and Claude 3.5 Sonnet, recording their first responses as final. Three retina specialists performed a qualitative analysis of the answers. Accuracy was determined by comparing responses to the official correct answers. Statistical analysis was conducted using chi-square tests and Cohen’s Kappa.
RESULTS: Claude 3.5 Sonnet achieved the highest overall accuracy (72.5%), followed by ChatGPT-4o (66.0%) and ChatGPT-4 (55.5%). Claude 3.5 Sonnet and ChatGPT-4o significantly outperformed ChatGPT-4 (p<0.01 and p=0.03, respectively), while no significant difference was observed between Claude 3.5 Sonnet and ChatGPT-4o (p=0.16). Model responses agreed 74.5% of the time, with a Cohen’s κ of 0.47. Retinal Pathology was the best-performing domain for all models, whereas Anatomy and Physiology of the Retina and Diagnosis and Treatment were the weakest domains for Claude 3.5 Sonnet and ChatGPT-4, respectively.
CONCLUSIONS: This study is the first to assess Claude 3.5 Sonnet, ChatGPT-4, and ChatGPT-4o in retina specialist certification exams. Claude 3.5 Sonnet and ChatGPT-4o significantly outperformed ChatGPT-4, highlighting their potential as effective tools for studying retina specialist board exams. These findings suggest that the enhanced functionalities of Claude 3.5 Sonnet and ChatGPT-4o offer substantial improvements in medical education contexts.

Keywords: Artificial intelligence; ChatGPT; Retina; Medical education; Ophthalmology, Large language model; Natural language processing

Abstract

PURPOSE: To evaluate the academic and professional trajectories of graduates from the Graduate Program in Ophthalmology and Visual Sciences at the Escola Paulista de Medicina, Universidade Federal de São Paulo, including geographic distribution, occupational characteristics, and scientific productivity.
METHODS: This descriptive, retrospective, quantitative study included 498 alumni who completed the program between 1979 and 2021. Data were obtained from institutional records and supplemented by public databases (Google Scholar, Lattes Platform, and LinkedIn). The analyzed variables included demographic characteristics, academic background, current professional role, and bibliometric indicators (citation count and h-index). Statistical analyses comprised nonparametric tests and linear regression, with a significance level set at 5%.
RESULTS: Most alumni were Brazilian (96.6%) and physicians (90.7%), predominantly located in the Southeast region (66.9%). Doctoral training was completed by 80.5% of participants. Alumni with current institutional ties to Universidade Federal de São Paulo or Hospital São Paulo demonstrated significantly higher citation counts and h-index values. No significant correlation was observed between time since graduation and citation count (p=0.185). Alumni engaged in academic roles or with postdoctoral training showed greater scientific productivity.
CONCLUSIONS: The findings highlight the strong academic performance and professional integration of alumni from Universidade Federal de São Paulo, particularly within public institutions and the Southeast region of Brazil. Doctoral training and institutional affiliation were associated with higher scientific productivity. Alumni tracking provides valuable insights into the impact of postgraduate programs and informs strategic planning and development.

Keywords: Factual databases; Program evaluation; Medical education; Graduate education; Ophthalmology; Linear models