ABSTRACT

PURPOSE: To translate and validate a questionnaire that evaluates the theoretical knowledge regarding fundus examination.
METHODS: A 60-item multiple-choice English questionnaire that investigates various aspects of knowledge regarding fundus examination was translated into Portuguese. The process involved translation, back-translation, and evaluation by an expert committee. The resulting questionnaire was applied to final-year medical students and ophthalmology residents. Each included subject answered the questionnaire twice, with an interval of one week between each application. Internal consistency, test-retest reliability, inter-rater reliability, and percentage agreement were calculated.
RESULTS: Thirty participants were included (25 medical students and 5 ophthalmology residents). The pass-fail cutoff was calculated at 46, the theoretical false positives were 8.7% and the theoretical false negatives were 2.8%. The observed false positive and false negative rates were 0%. Among the 60 items, test-retest reliability was strong in 17 items, which one had a negative correlation, moderate in 14 items, which one had a negative correlation, and weak in 29 items; inter-rater reliability of 34 items was under 0.4, 17 items were between 0.4 and 0.6, and 8 items were above 0.6. One item had a negative kappa. Among the percent agreement, 10 items were between 40%-60% agreement, 50 were above 60% agreement, and 18 were above 80%. Cronbach’s alpha was calculated as 0.674.
CONCLUSIONS: The translated questionnaire provided a standard instrument for future research and interventions to improve medical education in ophthalmology.

Keywords: Questionnaire and surveys; Translating; Fundus oculi

RESUMO

OBJETIVOS: Traduzir e validar para o português um questionário para avaliar o conhecimento teórico no exame de fundo de olho.
MÉTODOS: Um questionário de múltipla escolha de 60 questões em inglês, ao qual avalia diversos aspetos do conhecimento no exame de fundo de olho, foi traduzido para o português. O processo envolveu uma tradução, re-tradução e avaliação por um comitê de especialistas. O questionário resultante foi aplicado em estudantes de medicina do último ano e médicos-residentes em oftalmologia. Cada indivíduo respondeu o questionário duas vezes, com um intervalo de uma semana entre cada aplicação. A consistência interna, a confiança teste-reteste, a confiança entre avaliadores e a porcentagem de concordâncias foram calculadas.
RESULTADOS: Trinta participantes foram incluídos (25 estudantes de medicina e 5 residentes em oftalmologia). A nota de corte para aprovação-reprovação foi calculada em 46, sendo os falsos positivos teóricos de 8,7% e os falsos negativos teóricos 2,8%. No mesmo corte, os falsos positivos e falsos negativos observados foi de 0%. Dentre os 60 itens, a confiança teste-reteste foi forte em 17 itens, sendo que um apresentou uma correlação negativa, moderada em 14 itens, sendo que um apresentou uma correlação negativa, e fraca em 29 itens; a confiança inter-examinador foi inferior a 0,4 em 34 itens, entre 0,4 e 0,6 em 17 itens, e acima de 0,6 em 8 itens. Um dos itens apresentou um valor negativo. A porcentagem de concordância foi entre 40-60% em 10 itens, superior a 60% em 50 itens e acima de 80% em 18 itens. O alfa de Crombach foi calculado com 0,674.
CONCLUSÕES: O questionário traduzido propicia um instrumento padronizado para futuras pesquisas e intervenções às quais visem aprimorar o ensino em oftalmologia.

Descritores: Inquéritos e questionários; Tradução; Fundo de olho

INTRODUCTION

Blindness from reversible causes has been increasing globally. In 2020, 237 million people were estimated to have a moderate or severe visual impairment, while another 38 million were considered blind⁽¹⁾. Among irreversible causes of blindness, like diabetic retinopathy and glaucoma, an early diagnosis may slow down progression since treatment has been instituted to prevent a severe visual impairment. In this context, fundoscopy emerges as a fast, cheap, and effective strategy for tracking down eye pathologies.

Despite the importance of fundus examination, many doctors currently lack knowledge of the technique and/or self-confidence in performing the examination^(2-6). The International Council of Ophthalmology^(7,8) recommends knowledge of the basic use and handling of fundoscopy equipment and how to examine and identify the normal and pathological structures. In consonance, the Association of University Professors of Ophthalmology⁽⁹⁾ highlights the importance of fundoscopy in primary care and its various applications in prevention and health promotion.

Recently, a questionnaire to measure self-confidence in performing fundoscopy was translated and validated into Portuguese⁽⁶⁾. However, at the time, there is no available instruments in Portuguese to measure physicians’ and medical students’ knowledge regarding fundus examination. A questionnaire that evaluates several aspects of fundoscopy such as technique and normal and abnormal findings would help to standardize future research and to propose future interventions to improve medical education in ophthalmology. This study aimed to translate and validate a questionnaire into Portuguese to evaluate theoretical knowledge regarding fundus examination.

METHODS

This study was approved by the Ethics Committee of the University of Campinas and conducted in compliance with the Declaration of Helsinki. All procedures were fully explained and informed consent was obtained from all participants.

Translation, cross-cultural adaptation, and validation process

The questionnaire developed by Jørgensen et al.⁽¹⁰⁾ which was used as a reference, consists of 60 multiple-choice questions with 3 answers, of which only 1 is the correct answer. This questionnaire investigates the various aspects of knowledge in the fundoscopy examination, such as the details of the correct technique (items 1-6, 12, 13, 21-26, 43, 45, 56-58, 60), anatomical structures, and pathological findings (items 7-11, 14-20, 27-42, 44, 46-55, 59). Among the 60 questions, 18 were considered easy and 42 to be of moderate difficulty level.

The translation, cross-cultural adaptation, and validation process were followed as described in the literature^(11-13). The reference questionnaire was translated from English to Portuguese by two independent translators, both with high proficiency in English and one with an experience in the fundus examination and ophthalmology. Both the translations were then synthesized and evaluated by an expert committee, which evaluated the equivalence of the translation and made the necessary modifications. The back-translation from Portuguese to English was made by two independent translators with high proficiency in the English language. Both back translations were then compared to the original version of the questionnaire to evaluate the consistency of the translation process. Finally, the expert committee evaluated the final version of the questionnaire (available as Supplemental Material) to verify the semantic, idiomatic, and conceptual equivalence relative to the original questionnaire.

Test of the translated questionnaire

The resulting questionnaire was applied to final-year medical students and ophthalmology residents. Each subject answered the questionnaire twice, at an interval of 1 week between each application (test-retest). The participants who did not complete the questionnaire at both times were excluded. Internal consistency, test-retest reliability, inter-rater reliability, and percentage agreement were also calculated.

Statistical analysis

The statistical analysis was performed with the Statistical Package for Social Sciences-SPSS (IBM Corporation, Armon NY, USA, version 22.0). The normality of the mean score of the groups was calculated by Shapiro-Wilk test. The comparison was performed with Mann-Whitney U-test. P<0.05 was considered to indicate statistical significance.

The pass-fail cutoff was calculated with contrasting group methods as described by Jørgensen et al.⁽¹⁴⁾. Internal consistency, to evaluate the inter-correlation of the questionnaire items, was calculated by using coefficient alpha (Cronbach’s alpha-α), which varied from 0 to 1, while a value of >0.7 was considered adequate⁽¹⁵⁾.

Test-retest reliability was calculated by Pearson’s correlation (R), and inter-rater reliability was calculated by Cohen’s Kappa (κ), which varied from 0 to 1, and was classified as <0.2, poor agreement; 0.2-0.4, week agreement, 0.4-0.6, moderate agreement; 0.6-0.8, good agreement; >0.8 excellent agreement^(12,15). Percentage agreement was calculated by the number of agreements divided by the total number of answers⁽¹⁶⁾.

RESULTS

A total of 30 participants were included in the validation process, of which 25 were medical students and 5 were ophthalmology residents. The baseline characteristics are presented in table 1.

While the overall mean score was 39.83 ± 8.11 (66.38%), medical students’ mean score was 37.48 ± 6.6 (62.46%) and ophthalmology residents’ mean score was 51.6 ± 2.7 (86%) (p<0.0001). The pass-fail cutoff was calculated as 46, the theoretical false positives were 8.7%, and the theoretical false negatives were 2.8% (Figure 1). At this score, the observed rates of false positive and false negative were 0%.

Figure 1. Pass-fail (cutoff) score and theoretical false positives and false negatives.

Among the 60 items, test-retest reliability was strong in 17 items having negative correlation, moderate in 14 items having negative correlation and weak in 29 items. The inter-rater reliability was <0.4 in 34 items, between 0.4 and 0.6 in 17 items, and >0.6 in 8 items. Only one item had negative inter-rater reliability. Among the percent agreement, 10 items were between 40%-60% agreement, 50 items were above 60% agreement, and 18 items were above 80% agreement. Cronbach’s alpha was calculated as 0.674. The details of each item are presented in table 2.

DISCUSSION

The current study successfully translated a questionnaire into Portuguese that comprehensively evaluated the theoretical knowledge regarding fundus examination. The questionnaire developed by Jørgensen et al.⁽¹⁰⁾ recruited 30 participants for the validation process (20 medical students and 10 ophthalmology specialists). The mean score of the first group was 30.0 ± 4.3 and the mean score of the second group was 57.4 ± 1.6 (p<0.0001). The pass-fail cutoff was calculated as 49.7, which represented the point that no medical student passed the test (no false positive) and no ophthalmology resident failed the test (no false negative). Internal consistency was calculated as 0.95. The validation process revealed that there was a significant difference in the scores between the two groups. A lower pass-fail cutoff (46 vs 49.7) and a lower internal consistency (0.674 vs 0.95) was found in the current study.

Further, reliability is a method to estimate the random errors in an assessment. One way to evaluate reliability is through internal consistency, represented by Cronbach’s alpha. The level of accepted reliability for an instrument depends on its application. For instruments that assess subjects with major consequences for the society such as obtaining a medical license, a value >0.9 is desired; for assessments with moderate consequences such as end-of-year or end-of-course examinations, a minimum of 0.8 is acceptable, and a value <0.7 is acceptable when the instrument aims to assess classroom proficiency⁽¹⁷⁾. In the current study, the reliability was 0.674, close to the minimum 0.7 acceptable for the aim of the questionnaire, which was to evaluate theoretical proficiency in one specific item of the physical examination.

Another method to verify the validity of training and instruments is the proposed contrasting groups’ method⁽¹⁴⁾ which consists of a comparison between two groups with significantly different expertise in the subject, a group of novices and a group of experts. The calculation of the cutoff point and the theoretical false positives and false negatives englobes the mean and standard deviation of the score of each group, which are considered as a normal distribution. This reduces the main issue in validation studies, which is the small sample size, mainly due to the reduced availability of experts. The cutoff point is considered the intersection between the curves of both groups, as shown in figure 1, whereas the theoretical false positive rate is calculated based on the novices who scored higher than the cutoff point, and the false negative rate as the experts who scored lower than the pass-fail point⁽¹⁴⁾. The differences between observed and theoretical false positives and false negatives are due to the assumption of the normal distribution of the values, which may be considered a limitation of the study. The small samples are not always normally distributed, and so the outliers may have a strong influence on the calculations⁽¹⁴⁾. In our casuistic, low rates of theoretical false positives and false negatives were observed, which reflected a good representation of novices with low proficiency, and experts with high proficiency. Both Cronbach’s alpha and contrasting groups are valid methods to calculate reliability⁽¹⁷⁾.

Despite the successful translation and cross-cultural adaptation to Portuguese, the validation process had a few issues. One method of measuring inter-rater reliability was percentage agreement, which was calculated by the number of agreements divided by the total number of answers⁽¹⁶⁾. However, this method did not take into account the chance of agreement, which is a random agreement caused by guessing that resulted in a false agreement. Cohen’s kappa^(18,19) was introduced to calculate the agreement including this chance, which varied between -1 and +1, and a value closer to +1 represented a perfect agreement while a value closer to 0 represented no agreement. A negative value represented a disagreement between the observers. Kappa’s correlation also had a few limitations. Despite the minimally accepted values of 0.40-0.59, it represented only a 15%-35% reliability of the data⁽¹⁶⁾, which may be critical values depending on the subject. In our study, we obtained a <0.4 correlation in 35 items and a negative correlation in one item, which represented a limitation of the validation process. However, among the remaining 24 items, a moderate to strong kappa was calculated, which indicated a good agreement and a reliable validation process.

The current study had a few limitations. The population included in the validation process was composed mainly of medical students with moderate to low levels of proficiency in fundoscopy^(2-6) which could explain the lower level of agreement by a higher chance of guessing. Also, the extension of the questionnaire composed of 60-items would have influenced the concordance of the answers with a higher rate of concordance in the initial questions and a lower level in the following questions. A shorter questionnaire or a shorter version of the same instrument evaluating the same points would have improved the validation process.

In conclusion, the experts translated and validated the English questionnaire into Portuguese that comprehensively evaluated the knowledge regarding fundoscopy among medical students and ophthalmology residents. Further research is needed to measure the theoretical proficiency regarding fundus examination, which would help to propose effective interventions to improve medical education in ophthalmology.

REFERENCES

1. Flaxman SR, Bourne RRA, Resnikoff S, Ackland P, Braithwaite T, Cicinelli MV, Das A, Jonas JB, Keeffe J, Kempen JH, Leasher J, Limburg H, Naidoo K, Pesudovs K, Silvester A, Stevens GA, Tahhan N, Wong TY, Taylor HR; Vision Loss Expert Group of the Global Burden of Disease Study. Global causes of blindness and distance vision impairment 1990-2020: a systematic review and meta-analysis. Lancet Glob Health. 2017;5(12):e1221-e1234.

2. Wu EH, Fagan MJ, Reinert SE, Diaz JA. Self-confidence in and perceived utility of the physical examination: A comparison of medical students, residents, and faculty internists. J Gen Intern Med. 2007;22(12):1725-30.

3. Cordeiro MF, Jolly BC, Dacre JE. The effect of formal instruction in ophthalmoscopy on medical student performance. Med Teach. 1993;15(4):321-5.

4. Kelly LP, Garza PS, Bruce BB, Graubart EB, Newman NJ. Teaching Ophthalmoscopy to Medical Students (the TOTeMS Study). Am J Ophthalmol. 2013;156(5):1056-61.

5. Mackay DD, Garza PS, Bruce BB, Bidot S, Graubart EB, Newman NJ, et al. Teaching ophthalmoscopy to medical students (TOTeMS) II: A one-year retention study. Am J Ophthalmol. 2014;157(3):747-8.

6. Ayub G, Souza RB, de Albuquerque AM, de Vasconcellos JP. Comparison of conventional and wide field direct ophthalmoscopy on medical students’ self-confidence for fundus examination: a 1-year follow-up. BMC Med Educ. 2021;21:1-9.

7. Straatsma BR, Coscas GJ, Naumann GO, Spivey BE, Taylor HR, et al. International ophthalmology strategic plan to preserve and restore vision-vision for the future. Am J Ophthalmol. 2001;132(3):403-4.

8. International Task Force on Opthalmic Education of Medical Students; International Council of Opthalmology. Principles and guidelines of a curriculum for ophthalmic education of medical students. Klin Monbl Augenheilkd. 2006;223 Suppl 5:S1-19.

9. Stern GA. Teaching Ophthalmology to Primary Care Physicians. Arch Ophthalmol. 1995;113:722-4.

10. Jørgensen M, Savran MM, Christakopoulos C, Bek T, Grauslund J, Toft PB, et al. Development and validation of a multiple-choice questionnaire-based theoretical test in direct ophthalmoscopy. Acta Ophthalmol. 2019;97(7):700-6.

11. Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine (Phila Pa 1976). 2000;25(24):3186-91.

12. Gjersing L, Caplehorn JR, Clausen T. Cross-cultural adaptation of research instruments: Language, setting, time and statistical considerations. BMC Med Res Methodol. 2010;10. doi:10.1186/1471-2288-10-13

13. Sousa VD, Rojjanasrirat W. Translation, adaptation and validation of instruments or scales for use in cross-cultural health care research: A clear and user-friendly guideline. J Eval Clin Pract. 2011;17(2):268-74.

14. Jørgensen M, Konge L, Subhi Y. Contrasting groups’ standard setting for consequences analysis in validity studies: reporting considerations. Adv Simul. 2018;3(1):1-7.

15. Tsang S, Royse CF, Terkawi AS. Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine. Saudi J Anesth. 2017;11(5):80-9.

16. McHugh ML. Interrater reliability: the kappa statistic. Biochem Medica. 2012;22(3):276-82.

17. Downing SM. Reliability: On the reproducibility of assessment data. Med Educ. 2004;38(9):1006-12.

18. Cohen J. A Coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20(1):37-46.

19. McHugh ML. Interrater reliability: the kappa statistic. Biochem Medica. 2012;22(3):276.

Submitted for publication: January 11, 2022.
Accepted for publication: April 14, 2022.

Funding: This study received no specific financial support

Disclosure of potential conflicts of interest: None of the authors have any potential conflicts of interest to disclose

Approved by the following research ethics committee: Hospital de Clínicas da Universidade Estadual de Campinas (CAAE 46531321.5.0000.5404)