ABSTRACT

Purpose: To assess the microsurgery dexterity outcomes of two sequential training evaluations using virtual reality technology.
Methods: This was a multicenter cross-sectional study of all candidates who were accepted as first-year residents at one of six ophthalmology teaching institutions. Residents were subjected to two identical series of standardized, reproducible dexterity tests using virtual reality equipment (Eyesi^®): “sequence 1” and “sequence 2.” Each sequence consisted of five difficulty levels that were assessed using a proprietary scoring system. The data were tested for normality using the Shapiro-Wilk test. The differences between tests in sequences 1 and 2 were evaluated using the Wilcoxon signed-rank test.
Results: The data did not follow a normal distribution. There were improvements from sequence 1 in all the tests (all p values<0.05). The sum of all scores (total score) improved from sequence 1 (median= 152.50) to sequence 2 (median 256.00; p<0.001). There was no correlation between the delta sequence values and the average scores.
Conclusion: Two sequential training evaluations using virtual reality technology showed relevant improvement in quantifications of microsurgery dexterity. This information should be considered if virtual reality approaches are used for testing purposes, as previous experience may lead to improved test results.

Keywords: Motor skills; Virtual reality; Clinical competence; Ophthalmologic surgical procedures/education

RESUMO

Objetivo: Avaliar os resultados da destreza da microcirurgia de duas avaliações sequenciais de treinamento usando a tecnologia de realidade virtual.
Métodos: Estudo transversal multicêntrico em que todos os candidatos que foram aceitos como residentes de primeiro ano em uma de seis instituições de ensino de oftalmologia. Os residentes foram submetidos a duas séries idênticas de testes de destreza padronizados e reprodutíveis usando equipamento de realidade virtual (Eyesi^®): “sequência 1” e “sequência 2”. Cada sequência consistiu em 5 níveis de dificuldade que foram avaliados usando um sistema de pontuação proprietário. Os dados foram testados quanto à normalidade utilizando o teste de Shapiro-Wilk. As diferenças entre os testes nas sequências 1 e 2 foram avaliadas com o teste de Wilcoxon signed-rank.
Resultados: Os dados não seguiram uma distribuição normal. Houve melhora da sequência 1 em todos os testes (todos os valores de p<0,05). A soma de todas as pontuações (pontuação total) melhorou da sequência 1 (mediana= 152,50) para a sequência 2 (mediana= 256,00; p<0.001). Não houve correlação entre os valores da sequência delta e as pontuações médias.
Conclusão: Duas avaliações sequenciais de treinamento utilizando a tecnologia de realidade virtual mostraram melhora relevante nas quantificações da destreza da microcirurgia. Essas informações devem ser consideradas se abordagens de realidade virtual forem utilizadas para fins de teste, pois a experiência prévia pode levar a melhores resultados.

Descritores: Destreza motora; Realidade virtual; Competência clínica; Procedimentos cirúrgicos oftalmológicos/educação

INTRODUCTION

In recent years, legal and ethical concerns regarding the use of patients for training purposes have led to the development of alternative approaches for the surgical learning process. The surgical learning curve is known to be associated with increased complication rates, worse patient outcomes, increased operating room time, and increased costs^(1-7). Therefore, the well-known saying often heard in a number of teaching institutions, “see one, do one, teach one,” is increasingly being considered as unacceptable⁽⁸⁾.

Practice with animals has been recommended for the training of ophthalmic surgeons before they begin actual surgical procedures on human patients⁽⁹⁾. For the training of phacoemulsification procedures, slaughterhouse porcine eyes are frequently used; however, despite their anatomical resemblance to the human eye, they do not provide a true simulation of the procedure in human patients⁽¹⁰⁾. Notably, virtual reality (VR) technology has evolved rapidly in recent years and has become widespread in the entertainment industry and in teaching environments⁽¹¹⁾. Simulators have been used in situations where the possibility of errors during training must be minimal due to high human or economic costs. Therefore, they have been increasingly used for training involving invasive medical procedures^(12,13).

Currently, VR simulators can be used for the training of retrobulbar injections, phacoemulsification, and vitreoretinal surgeries⁽¹¹⁾. Among the available models, EyeSi^® (Vrmagic^®, Mannheim, Germany) has been shown to improve surgical skills⁽¹⁴⁾ and shorten the learning curve⁽¹⁵⁾. In addition to the use of VR simulators as training models, the standardized and controlled scenarios in these simulators may be used for the objective assessment of manual dexterity and surgical competence; to the best of our knowledge, there have been no studies in which VR has been used to evaluate manual dexterity as a potential factor for resident selection.

EyeSi^® consists of a mannequin head with an artificial eye, as well as a digital operating microscope and a screen connected to a personal computer. When specific probes (included with the system) are inserted inside the artificial eye, the microscope shows them as instruments to simulate distinct tasks in a VR environment. These tasks constitute a sequence of dexterity movements that increase in difficulty as surgeons complete each phase. The tasks begin with simple single-handed exercises that are not directly related to surgeries; when a threshold score is reached, the next level becomes available. With increasing difficulty, the user must use both hands and both feet to complete the tasks that become progressively similar to those in actual surgeries. The process resembles a video game, and a score is provided for each level.

Although VR is frequently used for training in Europe and the USA, there are minimal data regarding its use for the evaluation of dexterity. We presumed that VR could be used to evaluate manual dexterity in novice surgeons and that it could help to select best-fit candidates for residency and fellowship positions. However, we suspected that, owing to a fast learning curve, previous experience with VR may interfere with its use as a testing device. As such, we assessed the results of two sets of EyeSi^® dexterity tests of doctors selected as first-year residents in different ophthalmology institutions in Brazil, all of whom had no previous experience with ophthalmologic surgery or VR surgical simulators.

METHODS

This cross-sectional study was designed and conducted in accordance with the guideline for good clinical practice and was approved by the local ethics committee (UNIFESP no. 0111/2017). The study included all candidates who were accepted as first-year residents at one of six ophthalmology teaching institutions: Escola Paulista de Medicina, Fundação Banco de Olhos de Goiás, Hospital Oftalmológico de Brasília, Santa Casa de São Paulo, Universidade Estadual de São Paulo, and Universidade de São Paulo. None of the residents had started practical training or had previous experience with any ophthalmology VR simulator. Candidates who did not complete all the required tests were excluded from the study.

After enrollment and the provision of written informed consent, candidates were subjected to two series of standardized, reproducible dexterity tests. During these tests, residents manipulated instruments for which tips were inserted into the anterior chamber of an artificial eye.

The sequence of tests consisted of five levels of increasing difficulty: navigation (NAV), antitremor #1 (AT1), antitremor #6 (AT6), forceps (FOR), and bimanual (BM) (Figure 1). The ability to differentiate among the levels of ability, known as construct validity, has been previously demonstrated for NAV, AT1, AT6, and FOR^(16,17).

Figure 1. Visual representation of EyeSi^® dexterity tests performed in this study. A) Navigation dexterity test in which the tip of the indicator is used to point to spheres until they change colors. B) and C) Antitremor dexterity tests in which an indicator is used to follow the empty region (straight line (#1) and spiral area (#6), respectively). D) Forceps dexterity test in which a VR forceps is used to move virtual squares from outside to inside the central circle. E) Bimanual dexterity test in which two indicators are used to simultaneously touch spheres until they change colors.

In NAV, the tip of the instrument must be sequentially inserted into various adjacent small spheres and held for several seconds until the spheres change color. In AT, the user must “draw” a determined pattern (a horizontal line for AT1 and a spiral line for AT6) at constant speed. In FOR, the user must pinch small squares and bring them to a basket in the center of the eye. In BM, the user must place probes at the ends of a cylinder and hold them steadily for several seconds; if the movement is imprecise, the cylinder may rotate in different axes so that the task becomes more difficult to accomplish⁽¹⁸⁾.

Eyesi^® uses a complex proprietary scoring system that gives a total score between 0 and 100 points to each task. If all goals are reached within a task, 100 positive points are given. To perform some tasks, several subgoals must be reached; for example, to draw a spiral in AT6, the user must maintain consistent speed and remain within the intended pattern. Certain movements such as contact with the endothelium or anterior capsule, as well as the use of an open forceps during entry to the eye, are considered mistakes; these lead to a penalty, indicated by negative points. Excessive or unnecessary movements are registered by an odometer and penalized. If the sum is ≤0, the task is scored as 0 points, regardless of whether the user has reached all goals within that task⁽¹⁸⁾.

Before beginning the tests, candidates received instructions on microscope adjustment, positioning, and tasks that they were expected to perform. They also watched instructional videos demonstrating expectations within these tasks. The first sequence of tests was used as an introduction to the platform (sequence 1), because no candidates had previous experience with the simulator or ophthalmic surgeries. Subsequently, on the same day, a second sequence of tests was performed with the same tasks, in order to determine whether the candidates exhibited increased performance (sequence 2). Differences between the personal scores of each test within each sequence (delta sequence) were determined.

The names of the institutions were not attached to the results. A total of 48 medical residents (25 women, 52.1%) were recruited for this study: 8 from Institution A, 6 from Institution B, 7 from Institution C, 6 from Institution D, 7 from Institution E, and 14 from Institution F. The data were tested for normality using the Shapiro-Wilk test. The differences between tests in sequences 1 and 2 were evaluated using the Wilcoxon signed-rank test. Stata/IC 15 software was used for all statistical analysis.

RESULTS

The data did not follow a normal distribution. The results of the scores of different tests in sequences 1 and 2 are shown in table 1. Differences were observed between the tests in sequences 1 and 2 (p≤0.0001); there were improvements from sequence 1 in all the tests (NAV, p<0.001; AT1, p<0.001; AT6, p=0.005; FOR, p<0.001; BM, p=0.042).

The sum of all scores (total score) improved by 103.50 points from sequence 1 (median=152.50) to sequence 2 (median=256.00) (p<0.001), as shown in figure 2. There was no correlation between the delta sequence values and the average scores (Figure 3). Statistical differences between the institutions were detected in sequence 2 (p=0.025).

Figure 2. Scatterplot of Total Score results for dexterity tests (EyeSi^® simulator) at Sequence 1 and Sequence 2 in subjects that were accepted to be first-year residents in Ophthalmology. Sequence 1 was an introduction to the virtual platform, subsequently, in the same day; Sequence 2 of tests was performed to evaluate a possible performance increase. Each symbol represents one institution.

Figure 3. Bland-Altman plot of total score results for dexterity tests (EyeSi^® simulator) of first-year residents. Sequence 1 was an introduction to the virtual platform. Later on the same day, sequence 2 was performed to determine whether a performance increase was present. The delta sequence represents the improvement achieved after the initial sequence of tests. Each symbol represents one institution.

DISCUSSION

Previous studies have demonstrated a high variability in motor tasks among novice ophthalmic surgeons in EyeSi^®, wet-lab environments, and actual cataract surgeries⁽¹⁰⁾. Our study demonstrated differences between the two sequences of tasks performed on the same day, which indicated that a user with minimal experience could perform better than a user who had no prior contact with a VR simulator. Thus, previous experience may represent a bias in terms of the use of VR technology for testing purposes. To minimize this interference, we propose that a practice simulator round is provided before assessment; alternatively, we propose that tests are repeated, and possible performance increases are evaluated.

It has been previously suggested that inconsistency between tests is an indication that there are obstacles in predicting who might achieve the best surgical results⁽¹⁹⁾. A good score in the first attempt may not be fully representative of a trainee’s actual skills, because the trainee may be incapable of maintaining consistent performance. In contrast, a good candidate may fail in the initial attempt and then show rapid improvement with training.

Nevertheless, there is evidence of variations in the learning curves among novice general surgeons. A study of performance proficiency in a laparoscopy simulator showed four learning patterns⁽²⁰⁾. Group 1 demonstrated surgical proficiency at the beginning of the study; group 2 achieved proficiency after some training; group 3 showed an improvement with training, but did not reach the proficiency threshold; and group 4 underperformed at the beginning and did not improve with training. Although it was impossible to differentiate group 2 from group 3 in the initial tests, there were clear differences between groups 1 and 4; these may represent an opportunity to identify the most capable candidates for residency programs⁽²⁰⁾.

Furthermore, there is increasing evidence that not all trainees achieve proficiency by the end of the training period during general surgery residency. Some studies suggest that between 5% and 20% of residents do not reach competence, regardless of the use of simulated tasks and continued practice^(20-23). Although there have not been similar studies in ophthalmology, we presume that the rates of competency are similar. These estimates are alarming, considering the high costs related to medical training and the general expectation that a licensed surgeon is capable of performing procedures with competence at the end of the training period.

Although manual dexterity is not currently part of the selection process and it remains controversial as to whether such criteria should be used to select candidates, a test that can differentiate dexterity among novice surgeons may enable significant reductions in medical training costs. The cost of operating rooms in the USA is approximately 900-1200 USD per hour⁽²⁴⁾, and it is estimated that resident training may represent a cumulative annual cost of 53 million USD solely in terms of the added operating time⁽¹⁸⁾. If nontangible costs related to surgical complications are considered, this value is likely to be much higher. It is estimated that medical errors may represent costs of more than 1 billion USD per year in the USA⁽²⁵⁾.

In addition to the economic cost, there is an important social impact associated with the learning curve. It is well known that novice ophthalmic surgeons exhibit higher complication indexes with worse visual outcomes^(1,6); this has direct and indirect social and economic consequences for patients and their families⁽²⁵⁾. Studies have shown psychological consequences for novice physicians who are typically expected to be “error free”; many experience significant emotional distress, anxiety, guilt, or burnout syndrome during residency training^(26,27). A test that could determine the learning capabilities of residency candidates may also help to develop new learning processes that could be used for candidates with different learning profiles. In particular, it is not necessary for all residents to undergo the same surgical training; some residents may need more wet-lab experience or a greater number of supervised surgeries.

This study has some limitations. Notably, it was a cross-sectional study; therefore, we did not evaluate whether the scores were associated with differences in the learning curves among candidates. Future studies should assess whether manual dexterity is associated with the rates of surgical complications among specialties during medical residency. If such an association is demonstrated, dexterity tests may be useful as a selection tool for ophthalmology residency or eminent surgical fellowships, and our results may be used as a reference for the establishment of such tests.

REFERENCES

1. Randleman JB, Wolfe JD, Woodward M, Lynn MJ, Cherwek DH, Srivastava SK. The resident surgeon phacoemulsification learning curve. Arch Ophthalmol. 2007;125(9):1215-9.

2. Lee JS, Hou CH, Yang ML, Kuo JZ, Lin KK. A different approach to assess resident phacoemulsification learning curve: analysis of both completion and complication rates. Eye (Lond). 2009;23(3):683-7.

3. Bhagat N, Nissirios N, Potdevin L, Chung J, Lama P, Zarbin MA, et al. Complications in resident-performed phacoemulsification cataract surgery at New Jersey Medical School. Br J Ophthalmol. 2007;91(10):1315-7.

4. Narendran N, Jaycock P, Johnston RL, Taylor H, Adams M, Tole DM, et al. The Cataract National Dataset electronic multicentre audit of 55,567 operations: risk stratification for posterior capsule rupture and vitreous loss. Eye (Lond). 2009;23(1):31-7.

5. Carricondo PC, Fortes AC, Mourão PC, Hajnal M, Jose NK. Senior resident phacoemulsification learning curve (corrected from cure). Arq Bras Oftalmol. 2010;73(1):66-9.

6. Hashemi H, Mohammadpour M, Jabbarvand M, Nezamdoost Z, Ghadimi H. Incidence of and risk factors for vitreous loss in resident-performed phacoemulsification surgery. J Cataract Refract Surg. 2013;39(9):1377-82.

7. Hosler MR, Scott IU, Kunselman AR, Wolford KR, Oltra EZ, Murray WB. Impact of resident participation in cataract surgery on operative time and cost. Ophthalmology. 2012;119(1):95-8.

8. Kotsis SV, Chung KC. Application of the “see one, do one, teach one” concept in surgical training. Plast Reconstr Surg. 2013; 131(5):1194-201.

9. International Council of Ophthalmology (ICO). ICO Residency Curriculum [Internet]. 2nd ed. San Francisco, CA: ICO; 2016. [cited 2019 Aug 28]. Available from: http://www.icoph.org/refocusing_education/curricula.html

10. Thomsen AS, Subhi Y, Kiilgaard JF, la Cour M, Konge L. Update on simulation-based surgical training and assessment in ophthalmology: a systematic review. Ophthalmology. 2015;122(6):1111-1130.

11. Saleh GM, Lamparter J, Sullivan PM, O’Sullivan F, Hussain B, Athanasiadis I, et al. The international forum of ophthalmic simulation: developing a virtual reality training curriculum for ophthalmology. Br J Ophthalmol. 2013;97(6):789-92.

12. Elneel FH, Carter F, Tang B, Cuschieri A. Extent of innate dexterity and ambidexterity across handedness and gender: implications for training in laparoscopic surgery. Surg Endosc. 2008;22(1):31-7.

13. Van Herzeele I, O’Donoghue KG, Aggarwal R, Vermassen F, Darzi A, Cheshire NJ. Visuospatial and psychomotor aptitude predicts endovascular performance of inexperienced individuals on a virtual reality simulator. J Vasc Surg. 2010;51(4):1035-42.

14. Feudner EM, Engel C, Neuhann IM, Petermeier K, Bartz-Schmidt KU, Szurman P. Virtual reality training improves wet-lab performance of capsulorhexis: results of a randomized, controlled study. Graefes Arch Clin Exp Ophthalmol. 2009;247(7):955-63.

15. Thomsen AS, Bach-Holm D, Kjærbo H, Hjøgaard-Olsen K, Subhi Y, Saleh GM, et al. Operating room performance improves after proficiency-based virtual reality cataract surgery training. Ophthalmology. 2017;124(4):524-31.

16. Selvander M, Åsman P. Cataract surgeons outperform medical students in Eyesi virtual reality cataract surgery: evidence for construct validity. Acta Ophthalmol. 2013;91(5):469-74.

17. Mahr MA, Hodge DO. Construct validity of anterior segment anti- -tremor and forceps surgical simulator training modules: attending versus resident surgeon performance. J Cataract Refract Surg. 2008; 34(6):980-5.

18. VRMagic. EyeSi Scoring System Manual. https://www.vrmagic.com/ fileadmin/downloads/simulators/Manuals/surgical/3.4/Eyesi_Surgical_UG_v4_3_190320_EN_WEB.pdf.

19. Saleh GM, Theodoraki K, Gillan S, Sullivan P, O’Sullivan F, Hussain B, Bunce C, Athanasiadis I. The development of a virtual reality training programme for ophthalmology: repeatability and reproducibility (part of the International Forum for Ophthalmic Simulation Studies). Eye (Lond). 2013;27(11):1269-74.

20. Grantcharov TP, Funch-Jensen P. Can everyone achieve proficiency with the laparoscopic technique? Learning curve patterns in technical skills acquisition. Am J Surg. 2009;197(4):447-9.

21. Mattar SG, Alseidi AA, Jones DB, Jeyarajah DR, Swanstrom LL, Aye RW, et al. General surgery residency inadequately prepares trainees for fellowship: results of a survey of fellowship program directors. Ann Surg. 2013;258(3):440-9.

22. Schijven MP, Jakimowicz J. The learning curve on the Xitact LS 500 laparoscopy simulator: profiles of performance. Surg Endosc. 2004;18(1):121-7.

23. Sadideen H, Alvand A, Saadeddin M, Kneebone R. Surgical experts: born or made? Int J Surg. 2013;11(9):773-8.

24. Macario A. What does one minute of operating room time cost? J Clin Anesth. 2010;22(4):233-6.

25. Shreve J, Van Den Bos J, Gray T, Halford M, Rustagi K, Ziemkiewicz F. The economic measurement of medical errors. USA: Society of Actuaries; 2010.

26. Stangierski A, Warmuz-Stangierska I, Ruchała M, Zdanowska J, Głowacka MD, Sowiński J, et al. Medical errors - not only patients’ problem. Arch Med Sci. 2012;8(3):569-74.

27. Waterman AD, Garbutt J, Hazel E, Dunagan WC, Levinson W, Fraser VJ, et al. The emotional impact of medical errors on practicing physicians in the United States and Canada. Jt Comm J Qual Patient Saf. 2007;33(8):467-76.

Approved by the following research ethics committee: UNIFESP (# 0111/2017).

Funding: This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.

Disclosure of potential conflicts of interest: Dr. Vieira reports grants from Coordenação de Aperfeiçoamento de Pessoal de Nível Superior-Brasil (CAPES) during the conduct of the study. Dr. Sakaya has nothing to disclose. Dr. Soares has nothing to disclose. Dr. Chamon reports grants and personal fees from Johnson&Johnson, and another from Haag Streit outside the submitted work.