ABSTRACT

PURPOSE: The emergency medical service is a fundamental part of healthcare, albeit crowded emergency rooms lead to delayed and low-quality assistance in actual urgent cases. Machine-learning algorithms can provide a smart and effective estimation of emergency patients’ volume, which was previously restricted to artificial intelligence (AI) experts in coding and computer science but is now feasible by anyone without any coding experience through auto machine learning. This study aimed to create a machine-learning model designed by an ophthalmologist without any coding experience using AutoML to predict the influx in the emergency department and trauma cases.
METHODS: A dataset of 356,611 visits at Hospital da Universidade Federal de São Paulo from January 01, 2014 to December 31, 2019 was included in the model training, which included visits/day and the international classification disease code. The training and prediction were made with the Amazon Forecast by 2 ophthalmologists with no prior coding experience.
RESULTS: The forecast period predicted a mean emergency patient volume of 216.27/day in p90, 180.75/day in p50, and 140.35/day in p10, and a mean of 7.42 trauma cases/day in p90, 3.99/day in p50, and 0.56/day in p10. In January of 2020, there were a total of 6,604 patient visits and a mean of 206.37 patients/day, which is 13.5% less than the p50 prediction. This period involved a total of 199 trauma cases and a mean of 6.21 cases/day, which is 55.77% more traumas than that by the p50 prediction.
CONCLUSIONS: The development of models was previously restricted to data scientists’ experts in coding and computer science, but transfer learning autoML has enabled AI development by any person with no code experience mandatory. This study model showed a close value to the actual 2020 January visits, and the only factors that may have influenced the results between the two approaches are holidays and dataset size. This is the first study to apply AutoML in hospital visits forecast, showing a close prediction of the actual hospital influx.

Keywords: Machine learning; Emergency services, hospital; Eye injuries; Models, statistical; Algorithms

RESUMO

OBJETIVO: Esse estudo tem como objetivo criar um modelo de Machine Learning por um oftalmologista sem experiência em programação utilizando auto Machine Learning predizendo influxo de pacientes em serviço de emergência e casos de trauma.
MÉTODOS: Um dataset de 366,610 visitas em Hospital Universitário da Universidade Federal de São Paulo de 01 de janeiro de 2014 até 31 de dezembro de 2019 foi incluído no treinamento do modelo, incluindo visitas/dia e código internacional de doenças. O treinamento e predição foram realizados com o Amazon Forecast por dois oftalmologistas sem experiência com programação.
RESULTADOS: O período de previsão estimou um volume de 206,37 pacientes/dia em p90, 180,75 em p50, 140,35 em p10 e média de 7,42 casos de trauma/dia em p90, 3,99 em p50 e 0,56 em p10. Janeiro de 2020 teve um total de 6.604 pacientes e média de 206,37 pacientes/dia, 13,5% menos do que a predição em p50. O período teve um total de 199 casos de trauma e média de 6,21 casos/dia, 55,77% mais casos do que a predição em p50.
CONCLUSÃO: O desenvolvimento de modelos era restrito a cientistas de dados com experiencia em programação, porém a transferência de ensino com a tecnologia de auto Machine Learning permite o desenvolvimento de algoritmos por qualquer pessoa sem experiencia em programação. Esse estudo mostra um modelo com valores preditos próximos ao que ocorreram em janeiro de 2020. Fatores que podem ter influenciados no resultado foram feriados e tamanho do banco de dados. Esse é o primeiro estudo que aplicada auto Machine Learning em predição de visitas hospitalares com resultados próximos aos que ocorreram.

Descritores: Aprendizado de máquina; Serviço hospitalar de emergência; Traumatismos oculares; Modelos estatísticos; Algoritmos

INTRODUCTION

The ophthalmological emergency department is a fundamental part of medical care, with the facility of immediate medical consultation in severe and urgent cases, such as ocular trauma, retinal detachment, and ocular infection^(1,2).

In several countries, there is a lack of investment in disease prevention, which has led to an increase in the demand for immediate treatment by a large portion of the population⁽³⁾ with crowded emergency rooms (ERs)^(1,4,5) and delayed and low-quality assistance to actual urgent cases. Nonurgent visits have been reported between 8% and 62% of the total visits, especially at self-referral service centers⁽⁶⁾.

A possible solution to ER overcrowding is a quicker and more efficient patient movement through the emergency department⁽³⁾. Machine-learning algorithms can provide a smart and effective estimation of emergency patient volume for staff planning^(7,8), but the development of predictive models in healthcare had, so far, been exclusive to artificial intelligence (AI) experts with coding and computer science knowledge.

Transfer learning, where previously designed models can be adapted for training a new task by companies such as Amazon, Google, and Microsoft, has democratized access to AI and enabled any individual to develop AI models⁽⁹⁾.

Past studies from Faes and Antaki proved the feasibility of automated machine learning (autoML) in ophthalmology models built by physicians without any coding experience^(9,10) for the prediction of patient sex through retinal fundus photos⁽¹¹⁾, cardiovascular risk⁽¹²⁾, and visual acuity outcomes in neovascular age-related macular degeneration⁽¹³⁾.

This study aimed to create a machine-learning model designed by two ophthalmologists without any coding experience using the AutoML platform for predicting ophthalmological emergency department appointments and trauma cases.

METHODS

This study included data from emergency department visits at the Hospital da Universidade Federal de São Paulo, an academic tertiary hospital in Brazil, from January 2014 to December 2019. This study was approved by the institutional review board and followed the Helsinki Tenants.

The data was obtained from the hospital database, which extracted information from the patient’s charts. The algorithm training was performed by 2 ophthalmologists-LFN and LZR-with no previous coding and machine-learning experience after a period of self-study and reading the Amazon Web Services (AWS) Forecast documentation guide.

The outcome is a prediction of a monthly patient volume and trauma cases based on the ER’s previous 5 years’ visits and a comparison to the actual ER influx during the forecast period.

Study cohort

We included all patients who were evaluated at the emergency department of Sao Paulo Hospital from January 01, 2014 to December 31, 2019. A total of 356,611 visits were included.

Every patient classified with the international classification of diseases (ICD) code of “S” - injuries - was included in the group of trauma patients.

Dataset preparation and features collected.

The dataset was collected from relevant electronic medical records with nonidentified information. During the analysis, daily influx and trauma patients were included, classified according to the ICD classification, and imported to the Amazon Simple Storage Service.

In both the datasets, the columns were defined as item_id, timestamp, and target_value, according to the AWS Forecast guidelines, and the model was applied in predicting target_vaue according to item_id.

Automated machine learning

The training and prediction were made with the Amazon Forecast (https://aws.amazon.com/pt/forecast/), a fully managed service that utilizes statistical and machine-learning algorithms to deliver highly accurate time-series forecasts.

In this forecast, the Amazon Quantile Regression Amazon Convolutional Neural Networks-a causal Convolutional Neural Network-was applied.

The 5 years of visits to the ER were applied to predictor training.

Validation process

The Amazon Forecast performs backtesting by splitting the dataset into the training and testing datasets and then providing the following metrics to evaluate the model: root mean square error, weighted quantile loss, mean absolute percentage error, mean absolute scaled error, and weighted absolute percentage error⁽¹⁴⁾.

In this model, we considered the average weighted quantile loss, weighted absolute percentage error, and root means square error for model evaluation.

Next, we compared the model forecasting with the actual patient’s income in January 2020.

Statistical Analysis

Continuous variables were presented as the mean, median, range, and standard deviation. Categorical variables were presented as counts and percentages. We defined the statistical significance as p<0.05. All statistical tests and descriptions were performed using the JASP (JASP Team 2020-version 0.14.1).

RESULTS

The study dataset included 356,611 emergency visits, with 2,191 days and a mean of 162.76 daily visits (SD 52.05), mean age of 40.55 years (SD 20.45 years), and included 7,027 trauma cases with a mean of 3.20 cases/day (standard deviation of 2.40). In 277,020 (77.68%) visits, the ICD code was filled in the medical records.

During the forecast period, the mean prediction rate of emergency patients was 216.27/day in p90, 180.75/day in p50, and 140.35/day in p10.

The accuracy metrics in daily volume prediction presented an average weighted quantile loss of 0.09, weighted absolute percentage error of 0.12, and root means a square error of 31.61.

During the forecast period, the mean predicted emergency patients were 7.42 trauma cases/day in p90, 3.99/day in p50, and 0.56/day in p10.

The accuracy metrics in daily volume prediction presented an average weighted quantile loss of 0.288, weighted absolute percentage error of 0.48, and root means a square error of 2.61.

In the January 2020, there were a total of 6,604 patient visits and a mean of 206.37 patients/day, which is 13.5% less than that by the p50 prediction. This period included a total of 199 trauma cases and a mean of 6.21 cases/day, which is 55.77% more traumas than that by the p50 prediction. The daily distribution is illustrated in table 1 and the predictions in figure 1.

Figure 1. Cases of December 01 to 31, 2019, and prediction of January 01 to 30. (A) Daily emergency predicted cases. (B) Daily traumas predicted cases.

DISCUSSION

The emergency medical department is a fundamental part of health care, but crowded ERs led to delayed and low-quality assistance in actual urgent cases^(1,4,5).

AI algorithms can provide intelligent and effective forecasting of emergency department influx to better plan the allocation of staff and hospital resources^(7,8). The development of AI models was previously restricted to data scientists who are experts in coding and computer science, but the transfer learning autoML has enabled AI development by any person without any coding experience.

Auto machine-learning modeling in image analysis is accessible to everyone with no coding experience, with feasibility proven by previous studies⁽¹⁰⁾.

This is the first study to apply an autoML model for the prediction of the ER influx and traumas cases with values close to the actual values recorded in January 2020 visits and traumas cases. This is a reproducible study that enabled an AI forecast prediction by a physician with no coding experience.

This study has some limitations. We opted to perform prediction in a prepandemic period of COVID19 due to the sudden decrease in patient volume after the pandemic period. This event may have influenced the results of the holidays on January 1 (New year) and January 25 (Birthday of Sao Paulo), the period of school holidays, and the dataset size, with a small prevalence of ocular traumas.

Incomplete ICD in medical records remains a problem in data quality, and we hence did not include these patients in trauma analysis.

More studies are warranted to assess the performance of auto ML using a more extensive dataset as well as to attempt to forecast patient visits in distinct daytimes for better staff planning.

This forecast applied a single AutoML platform to conduct a comparative study between other platforms such as Google’s and Microsoft Azure.

In conclusion, this is the first study to apply AutoML in hospital visits’ forecast prediction by 2 ophthalmologists with no prior coding experience to give a prediction close to that actual hospital influx.

ACKNOWLEDGMENTS

We would like to thank the staff of Hospital São Paulo - Universidade Federal de São Paulo emergency sector for their contribution.

REFERENCES

1. Oktay C, Cete Y, Eray O, Pekdemir M, Gunerli A. Appropriateness of emergency department visits in a Turkish university hospital. Croat Med J. 2003;44(5):585-91.

2. Pierre Filho P de T, Gomes PR, Pierre, Pinheiro Neto FB. Profile of ocular emergencies in a tertiary hospital from Northeast of Brazil. Rev Bras Oftalmol [Internet]. 2010 [cited 2020 Jul 21];69(1):12-7. Available from: SciELO - Brasil - Profile of ocular emergencies in a tertiary hospital from Northeast of Brazil Profile of ocular emergencies in a tertiary hospital from Northeast of Brazil

3. Morley C, Unwin M, Peterson GM, Stankovich J, Kinsman L. Emergency department crowding: A systematic review of causes, consequences and solutions. PLoS One [Internet]. 2018[cited 2020 Nov 21];13(8):e0203316. Available from: Emergency department crowding: A systematic review of causes, consequences and solutions - PMC (nih.gov)

4. Alabbasi OM, Al-Barry M, Albasri RF, Khashim HF, Aloufi MM, Abdulaal MF, et al. Patterns of ophthalmic emergencies presenting to a referral hospital in Medina City, Saudi Arabia. Saudi J Ophthalmol [Internet] 2017[cited 2020 Oct 15];31(4):243-6. Available from: Patterns of ophthalmic emergencies presenting to a referral hospital in Medina City, Saudi Arabia - PMC (nih.gov)

5. Singman EL, Smith K, Mehta R, Boland MV, Srikumaran D, Frick K, et al. Cost and visit duration of same-day access at an Academic Ophthalmology Department vs Emergency Department. JAMA Ophthalmol. 2019;137(7):729-35. Comment in: JAMA Ophthalmol. 2019;137(7):736-7.

6. Sridhar J, Isom RF, Schiffman JC, Ajuria L, Huang LC, Gologorsky D, et al. Utilization of ophthalmology-specific emergency department services. Semin Ophthalmol. 2018;33(2):185-90.

7. Kadri F, Harrou F, Chaabane S, Tahon C. Time series modelling and forecasting of emergency department overcrowding. J Med Syst. 2014;38(9):107.

8. Cheng Q, Argon NT, Evans CS, Liu Y, Platts-Mills TF, Ziya S. Forecasting emergency department hourly occupancy using time series analysis. Am J Emerg Med. 2021;48:177-82.

9. Antaki F, Kahwati G, Sebag J, Coussa RG, Fanous A, Duval R, et al. Predictive modeling of proliferative vitreoretinopathy using automated machine learning by ophthalmologists without coding experience. Sci Rep [Internet]. 2020[cited 2021 Jun 24];10(1):19528. Available from: Predictive modeling of proliferative vitreoretinopathy using automated machine learning by ophthalmologists without coding experience - PMC (nih.gov)

10. Faes L, Wagner SK, Fu DJ, Liu X, Korot E, Ledsam JR, et al. Automated deep learning design for medical image classification by health-care professionals with no coding experience: a feasibility study. Lancet Digit Health. 2019;1(5):e232-42.

11. Korot E, Pontikos N, Liu X, Wagner SK, Faes L, Huemer J, et al. Predicting sex from retinal fundus photographs using automated deep learning. Sci Rep [Internet]. 2021[cited 2022 Jan 21];11(1):10286. Available from: Predicting sex from retinal fundus photographs using automated deep learning - PMC (nih.gov)

12. Alaa AM, Bolton T, Di Angelantonio E, Rudd JHF, van der Schaar M. Cardiovascular disease risk prediction using automated machine learning: a prospective study of 423,604 UK Biobank participants. PLoS One [Internet]. 2019 [cited 2020 May 15];14(5):e0213653. Available from: Cardiovascular disease risk prediction using automated machine learning: A prospective study of 423,604 UK Biobank participants - PMC (nih.gov)

13. Abbas A, O’Byrne C, Fu DJ, Moraes G, Balaskas K, Struyven R, et al. Evaluating an automated machine learning model that predicts visual acuity outcomes in patients with neovascular age-related macular degeneration. Graefes Arch Clin Exp Ophthalmol [Internet]. 2022 [cited 2022 Jun 5];260(8):2461-73. Available from: Evaluating an automated machine learning model that predicts visual acuity outcomes in patients with neovascular age-related macular degeneration - PMC (nih.gov)

14. Amazon Web Services. AWS. Amazon forecast developer guide. [Internet]. [cited 2022 Jun 12]. Available from: GitHub - awsdocs/amazon-forecast-developer-guide: The open source version of the Amazon Forecast docs. You can submit feedback & requests for changes by submitting issues in this repo or by making proposed changes & submitting a pull request.

Submitted for publication: March 30, 2022.
Accepted for publication: July 30, 2022.

Funding: This study received no specific financial support

Disclosure of potential conflicts of interest: None of the authors have any potential conflicts of interest to disclose

Approved by the following research ethics committee: Universidade Federal de São Paulo (# 1271/2018)