Abstract

Background: Unjustified emergency department (ED) visits are a global issue, contributing to overcrowding, reduced care quality, decreased patient satisfaction, and increased healthcare costs. A significant proportion of ED referrals are unjustified, likely due to the challenge of accurately determining a patient’s urgency and the lack of effective classification algorithms, particularly within Israel’s healthcare system. To address this gap, this study aims to develop and validate an algorithm to assist community-based physicians in classifying ED visits as justified or unjustified, ultimately improving referral decision-making. Methods: The algorithm was developed in two stages. First, 577 healthcare professionals classified ED visit characteristics—including diagnostic tests, treatments, and clinical conditions—as justified or unjustified. These classifications, along with a literature review and expert consultation, served as the foundation for the algorithm. In the second stage, the algorithm was validated using the Delphi method to achieve expert consensus. Results: In the first stage, 577 healthcare professionals classified ED visit reasons as justified or unjustified, with significant agreement (p < 0.001) on 29 out of 31 cases, while two cases required expert panel review and were ultimately deemed justified. The second stage involved developing a four-step classification algorithm based on diagnostic tests, procedures, treatments, and discharge diagnoses, which was refined through expert discussions and validated using the Delphi method. The final algorithm serves as a structured decision-making tool to assess the justification of ED visits, supporting more accurate referral decisions in both community and emergency care settings. Conclusions: The validated four-step algorithm provides a structured tool for improving referral decision-making, potentially reducing unjustified ED visits and optimizing healthcare resource utilization.

Keywords

Unjustified ED Visits, Algorithm, Healthcare System, Patient Referral, Emergency Care Optimization

Share and Cite:

1. Introduction

Unjustified ED visits represent a global challenge with widespread consequences for healthcare systems. The increasing patient load places significant pressure on healthcare systems. This results in prolonged waiting times, reduced quality of care, decreased patient satisfaction, lower motivation among healthcare professionals, and substantial financial burdens for patients and healthcare providers [1]-[4]. In Israel, ED visits have been steadily increasing, reaching 1.9 million cases in 2022, including 1.8 million non-maternity visits, up from 1.8 million in 2021 and 1.7 million in 2020. The age-adjusted visit rate for non-maternity cases was 189 per 1000 residents, marking a 5% increase from 2021 and 3% from 2019. Additionally, 30% of patients visited the ED at least twice, and return visits within 48 hours accounted for a significant proportion of cases, highlighting potential gaps in care continuity and referral pathways. Despite this rise, only 24% of visits resulted in hospitalization, indicating that a significant portion could have been managed in alternative community-based settings [5].

Despite the availability of community-based medical services, such as urgent care centers, many patients still choose to visit the ED. This tendency is often driven by misconceptions regarding the severity of their condition, lack of trust in the community healthcare system, convenience factors such as service availability and proximity, and recommendations from primary care physicians or nurses [4].

Addressing this issue requires effective decision-making tools to assist healthcare providers in determining the necessity of ED referrals. In recent years, various algorithms have been developed to classify ED visits as justified or unjustified. The NYU-ED Algorithm categorizes visits using retrospective ICD codes, limiting its real-time applicability in referral decision-making [6] [7]. The Minnesota Algorithm improves accuracy by incorporating clinical procedures but is primarily designed for insurance-based healthcare models, where classification is closely tied to reimbursement structures [8]. Similarly, the PERSEE Algorithm enhances ED triage efficiency by evaluating patient severity but does not explicitly determine whether a referral to the ED was justified [9]. Due to these limitations, existing models do not fully align with Israel’s healthcare system, which demands a structured and localized approach to referral decision-making [10]-[12].

Building upon previous research that established criteria for unjustified ED visits [13], this study develops and validates a classification algorithm tailored to the Israeli healthcare system. The algorithm is designed to assist community-based physicians and nurses in making informed decisions regarding patient referrals to the most appropriate healthcare settings. To achieve this, the study first defines criteria for classifying ED visits as justified or unjustified based on diagnostic tests, procedures, treatments, medical conditions, and diagnoses. Next, a structured classification algorithm for ED visits is developed. Finally, the algorithm is validated through expert panel consensus using the Delphi method [14].

2. Methods

2.1. Study Design

This study employed a mixed-methods approach, consisting of two stages: an initial quantitative cross-sectional study, followed by a qualitative validation phase. In the first phase, physicians and nurses completed a structured questionnaire, classifying ED visits as justified or unjustified based on a predefined list of 31 common clinical conditions, diagnostic tests, procedures, and treatments. The results from this stage were used to develop the classification algorithm. In the second stage, the algorithm was reviewed by an expert panel and validated using the Delphi method [14].

2.2. Stage 1—Quantitative Classification of ED Visits

The study included physicians and nurses working in general hospital EDs and community healthcare settings. Participants were required to meet the following inclusion criteria: physicians were required to have at least one year of experience in emergency medicine, internal medicine, pediatrics, or general surgery. Nurses were eligible if they had prior work experience in either EDs or community-based healthcare settings. Participants who did not meet these criteria or failed to complete the questionnaire fully were excluded from the study.

A total of 577 participants were included, of whom 71.2% were physicians (n = 411) and 28.8% were nurses (n = 166) (Table 1). Sample size determination was based on Ministry of Health [15] data, which reported 8537 board-certified physicians and 2934 residents specializing in emergency medicine, internal medicine, pediatrics, or general surgery, and 1270 emergency medicine-trained nurses in Israel. Given an estimated response rate of 10% - 25% [16], the required sample size was calculated for a target population of 12,741 individuals, using a 95% confidence interval and a 5% margin of error. The minimum sample size needed was 373 participants, but 577 participants were successfully recruited, exceeding the required threshold for statistical reliability.

Participants were recruited through cluster sampling, using direct outreach to nursing managers and medical directors in hospitals and community health centers across Israel. Medical and nursing directors from emergency departments (EDs) and community clinics were contacted directly, including Kupat Holim (HMO) district managers and hospital administrators, and were asked to distribute the survey within their institutions. Once institutional approval was obtained, survey links were disseminated through internal mailing lists, ensuring representation from both hospital- and community-based healthcare professionals. The survey was administered electronically via REDCap, a secure platform that ensured data integrity and anonymous participation.

Participants completed a structured questionnaire listing 31 common clinical conditions, classifying each case as justified or non-urgent for an ED visit (Appendix 1).

ED visit classification was based on three categories: justified, unjustified, and inconclusive. Initially, visits were considered unjustified if more than 50% of participants categorized them as such. However, to enhance classification rigor and reliability, a statistical significance threshold was applied. In addition to surpassing the 50% agreement threshold (based on frequency), statistical significance was assessed using a Z-test for one proportion, comparing the observed percentage to a null hypothesis of 50%.

To ensure consistency between physicians and nurses, separate Z-tests for one proportion were conducted for each professional group. A visit was classified as unjustified only when both groups independently reached a clear consensus. If no definitive agreement was reached, the case was forwarded to the expert panel for further evaluation in Stage 2.

2.3. Stage 2—Qualitative Algorithm Development and Validation

The second stage of the study was qualitative and involved the development and validation of the classification algorithm. The process began with the formulation of criteria for justified ED visits, which were based on a comprehensive review of the scientific literature, an analysis of ED visit data, and the Israeli Ministry of Health’s guidelines regarding exemptions from ED admission fees.

An expert panel consisting of five senior emergency medicine physicians was convened to structure the algorithm. The algorithm was designed to classify ED visits as justified or unjustified based on three key domains: diagnostic tests, procedures and treatments performed in the ED, and discharge diagnoses. The panel refined the algorithm to ensure that it accurately reflected clinical practice and healthcare policies in Israel.

The validation process was conducted using the Delphi method, in which multiple rounds of classification and evaluation were carried out until expert consensus was achieved. In each round, experts reviewed and reassessed the classification criteria, providing feedback and modifications as necessary. Decisions were finalized once at least 70% of the panel members reached an agreement, ensuring the final algorithm’s high reliability and clinical applicability (10).

3. Results

3.1. Phase 1—Quantitative Classification of ED Visit Reasons

The first stage of the study included 577 participants: 411 physicians (71.2%) and 166 nurses (28.8%). Most participants were secular Jewish women. Physicians primarily specialized in family medicine or pediatrics, with some holding dual specializations, while nurses had post-basic training (Table 1).

ED visit reasons were classified as justified if they involved diagnostic tests, procedures, treatments, or discharge diagnoses deemed necessary by the participants (Table 2). A visit was considered unjustified if more than 50% of participants classified it as such, with statistical significance assessed using Z-tests for one proportion (p < 0.001) to ensure that agreement was not due to random variation (Appendix 2).

In most cases, there was strong agreement between physicians and nurses. However, when the classification agreement fell below 65%, the case was referred to the expert panel for final evaluation to ensure accuracy and consistency. Two specific cases—suturing a wound under local anesthesia without requiring a plastic surgeon and immobilization of a non-displaced fracture using splinting, casting, or bandaging—showed classification discrepancies between physicians and nurses. Following an expert panel review, both cases were ultimately classified as justified ED visits, reflecting their necessity based on clinical best practices and emergency care standards.

Table 1. Background characteristics of study participants.

Table 1 presents the means and standard deviations (SD) for demographic and occupational variables of physicians (N = 411) and nurses (N = 166), including age, years of experience, gender, nationality, and religion. The mean age of physicians was 49.95 years (SD = 12.52), compared to 37.17 years (SD = 12.55) among nurses. The mean years of experience in the profession were 20.46 years (SD = 12.99) for physicians and 9.61 years (SD = 13.07) for nurses. Among physicians, 53.4% were male, compared to 17.5% of nurses, while 46.6% of physicians and 82.5% of nurses were female. In terms of nationality, 78.0% of physicians and 87.3% of nurses were Jewish, whereas 19.3% of physicians and 9.1% of nurses were Arab. Regarding religious affiliation, 67.6% of physicians and 52.4% of nurses identified as secular, while 18.4% of physicians and 27.7% of nurses identified as religious.

Table 2. Classification of justified and unjustified ed visits–assessment by study participants.

Table 2 presents the classification of all study participants (N = 577) regarding whether each ED visit reason was justified or unjustified. An ED visit was considered justified when classification consensus exceeded 65%. Statistical significance was determined using the Z- test for one proportion, with a significance level of α = 0.05.

3.2. Stage 2—Qualitative Algorithm Development and Validation by Expert Panel

The second stage of the study focused on the development and validation of a classification algorithm by an expert panel.

3.2.1. Algorithm Development Process

The algorithm was developed and refined through iterative expert panel discussions using the Delphi method. The process followed four key steps. The first step involved defining the algorithm’s structure. The expert panel reached a consensus on a stepwise classification framework (“blocks”), in which each block contained predefined classification criteria. If an ED visit met at least one criterion within a block, it was classified as justified; otherwise, it proceeded to the next block.

Following this, the classification framework was established. The final algorithm consists of four sequential blocks, each representing a key aspect of ED visit classification: administrative characteristics of the visit, diagnostic tests performed, treatments provided, and discharge diagnoses.

The next step focused on defining the classification criteria. The classification of diagnostic tests (imaging, laboratory), specialist consultations, procedures, and treatments was based on two guiding principles: (1) the test or procedure must require ED-level care, and (2) the test or procedure must be widely available in community-based urgent care centers. For discharge diagnoses (Block 4), the panel reviewed over 12,000 ED discharge records (excluding hospitalized cases) and categorized them into two groups: (1) conditions commonly managed in primary or secondary care, including urgent care centers, and (2) conditions requiring ED evaluation (Appendix 3).

Finally, the algorithm was validated and finalized through multiple Delphi method rounds, ensuring expert consensus. The final four-step classification process is presented in Figure 1, guiding the determination of justified vs. unjustified ED visits. ED visits are first assessed using Blocks 1, 2, and 3. If no criteria are met, Block 4 is used to determine classification based on discharge diagnoses (Appendix 3).

Figure 1. Algorithm for classifying ed visits as justified or unjustified (Diagnoses in Appendix 3).

Figure 1 presents the final version of the algorithm for classifying ED visits, which consists of four steps. The algorithm determines whether a visit is justified or unjustified by following the outlined steps. In the first step, the patient’s condition is evaluated against the criteria listed in Blocks 1, 2, and 3. If none of these criteria are met, the next step, as described in Block 4, directs the user to consult the list of diagnoses found in Appendix 3.

3.2.2. Theoretical Examples of Algorithm Application

To illustrate the practical application of the classification algorithm, two theoretical case studies were analysed. These cases were selected as they represent common clinical presentations that often raise uncertainty in ED referral decisions. In the first case, a 40-year-old male presented to the ED alone, complaining of a headache with no additional symptoms. He was examined by an ED physician and subsequently discharged without receiving any treatment. Since the visit did not meet the criteria in Block 1 (e.g., no hospitalization), it was further assessed against Block 2, which evaluates diagnostic testing. As no imaging tests (such as CT) were performed, the case proceeded to Block 3, where the absence of specialist consultations was noted. The final classification relied on Block 4, in which the diagnosis “headache” was not included in the list of conditions warranting ED care. Consequently, this visit was classified as unjustified.

In the second case, a 62-year-old female presented to the ED with upper back pain. Similar to the first case, her visit did not meet any criteria in Block 1 (e.g., no hospitalization). However, a troponin test was performed during her ED visit, which, according to Block 2, is a key diagnostic test that necessitates ED evaluation. As a result, her visit was automatically classified as justified, without the need for additional assessment in Blocks 3 or 4.

4. Discussion

This study builds upon and refines previous classification models by incorporating additional filtering steps and real-time clinical assessments, thereby enhancing the accuracy of unjustified ED visit identification. This study aimed to develop and validate an algorithm capable of classifying ED visits as justified or unjustified. To achieve this, a survey was conducted among physicians and nurses, who were asked to classify common ED visit reasons as justified or unjustified. Based on the collected data, along with a review of the existing literature, an algorithm was developed and subsequently validated by an expert panel.

The findings from the quantitative phase of the study indicate that justified ED visits were characterized by the performance of specific diagnostic tests, procedures, or treatments. These included CT scans, ultrasound, troponin or D-dimer tests, gastroscopy, arterial blood gas analysis, and blood cultures, as well as tendon suturing, immobilization of non-displaced fractures, reduction of displaced fractures under sedation, and ED visits for chest pain evaluation. Similar findings have been reported in previous studies. For instance, Lin and Lee [17] defined justified ED visits as those that could not be adequately managed in community-based settings and required immediate diagnosis or treatment in the ED. Similarly, Leshinski et al. [13] classified justified ED visits as those involving specific laboratory tests (e.g., troponin), imaging studies, and essential treatments such as fracture management, ophthalmic procedures, and acute injury care.

Regarding the use of ED discharge diagnoses for visit classification, which was implemented in the qualitative phase of the study, our findings are supported by previous literature. For example, Chen et al. [10] reviewed 15 studies that used ICD coding systems (ICD-9 and ICD-10) to identify ED visits with low urgency levels. These studies found that different classification algorithms produced varying results and that, in most cases, existing algorithms had limited accuracy in identifying unjustified ED visits—those that could have been managed in primary care settings. Unlike these studies, the current study did not rely solely on ICD coding but incorporated additional filtering steps based on diagnostic tests, procedures, treatments, and specialist consultations. This approach ensured better alignment with the Israeli healthcare system, distinguishing between services readily available in community urgent care centers and those requiring ED-level care.

Another widely used classification tool is the NYU-ED Algorithm (The Billings New York University Emergency Department Algorithm), which determines the justification of ED visits. In contrast to our research, the NYU-ED algorithm considers clinical evaluations and physician opinions in addition to ICD coding [6] [17]. The NYU-ED algorithm categorizes ED visits into five levels, ranging from completely unjustified to requiring ED care. However, while the NYU-ED algorithm provides an initial classification of ED visit justification, it does not consider procedures performed during the visit, relying instead on symptoms and patient history alone [6] [7]. Unlike the NYU-ED algorithm, which relies on retrospective coding, our algorithm allows for real-time classification by incorporating diagnostic tests and clinical assessments, making it more applicable for primary care and triage decision-making.

To address this limitation, the Minnesota Algorithm was developed. This algorithm accounts for both presenting symptoms and procedures performed during the ED visit, utilizing real-time patient data to improve classification accuracy. Studies have shown that the Minnesota Algorithm is particularly effective in predicting cases that could be managed in community settings. Moreover, it has demonstrated greater precision in predicting illness severity, ED-related complications, and mortality risk, especially among older adults (aged 65 and above), whose healthcare records tend to be more comprehensive in the U.S. insurance system [8]. Corresponding to the algorithm created in this study, the Minnesota Algorithm incorporates clinical judgment, ED procedures, and a comprehensive list of clinical diagnoses.

Some classification models, such as PERSEE, incorporate additional triage elements that can assist in decision-making upon the patient’s arrival at the ED. The PERSEE algorithm is designed for triage within the ED, regardless of whether the patient arrives independently or with a physician’s referral. It is based on two widely used clinical tools: (1) the ELISA Scale, which classifies patients into five severity levels based on their clinical condition, facilitating appropriate referrals to community healthcare settings, and (2) the SALOMON Scale, which categorizes patients into four severity levels according to the treatment required [9].

The PERSEE algorithm integrates both clinical diagnosis and expected treatment pathways, similar to the approach taken in the current study. Research on PERSEE has demonstrated its effectiveness in reducing unjustified ED visits, although a classification error rate of approximately 7% has been reported [9]. Unlike PERSEE, which is implemented within the ED itself, the algorithm developed in this study is intended for use both in primary care settings and at the ED triage stage, enabling proactive referral guidance before a patient arrives at the ED. Furthermore, while PERSEE primarily refers unjustified cases to community-based primary care, the current algorithm also considers urgent care centers as an alternative treatment pathway, reflecting the broader range of healthcare options available in Israel [18].

Given the increasing burden on EDs, policymakers should consider integrating this classification algorithm into national triage protocols to optimize healthcare resource allocation and improve patient outcomes. From a policy perspective, implementing this classification system in community healthcare settings and ED triage could optimize referral accuracy, reduce unnecessary hospital burdens, and support cost-effective resource allocation. Given that most ED visits in Israel require a physician or nurse referral, incorporating this algorithm into primary care decision-making may improve patient flow and enhance resource distribution within the healthcare system. Additionally, refining the algorithm to integrate with real-time clinical decision support tools could further enhance its applicability.

Future research should focus on external validation using real-world patient data, assessing the algorithm’s impact on actual referral patterns and ED overcrowding, and exploring its potential integration into digital health platforms for automated triage assistance. While this study presents a novel classification approach, further real-world validation is required to assess its effectiveness in diverse healthcare settings. Long-term studies examining how this algorithm affects patient outcomes, healthcare costs, and physician decision-making behaviours will be essential for its continued refinement and implementation.

5. Conclusions

Several algorithms have been developed for classifying ED visits, but many fail to encompass all relevant clinical aspects and do not consider a wide range of diagnoses that may affect visit justification. Additionally, many existing models do not account for alternative care pathways, particularly for patients requiring urgent care outside of the standard operating hours of the community health setting. The algorithm developed in this study addresses these gaps by providing a more comprehensive classification system, incorporating a broader range of unjustified ED visit reasons, and facilitating referrals to appropriate community-based healthcare settings.

This algorithm is designed to assist community-based physicians and nurses working in telehealth triage services in making evidence-based decisions regarding ED referrals. In Israel, most ED visits require a referral from a physician or nurse, which implies that this algorithm could help reduce the number of unjustified ED visits while also supporting decision-making processes related to reimbursement policies for self-referred patients.

Appendices

Appendix 1

Questionnaire on Reasons for Unjustified ED Visits

Dear Participant,

This questionnaire is part of a doctoral research study examining the factors influencing emergency department (ED) visits. The study is conducted by Roman Leshinski under the supervision of Dr. Ygal Plakht from the Department of Nursing at Ben-Gurion University of the Negev. The questionnaire is entirely anonymous. While participation is voluntary, completing the full questionnaire will greatly contribute to the research.

Below are possible reasons for visits to general hospital emergency departments (EDs). Please indicate next to each reason whether you consider it justified or unjustified. Each reason should be evaluated separately.

(1) General information

Please fill in the answers or select the most appropriate option.

(2) ED visits classification table

Please select from the list of ED visit reasons, diagnostic tests, imaging, treatments, and discharge diagnoses regarding the justification for an ED visit from your perspective (referring to ED visits that ended in discharge without hospitalization).

Appendix 2

Appendix 2. Table describing the classification of nurses and physicians for each criterion/reason/diagnosis as justified or unjustified. A visit was classified as justified if the classification was definitive (above 65%). Statistical significance was calculated using a Z-test for one proportion, separately for physicians and nurses. The significance level was set at α = 0.05.

Appendix 3

Link to the list of discharge diagnoses that are considered justified for ED visits. https://docs.google.com/spreadsheets/d/1APZOGuBs4dGkCJkh79wO080oD8mOPJI5/edit?gid=1709948223#gid=1709948223&range=B2

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References