This study employed a mixed-methods approach, consisting of two stages: an initial quantitative cross-sectional study, followed by a qualitative validation phase. In the first phase, physicians and nurses completed a structured questionnaire, classifying ED visits as justified or unjustified based on a predefined list of 31 common clinical conditions, diagnostic tests, procedures, and treatments. The results from this stage were used to develop the classification algorithm. In the second stage, the algorithm was reviewed by an expert panel and validated using the Delphi method [14] .

The study included physicians and nurses working in general hospital EDs and community healthcare settings. Participants were required to meet the following inclusion criteria: physicians were required to have at least one year of experience in emergency medicine, internal medicine, pediatrics, or general surgery. Nurses were eligible if they had prior work experience in either EDs or community-based healthcare settings. Participants who did not meet these criteria or failed to complete the questionnaire fully were excluded from the study.

A total of 577 participants were included, of whom 71.2% were physicians (n = 411) and 28.8% were nurses (n = 166) ( Table 1 ). Sample size determination was based on Ministry of Health [15] data, which reported 8537 board-certified physicians and 2934 residents specializing in emergency medicine, internal medicine, pediatrics, or general surgery, and 1270 emergency medicine-trained nurses in Israel. Given an estimated response rate of 10% - 25% [16] , the required sample size was calculated for a target population of 12,741 individuals, using a 95% confidence interval and a 5% margin of error. The minimum sample size needed was 373 participants, but 577 participants were successfully recruited, exceeding the required threshold for statistical reliability.

Participants were recruited through cluster sampling, using direct outreach to nursing managers and medical directors in hospitals and community health centers across Israel. Medical and nursing directors from emergency departments (EDs) and community clinics were contacted directly, including Kupat Holim (HMO) district managers and hospital administrators, and were asked to distribute the survey within their institutions. Once institutional approval was obtained, survey links were disseminated through internal mailing lists, ensuring representation from both hospital- and community-based healthcare professionals. The survey was administered electronically via REDCap, a secure platform that ensured data integrity and anonymous participation.

Participants completed a structured questionnaire listing 31 common clinical conditions, classifying each case as justified or non-urgent for an ED visit ( Appendix 1 ).

ED visit classification was based on three categories: justified, unjustified, and inconclusive. Initially, visits were considered unjustified if more than 50% of participants categorized them as such. However, to enhance classification rigor and reliability, a statistical significance threshold was applied. In addition to surpassing the 50% agreement threshold (based on frequency), statistical significance was assessed using a Z-test for one proportion, comparing the observed percentage to a null hypothesis of 50%.

To ensure consistency between physicians and nurses, separate Z-tests for one proportion were conducted for each professional group. A visit was classified as unjustified only when both groups independently reached a clear consensus. If no definitive agreement was reached, the case was forwarded to the expert panel for further evaluation in Stage 2.

The second stage of the study was qualitative and involved the development and validation of the classification algorithm. The process began with the formulation of criteria for justified ED visits, which were based on a comprehensive review of the scientific literature, an analysis of ED visit data, and the Israeli Ministry of Health’s guidelines regarding exemptions from ED admission fees.

An expert panel consisting of five senior emergency medicine physicians was convened to structure the algorithm. The algorithm was designed to classify ED visits as justified or unjustified based on three key domains: diagnostic tests, procedures and treatments performed in the ED, and discharge diagnoses. The panel refined the algorithm to ensure that it accurately reflected clinical practice and healthcare policies in Israel.

The validation process was conducted using the Delphi method, in which multiple rounds of classification and evaluation were carried out until expert consensus was achieved. In each round, experts reviewed and reassessed the classification criteria, providing feedback and modifications as necessary. Decisions were finalized once at least 70% of the panel members reached an agreement, ensuring the final algorithm’s high reliability and clinical applicability (10).

The first stage of the study included 577 participants: 411 physicians (71.2%) and 166 nurses (28.8%). Most participants were secular Jewish women. Physicians primarily specialized in family medicine or pediatrics, with some holding dual specializations, while nurses had post-basic training ( Table 1 ).

ED visit reasons were classified as justified if they involved diagnostic tests, procedures, treatments, or discharge diagnoses deemed necessary by the participants ( Table 2 ). A visit was considered unjustified if more than 50% of participants classified it as such, with statistical significance assessed using Z-tests for one proportion (p < 0.001) to ensure that agreement was not due to random variation ( Appendix 2 ).

In most cases, there was strong agreement between physicians and nurses. However, when the classification agreement fell below 65%, the case was referred to the expert panel for final evaluation to ensure accuracy and consistency. Two specific cases—suturing a wound under local anesthesia without requiring a plastic surgeon and immobilization of a non-displaced fracture using splinting, casting, or bandaging—showed classification discrepancies between physicians and nurses. Following an expert panel review, both cases were ultimately classified as justified ED visits, reflecting their necessity based on clinical best practices and emergency care standards.

Table 1 presents the means and standard deviations (SD) for demographic and occupational variables of physicians (N = 411) and nurses (N = 166), including age, years of experience, gender, nationality, and religion. The mean age of physicians was 49.95 years (SD = 12.52), compared to 37.17 years (SD = 12.55) among nurses. The mean years of experience in the profession were 20.46 years (SD = 12.99) for physicians and 9.61 years (SD = 13.07) for nurses. Among physicians, 53.4% were male, compared to 17.5% of nurses, while 46.6% of physicians and 82.5% of nurses were female. In terms of nationality, 78.0% of physicians and 87.3% of nurses were Jewish, whereas 19.3% of physicians and 9.1% of nurses were Arab. Regarding religious affiliation, 67.6% of physicians and 52.4% of nurses identified as secular, while 18.4% of physicians and 27.7% of nurses identified as religious.

Table 2 presents the classification of all study participants (N = 577) regarding whether each ED visit reason was justified or unjustified. An ED visit was considered justified when classification consensus exceeded 65%. Statistical significance was determined using the Z- test for one proportion, with a significance level of α = 0.05.

The second stage of the study focused on the development and validation of a classification algorithm by an expert panel.

The algorithm was developed and refined through iterative expert panel discussions using the Delphi method. The process followed four key steps. The first step involved defining the algorithm’s structure. The expert panel reached a consensus on a stepwise classification framework (“blocks”), in which each block contained predefined classification criteria. If an ED visit met at least one criterion within a block, it was classified as justified; otherwise, it proceeded to the next block.

Following this, the classification framework was established. The final algorithm consists of four sequential blocks, each representing a key aspect of ED visit classification: administrative characteristics of the visit, diagnostic tests performed, treatments provided, and discharge diagnoses.

The next step focused on defining the classification criteria. The classification of diagnostic tests (imaging, laboratory), specialist consultations, procedures, and treatments was based on two guiding principles: (1) the test or procedure must require ED-level care, and (2) the test or procedure must be widely available in community-based urgent care centers. For discharge diagnoses (Block 4), the panel reviewed over 12,000 ED discharge records (excluding hospitalized cases) and categorized them into two groups: (1) conditions commonly managed in primary or secondary care, including urgent care centers, and (2) conditions requiring ED evaluation ( Appendix 3 ).

Finally, the algorithm was validated and finalized through multiple Delphi method rounds, ensuring expert consensus. The final four-step classification process is presented in Figure 1 , guiding the determination of justified vs. unjustified ED visits. ED visits are first assessed using Blocks 1, 2, and 3. If no criteria are met, Block 4 is used to determine classification based on discharge diagnoses ( Appendix 3 ).

Figure 1 presents the final version of the algorithm for classifying ED visits, which consists of four steps. The algorithm determines whether a visit is justified or unjustified by following the outlined steps. In the first step, the patient’s condition is evaluated against the criteria listed in Blocks 1, 2, and 3. If none of these criteria are met, the next step, as described in Block 4, directs the user to consult the list of diagnoses found in Appendix 3 .

To illustrate the practical application of the classification algorithm, two theoretical case studies were analysed. These cases were selected as they represent common clinical presentations that often raise uncertainty in ED referral decisions. In the first case, a 40-year-old male presented to the ED alone, complaining of a headache with no additional symptoms. He was examined by an ED physician and subsequently discharged without receiving any treatment. Since the visit did not meet the criteria in Block 1 (e.g., no hospitalization), it was further assessed against Block 2, which evaluates diagnostic testing. As no imaging tests (such as CT) were performed, the case proceeded to Block 3, where the absence of specialist consultations was noted. The final classification relied on Block 4, in which the diagnosis “headache” was not included in the list of conditions warranting ED care. Consequently, this visit was classified as unjustified.

In the second case, a 62-year-old female presented to the ED with upper back pain. Similar to the first case, her visit did not meet any criteria in Block 1 (e.g., no hospitalization). However, a troponin test was performed during her ED visit, which, according to Block 2, is a key diagnostic test that necessitates ED evaluation. As a result, her visit was automatically classified as justified, without the need for additional assessment in Blocks 3 or 4.

Questionnaire on Reasons for Unjustified ED Visits

Dear Participant,

This questionnaire is part of a doctoral research study examining the factors influencing emergency department (ED) visits. The study is conducted by Roman Leshinski under the supervision of Dr. Ygal Plakht from the Department of Nursing at Ben-Gurion University of the Negev. The questionnaire is entirely anonymous. While participation is voluntary, completing the full questionnaire will greatly contribute to the research.

Below are possible reasons for visits to general hospital emergency departments (EDs). Please indicate next to each reason whether you consider it justified or unjustified. Each reason should be evaluated separately.

(1) General information

Please fill in the answers or select the most appropriate option.

(2) ED visits classification table

Please select from the list of ED visit reasons, diagnostic tests, imaging, treatments, and discharge diagnoses regarding the justification for an ED visit from your perspective (referring to ED visits that ended in discharge without hospitalization).

Appendix 2. Table describing the classification of nurses and physicians for each criterion/reason/diagnosis as justified or unjustified. A visit was classified as justified if the classification was definitive (above 65%). Statistical significance was calculated using a Z-test for one proportion, separately for physicians and nurses. The significance level was set at α = 0.05.

Link to the list of discharge diagnoses that are considered justified for ED visits. https://docs.google.com/spreadsheets/d/1APZOGuBs4dGkCJkh79wO080oD8mOPJI5/edit?gid=1709948223#gid=1709948223&range=B2