Advertisement

Developing Quality Indicators for Helicopter Emergency Medical Services Coordination in Norwegian Emergency Medical Communication Centrals: A Consensus Process

Open AccessPublished:December 11, 2020DOI:https://doi.org/10.1016/j.amj.2020.11.010

      Abstract

      Objective

      Efforts to optimize the use, availability, and safety of helicopter emergency medical services (HEMS) is important. A lack of consistent and comprehensive flight dispatch procedures and a lack of use of safety technology are recurring safety problems. Reports after several major incidents pointed toward a possible gain by coordinating Norwegian HEMS from regional emergency medical communication centrals. Our objective was to develop and implement relevant quality indicators before such implementation in central Norway.

      Methods

      We recruited an expert panel of 24 persons representing Norwegian health authorities, emergency medical communication centrals, and HEMS bases and performed a 3-step e-mail–based Delphi process to develop relevant quality indicators. Each indicator was assessed according to their feasibility, rankability, actionability, and variability. To reach a consensus, a median score of 5 or more on a 6-point Likert scale in step 3 was needed.

      Results

      A total of 61 quality indicators were proposed. Of the 14 indicators that reached a consensus, 12 of these were considered process indicators, and 2 were bordering to outcome indicators.

      Conclusion

      We applied a Delphi process method to develop quality indicators for HEMS coordination and flight following. An experienced and heterogeneous expert panel suggested and reached a consensus on which quality indicators should be applied.
      Helicopter emergency medical services (HEMS) are an integrated part of health care systems, especially in high-income countries.

      World Health Organization. WHO Emergency care system framework, 2018. Available at: https://www.who.int/publications-detail/who-emergency-care-system-framework. Accessed June 30, 2020.

      ,
      • Taylor CB
      • Stevenson M
      • Jan S
      • et al.
      An investigation into the cost, coverage and activities of helicopter emergency medical services in the state of New South Wales, Australia.
      The main purpose of HEMS is to bring the competency of the HEMS crew to the scene, provide advanced interventions beyond the scope of most emergency medical services (EMS), shorten the transport time to the hospital for patients with time-critical conditions, and provide access to locations not reachable by other means of transportation.
      • Reid BO
      • Rehn M
      • Uleberg O
      • et al.
      Physician-provided prehospital critical care, effect on patient physiology dynamics and on-scene time.
      ,
      • Røislien J
      • van den Berg PL
      • Lindner T
      • et al.
      Comparing population and incident data for optimal air ambulance base locations in Norway.
      Because this service is a cost-intensive and limited resource, interventions to provide efficient use are important. To achieve the optimal use and availability of air ambulance resources, indications for HEMS dispatch and guidelines in choosing the most appropriate resource must be available. Moreover, flight safety must be ensured because the risks involved in HEMS missions are substantial. A lack of consistent and comprehensive flight dispatch procedures and a lack of use of safety technology are identified by the National Transportation Safety Board as recurring safety problems.
      • Haugland H
      • Uleberg O
      • Klepstad P
      • et al.
      Quality measurement in physician-staffed emergency medical services: a systematic literature review.
      Risk mitigation should be an obligate task for any HEMS provider.
      European civilian air medical transport is subject to the regulations of the European Union Aviation Safety Agency.

      European Aviation Safety Agency (EASA). Acceptable means of compliance (AMC) and guidance material (GM). Available at: https://www.easa.europa.eu/sites/default/files/dfu/Consolidated%20unofficial%20AMC%26GM_Annex%20V%20Part-SPA.pdf. Accessed June 30, 2020.

      Norwegian Air Ambulance (Norsk Luftambulanse AS), the operator of all HEMS bases in Norway, performs single-pilot HEMS crew operations, with 24-hour availability 7 days a week. European Union Aviation Safety Agency regulations require an established flight following system. A flight following system consists of 1 or more persons who are, on behalf of the operator, continuously monitoring HEMS actions, both visually using transponder technology as well as radio communication. Flight following is performed throughout any HEMS action, from takeoff to landing on each leg throughout its operational area. This is comparable to the tasks performed by air traffic control (ATC) for commercial airlines. Because of the fact that HEMS operations are mainly performed outside controlled airspaces and the Nordic geography provides poor flight radio (very high frequency) coverage below commercial flight levels, it is not feasible nor sufficient to let ATC perform HEMS flight following. In January 2017, the Norwegian Minister of Health decided to centralize HEMS coordination and flight following (ie, 4 regional emergency medical communication centrals [EMCCs] rather than 12 local EMCCs located near the HEMS bases). This decision was based on a governmental report published in 2016 and several other reports recommending a more centralized HEMS coordination.
      Norwegian Government
      Ministry of Health and Care Services. Central elements regarding the organization of emergency medical dispatch centrals [Sentrale elementer vedrørende organisering av AMK-Sentralene].
      Norwegian Government
      Ministry of Health and Care Services. First and foremost – a comprehensive system for dealing with acute diseases and injuries outside hospitals. NOU 2015:17. Norwegian Government.
      Norwegian Directorate of Health
      Learning for better emergency preparedness – the medical response to the terrorist incidents of 22 July 2011.IS-1984.
      National Air Ambulance Service
      Evaluation report following the Oslo Government District Bombing and Utøya Island Shooting July 22, 2011.
      Severe incidents like the mass shooting at Utøya Island (July 22, 2011) and the fatal HEMS crash at Sollihøgda near Oslo in 2014 led to recommendations to improve HEMS coordination and flight following.
      Norwegian Directorate of Health
      Learning for better emergency preparedness – the medical response to the terrorist incidents of 22 July 2011.IS-1984.
      ,
      National Air Ambulance Service
      Evaluation report following the Oslo Government District Bombing and Utøya Island Shooting July 22, 2011.
      Centralization of these functions to the regional EMCCs was anticipated to ensure a less fragmented and a more efficient use of HEMS resources. In addition, the extended use of instrument flight rules capabilities enabling operations in more challenging weather conditions required a stricter control of HEMS operations in uncontrolled airspace to ensure flight safety. Norwegian EMCCs, which are coordinating EMS resources, are also performing HEMS dispatch and compulsory flight following for the National Air Ambulance Service (NAAS).
      In general, all EMS systems changes should be accompanied by continuous evaluations to ensure that the desired effect is achieved.
      • Haugland H
      • Uleberg O
      • Klepstad P
      • et al.
      Quality measurement in physician-staffed emergency medical services: a systematic literature review.
      In our case, documenting a set of relevant quality indicators (QIs) could enable studying the effect of centralized HEMS coordination and flight following in our region. As such, the aim of this study was to develop and implement QIs before establishing HEMS coordination and flight following by the regional EMCC in central Norway.

      Methods

      Study Setting

      Norway has a national and publicly funded air ambulance network consisting of 12 HEMS bases and 7 fixed wing bases. The HEMS team consists of a consultant-level anesthesiologist, a pilot, and an HEMS crewmember (HCM).
      • Østerås Ø
      • Brattebø G
      • Heltne J-K
      Helicopter-based emergency medical services for a sparsely populated region: a study of 42,500 dispatches.
      The HCM is a specially trained paramedic adhering to the national Norwegian HCM requirements including a bachelor's degree in paramedicine or nursing, at least 2 years of prehospital clinical experience, and competencies within rescue techniques.
      Norwegian Government
      Ministry of Health and Care Services. National Standard Requirements for rescuers in the air ambulance service, the rescue helicopter service and SAR – offshore. [Nasjonal standard for redningsmenn i luftambulansetjenesten, redningshelikoptertjenesten og SAR – offshore].
      In addition, 6 search and rescue (SAR) helicopter bases complement the NAAS. The objective of the service is to provide advanced medical care to critically injured or ill patients. The service operates 24 hours a day, 7 days a week, 365 days a year. Four regional health authorities (RHAs) manage the Norwegian health care system. Until 2019, regional coordination and flight following were only performed in the South-East RHA. In other RHAs, local EMCCs in the vicinity of each HEMS base performed dispatch and flight following for their respective HEMS resources. An important task of the HEMS crew after dispatch but before initiating the mission is to perform safety and weather checks to reduce operational risk. Since the start-up in South-East RHA in 2014, the only followed QI for HEMS coordination was the delay caused by the local EMCC calling the regional HEMS EMCC instead of directly dispatching the HEMS resource. The indicator should monitor and ensure a maximal delay of 2 minutes. After 6 months of consistent results of delay averaging at 35 seconds, documentation of the indicator was discontinued.
      In November 2017, the Central Norway RHA initiated the process of establishing centralized regional HEMS coordination and flight following. There are 3 local EMCCs (2020) in this region, covering a population of 733,940 inhabitants (2020).
      Statistics Norway
      Population data.
      A task group assessed the appropriate location for a regional HEMS coordinating EMCC, which resulted in the establishment of this service at the St. Olav's University Hospital in Trondheim.
      Central Norway Regional Health Authority
      Coordination of air ambulance helicopters in Central Norway [in Norwegian].
      There are 2 HEMS bases and 1 SAR helicopter within the region of central Norway. Three other HEMS bases and 1 SAR helicopter are located in close proximity to the region and are often used for missions in the outskirts of the region (Fig. 1).
      Figure 1
      Figure 1An overview of the operational area. The dark gray area outlines Central Norway RHA with regional and extraregional HEMS and SAR resources, EMCCs, and hospitals.

      Concept: QIs

      According to the framework described by Donabedian,
      • Donabedian A
      The quality of care. How can it be assessed?.
      ,
      • Donabedian A
      Evaluating the quality of medical care.
      QIs are grouped into structure, process, or outcome indicators. Structure indicators describe the infrastructure of a system, like the competence of providers, the available equipment, and response times. Process indicators evaluate the performance of key processes in a system. In an EMCC, a key process would be providing information to the responding ambulances and HEMS. Outcome indicators assess the achieved result of the performed processes. Each of these 3 groups should ideally be represented in a comprehensive set of QIs. Moreover, the indicators should be feasible, rankable, actionable, and variable.
      • Rubin HR
      • Pronovost P
      • Diette GB
      From a process of care to a measure: the development and testing of a quality indicator.
      ,
      • Haugland H
      • Olkinuora A
      • Rognås L
      • et al.
      Testing quality indicators and proposing benchmarks for physician-staffed emergency medical services: a prospective Nordic multicentre study.
      By this, one understands that a QI, which is nonmeasurable, is of little value. The QI should also be possible to rank, from high to low or good to bad. Furthermore, the QI should enable us to take action to influence the QI performance, and if there is no variation for a QI, there is little need to monitor it and often even little room for improvement. Finally, for a QI to be valid, there should be a documented relationship between the structure or process measured by the QI and an improved outcome. In general, this is a challenge in EMS because of the lack of documentation in certain aspects of prehospital care. More specifically, for the evaluation of HEMS coordination and flight following, we consider it an even bigger challenge because only a few studies exist pertaining to this service.

      Literature Search

      To inform the upcoming consensus process and to ensure adherence to eventual similar international processes, we performed a nonsystematic search in PubMed, Scopus, and INSPEC to identify literature on QIs related to HEMS coordination and flight following. No systematic literature review was performed.

      Delphi Process

      The availability of scientific evidence and literature on QIs related to HEMS coordination and flight following is scarce. Therefore, we planned to do a Delphi process. A Delphi process is a recognized technique for developing QIs when literature does not provide sufficient answers.
      • Murphy MK
      • Black NA
      • Lamping DL
      • et al.
      Consensus development methods, and their use in clinical guideline development.
      The study group planned and designed a 3-step e-mail–based Delphi process to develop QIs for HEMS coordination and flight following. In a Delphi process, an expert panel is established to reach consensus over a specific topic. This is done through several steps of e-mail correspondence with or without a panel gathering.
      • Thangaratinam S
      • Redman CWE
      The Delphi technique.

      Expert Panel

      The composition of an expert panel developing QIs for HEMS coordination and flight following had to reflect the diversity of professionals involved in these services. This is imperative to give the results the necessary credibility in the target audience.
      • Murphy MK
      • Black NA
      • Lamping DL
      • et al.
      Consensus development methods, and their use in clinical guideline development.
      To secure a comprehensive selection of opinions from different stakeholders in HEMS operations, several organizational levels of interest were identified (Table 1). In addition, geographic representation was emphasized. We invited 38 possible participants to join the expert panel. Twenty-four of the 38 invited experts accepted the invitation. The experts represented 4 different Norwegian EMCCs, 4 Norwegian HEMS bases, the Central Norway RHA, and the health authority responsible for the operational aspects of the NAAS (Luftambulansetjenesten HF). Professions represented in the panel included doctors, nurses, administrators, HCMs, and pilots. The reference group consisted of the included authors of this article.
      Table 1An Overview of the Composition of the Expert Panel
      OrganizationTitleRoleAdditional Functions
      HEMS bases in central NorwayPilot (2)

      HCM (2)

      Doctor (4)
      OperativeBase chief pilot (1)

      Lead HCM (1)

      Medical advisor to EMCC (2)
      HEMS bases in southeast NorwayPilot (1)

      HCM (2)

      Doctor (1)
      OperativeSafety manager (1)
      EMCC Sør TrøndelagEMCC operator (3)

      Head of department (1)
      Operative
      EMCC Møre and RomsdalEMCC operator (1)

      Head of Department (1)
      Operative

      Administrative
      EMCC Nord-TrøndelagHead of Department (1)

      Medical advisor (1)
      Administrative

      EMCC HEMS OsloHEMS coordinator (1)Operative
      Luftambulanstjenesten HFMedical advisor (2)Administrative
      Central Norway RHAMedical advisor (1)Administrative
      EMCC = emergency medical coordination center; HCM = helicopter emergency medical services crewmember; HEMS = helicopter emergency medical services; RHA = regional health authority.
      Table 2An Overview of the Final Quality Indicators (QIs)
      QI #Mission PhaseQI DesignationCategoryQI Definition
      2Initial phaseAccess time HEMS coordinatorProcessTime from local EMCC calls HEMS coordinator to answered call
      13Reaction phaseLocal EMCC available on radioProcessLocal EMCC is available for supplement on radio after alarm information is provided by HEMS coordinator
      21Reaction phasePrehospital response timeProcessTime from need for HEMS is established to HEMS resource arrives at patient location
      28Reaction phaseCooperation between HEMS EMCCsProcessNumber of missions solved by extraregional HEMS and HEMS coordinator reaction time
      36Flight followingAvailability HEMS coordinatorProcessPerceived availability of HEMS coordinator by crew while on mission
      42Flight followingHEMS coordinator provides known landing siteProcessNumber of missions where LZ North
      https://lznorth.no/.
      (landing site application) was used to inform crew on nearby known landing sites to reduce risk
      43In flightIn-flight dispatchProcessProportion of missions in which HEMS coordinator dispatch alarm while HEMS resource is in flight; reaction time ≤ 2 minutes
      49EvaluationHEMS availability in the regionProcessOverview of availability of HEMS resources in the region; described by off-duty percentage; missions where HEMS is not available; and rejected missions due to weather, duty time, and simultaneous missions
      101EvaluationSatisfactory HEMS coordinator performanceProcess/outcomeIs the HEMS crew satisfied with HEMS coordinator performance throughout the mission?
      102Reaction phaseReaction time HEMS coordinatorProcessMeasures delay of HEMS coordinator from time of request for HEMS resource by local EMCC to HEMS resource receives alarm call
      102dReaction phaseOther factors than proximity in decision on which HEMS resource to dispatch. If yes, which?ProcessMeasures frequency of other considerations than proximity when choosing HEMS resource for a mission
      105EvaluationProportion of missions executed by extraregional HEMS resourceProcessMeasures how often and to which areas extraregional HEMS resources are used to solve missions in our region
      106AllProportion of missions accepted while engaged in other missionProcessMeasures share of cancelled missions due to another simultaneous mission with higher priority
      107EvaluationSatisfactory flight following by HEMS crewProcess/outcomeIs HEMS coordinator satisfied with HEMS crew flight following throughout the mission?
      EMCC = emergency medical coordination center; HCM = helicopter emergency medical services crewmember; HEMS = helicopter emergency medical services; RHA = regional health authority.
      The table shows the 14 indicators which reached a consensus, defined by a median score of 5 or more. Each indicator's mission phase, QI category, and definition is shown.

      Step 1

      An information letter was sent to the expert panel participants defining the scope of HEMS coordination and flight following together with information on the Delphi method and QI theory. The participants also received an individual and anonymized Excel (Microsoft, Redmond, WA) sheet with a list of QIs suggested by the study group for several defined phases of an HEMS mission. The study group suggested 16 QIs in the predefined Excel sheet. We asked the expert panel to suggest further QIs if they assessed the list of QIs to be incomprehensive. After receiving all responses, the study group revised the list of suggested indicators. The purpose of this revision was to redact all individual responses into a coherent list of suggestions. Thus, QIs with similar or identical meaning were merged; however, no content of the suggested QIs was excluded by the study group.

      Step 2

      The experts were asked to score the QIs in the revised spreadsheet using a 6-point Likert scale. One point described the indicator as not suitable, and a 6-point score described the indicator as very well suitable. Before reaching a score of the suggested indicators, the experts were asked to assess feasibility, rankability, actionability, and variability of each indicator. Based on the previous experience of Madsen et al,
      • Madsen MM
      • Eiset AH
      • Mackenhauer J
      • et al.
      Selection of quality indicators for hospital-based emergency care in Denmark, informed by a modified-Delphi process.
      we defined that a QI had to reach a median score of 5 or more in the last step to reach a consensus in the expert panel. In addition, QIs with a mean score of 2 points or less would be excluded from the further process. The median, mean, minimum, and maximum scores for each indicator were calculated.

      Step 3

      In step 3, the experts had their individual spreadsheet with their personal QI scores returned from the study group. In addition, the spreadsheets specified the median and mean value scores for all QIs scored by the expert panel in step 2. Thus, the experts could compare their own scores from step 2 with the median and mean scores for each indicator from the expert panel, thereby providing an indirect feedback of their own personal scores compared with the expert peers. The spreadsheets also included the minimum and maximum scores for each indicator to illustrate the range of the panel scores in step 2. Given this new information, the experts scored each QI again using the 6-point Likert scale. New median, mean, minimum, and maximum scores were calculated from the returned spreadsheets.

      Comments

      In addition to following the rigorous setup of the spreadsheets, we also allowed for general and elaborating comments on each QI in all 3 steps. The comments were made available for the expert panel in steps 2 and 3 and the study group after step 3, thus providing a possibility for a discussion in absence of physical meetings in the Delphi process.

      Results

      The literature search identified 165 articles results that to some degree related to quality management within HEMS coordination and flight following. One article described a continuous quality improvement system regarding flight following at Eastcare in Eastern Carolina.
      • Benson NH
      • Hunt RC
      • Tolson J
      • et al.
      Improved flight following through continuous quality improvement.
      Failing to comply with flight following procedures at takeoff and landing as well as flight following intervals exceeding 15 minutes were the main challenges in the described service. Published articles containing key performance indicators for EMCCs were related to specific diagnosis or time aspects of EMS operations. We found no relevant QIs for HEMS coordination and flight following in any of these identified articles.
      Pertaining to the Delphi process, all participants returned their Excel sheets from step 1. We received 61 QI proposals (Fig. 2). Upon revision of the proposals by the study group, 42 QIs were included for step 2. None of the proposed QIs achieved a mean score of 2 or less, defined as the exclusion criterion in steps 2 and 3. All participants returned their spreadsheets in both steps 2 and 3. After completion of step 2, 13 proposed QIs had reached a median score of 5 or higher, thus fulfilling the consensus criterion. The score range was wide for most of the proposed QIs; there was a 5-point range in 9 (21%), a 4-point range in 23 (55%), a 3-point range in 8 (19%) and a 2-point range in 2 (5%) of the 42 suggested indicators in step 2. Before step 3, the expert panel received further instructions on how to evaluate the proposed QIs in order to reduce task bias (ie, when individuals in a group solve the task in a way that deviates from the instructions given). The score range in step 3 was a 5-point range in 22 (52%), a 4-point range in 14 (33%), a 3-point range in 4 (10%), a 2-point range in 1 (2%), and a 1-point range in 1 (2%) of the 42 suggested indicators in step 3 (Fig. 3). Fourteen QIs reached a consensus in step 3 (Table 2).
      Figure 2
      Figure 2The Delphi process flowchart. New indicators were added according to the experts’ comments and suggestions at each stage. Similar suggestions and indicators were merged.
      Figure 3
      Figure 3The score distribution of each QI. The figure shows a stacked score distribution of the 6-point Likert scale as a bar for each indicator, with scores 5 and 6 in the positive range (blue) and scores under 5 in the negative range (red-yellow). A median score ≥ 5 is shown as right bar > left bar. The number on the x-axis expresses the number of experts scoring each indicator. The figure provides a visual assessment of the score range and whether a consensus was reached or not for each indicator (median score above or below five).
      Thirty-eight of the 42 QIs included in steps 2 and 3 were process indicators. Four were outcome indicators, but none of these reached a consensus. Ten of the 42 QIs were related to flight following, but only 2 reached a consensus. The remaining 32 QIs were related to HEMS coordination, of which 12 reached a consensus.

      Discussion

      This article presents a set of potential QIs designated for use in HEMS coordination and flight following. Using a Delphi consensus process, an expert panel developed and reached a consensus on 14 QIs, which mainly include process-type indicators and cover all phases of a typical HEMS mission.
      The operational setting described in this study consists of an integrated dispatch service to provide both HEMS coordination as well as flight following during missions. Both tasks are performed by specially trained health care personnel in an EMCC rather than ordinary flight following performed by ATC. This model might seem untraditional but provides some significant expected benefits because HEMS coordination can be suited into the operational requirements in a medical emergency trajectory. However, this way of organizing HEMS operations requires continuous governance. We argue that using a set of indicators to monitor the desired quality is necessary and feasible.
      The focus on QIs in health care is increasing.
      • Saver BG
      • Martin SA
      • Adler RN
      • et al.
      Care that matters: quality measurement and health care.
      The traditional way to use QIs includes finding structural, processual, and outcomes indicators, often related to certain emergency medical trajectories; measuring and monitoring them; and initiating corrective measures when desired.
      • Palmer RH
      Using health outcomes data to compare plans, networks and providers.
      However, when scientific evidence is lacking, QIs could be determined by an expert panel of health professionals in a consensus process based on their professional experience.
      • Mainz J
      Defining and classifying clinical indicators for quality improvement. Defining and classifying clinical indicators for quality improvement.
      A good quality assessment system consists of multiple indicators addressing different dimensions of quality. Moreover, Haugland et al
      • Haugland H
      • Olkinuora A
      • Rognås L
      • et al.
      Testing quality indicators and proposing benchmarks for physician-staffed emergency medical services: a prospective Nordic multicentre study.
      recently suggested some characteristics essential for QIs in order to make them as useful in clinical practice as possible. Most of our developed indicators were process indicators. Four of the suggested indicators reflected outcome (QI #45, 47, 54, and 55), but none of them reached a consensus. Two of the indicators (QI #101 and 107) that reached a consensus are bordering on being outcome indicators because they are addressing subjectively perceived outcome of flight following and coordination of each HEMS mission in the region. Good outcome indicators are difficult to find when applied to a system role rather than a treatment bundle. One could argue that outcome indicators could be expressed as the number of helicopter accidents in the HEMS system over a period of time or a number of missions, reflecting safety. Such an indicator would score well on feasibility, measurability, and rankability, but the incidence rate of accidents in a relatively small service such as the Norwegian HEMS would reduce the variability and thus the overall suitability of the indicator. Factors like the number of staff, the extent of education and training, and the framework for professional development are highly relevant but were not covered by the scope of this study. The theoretical framework suggested by Haugland et al is quite new, and the experts might not have been familiar with the concept. With that being said, we believe the suggested QIs cover areas that are important for a wide range of stakeholders.
      The wide score range noted in steps 2 and 3 possibly points out how differently the suggested indicators were perceived by the expert panel. The range of scores for each indicator was wide for most indicators, indicating a difference of opinion and perspective among the expert panel participants. As such, we believe the current set of QIs represents an important starting point for better quality improvement efforts in the field of the combined system of HEMS coordination and flight following. Ideally, the QIs should monitor how well the HEMS EMCC performs on HEMS coordination and flight following and give guidance on areas for improvement. Flight safety and patient safety should be considered the most important areas. We argue that at least some of the established QIs will give adequate monitoring on these 2 very important issues.
      QIs regarding HEMS is a field gaining momentum over the last few years. The EQUIPE project has identified a set of multidimentional QIs that are suitable for quality assurance and improvement related to HEMS.
      • Haugland H
      • Rehn M
      • Klepstad P
      • et al.
      Developing quality indicators for physician-staffed emergency medical services: a consensus process.
      We also see HEMS services adapting the EQUIPE indicators in evaluation of their system quality performance, and we expect more services to follow.
      • Alstrup K
      • Petersen J
      • Barfod C
      • et al.
      The Danish helicopter emergency medical service database: high quality data with great potential.
      Another example is the work done by the Commission on Accreditation of Medical Transport Systems.
      Commission on Accreditation of Medical Transport Systems
      Accreditation Standards.
      Our consideration is that quality assurance and improvement of HEMS dispatch, coordination, and flight following should emphasize the aspects concerning the facilitation of HEMS missions provided by air ambulance coordinators in our system rather than HEMS performance itself. Therefore, most currently published QIs and performance indicators do not fulfill our requirements.
      There are several strengths of this study. One is the diversity of participants, both in experience and perspective. Invitations to participate in the expert panel focused on a broad geographic and institutional representation rather than personal preference of who to invite. In addition, all the 24 participants completed the 3 steps in the process. This increases the completeness rate and increases the internal validity of the study. By using an anonymized, e-mail–based Delphi process, we minimized the possible bias from influence by dominant participants in the group. Using a median of 5 or higher, we assured that more than 50% of the participants assessed the indicators as being well suitable or very well suitable. In all expert panel methods, the selection of experts will affect the result, and experts should be selected with caution. Our final set of QIs are focused on HEMS coordination, and we suspect that flight following–related indicators gained less attention in the expert panel process. As such, our selection of experts might be unbalanced with respect to the number of health care experts versus flight operative experts. This might be the reason that only 2 of the 10 suggested QIs relating to flight following reached a consensus. In hindsight, it seems clear that we should have recruited experts reflecting both aspects more equally. However, because the scope of all HEMS systems is to provide emergency health care, a skewed focus toward the emergency health functions in HEMS coordination is acceptable.
      There are few scientific reports on this topic, and, as such, we had to design the theoretical framework and concept. This might have created some uncertainties for the experts. There may be limited external validity to our findings because of specific features of the Norwegian HEMS and EMCC systems. The fact that the score range was even wider after clarifying the scoring instructions makes it more plausible that the spread is caused by a real difference in the expert assessments rather that a systematic task bias by individuals. However, the latter cannot be ruled out. Finally, a set of QIs is of little value until they are documented and reported using appropriate methods. Therefore, systems for reporting must be established, and the compliance of reporting must be ensured.

      Conclusion

      We have applied a Delphi process method to develop QIs for HEMS coordination and flight following. An experienced and heterogeneous expert panel was used to suggest and reach a consensus on which QIs could be applied. Fourteen QIs reached the predefined limit of consensus.

      References

      1. World Health Organization. WHO Emergency care system framework, 2018. Available at: https://www.who.int/publications-detail/who-emergency-care-system-framework. Accessed June 30, 2020.

        • Taylor CB
        • Stevenson M
        • Jan S
        • et al.
        An investigation into the cost, coverage and activities of helicopter emergency medical services in the state of New South Wales, Australia.
        Injury. 2011; 42: 1088-1094
        • Reid BO
        • Rehn M
        • Uleberg O
        • et al.
        Physician-provided prehospital critical care, effect on patient physiology dynamics and on-scene time.
        Eur J Emerg Med. 2018; 25: 114-119
        • Røislien J
        • van den Berg PL
        • Lindner T
        • et al.
        Comparing population and incident data for optimal air ambulance base locations in Norway.
        Scand J Trauma Resusc Emerg Med. 2018; 26: 42
        • Haugland H
        • Uleberg O
        • Klepstad P
        • et al.
        Quality measurement in physician-staffed emergency medical services: a systematic literature review.
        Int J Qual Health Care. 2019; 31: 2-10
      2. European Aviation Safety Agency (EASA). Acceptable means of compliance (AMC) and guidance material (GM). Available at: https://www.easa.europa.eu/sites/default/files/dfu/Consolidated%20unofficial%20AMC%26GM_Annex%20V%20Part-SPA.pdf. Accessed June 30, 2020.

        • Norwegian Government
        Ministry of Health and Care Services. Central elements regarding the organization of emergency medical dispatch centrals [Sentrale elementer vedrørende organisering av AMK-Sentralene].
        Norwegian Government, Oslo, Norway2016 (Available at:)
        • Norwegian Government
        Ministry of Health and Care Services. First and foremost – a comprehensive system for dealing with acute diseases and injuries outside hospitals. NOU 2015:17. Norwegian Government.
        Norwegian Government, Oslo, Norway2015 (Available at:)
        • Norwegian Directorate of Health
        Learning for better emergency preparedness – the medical response to the terrorist incidents of 22 July 2011.IS-1984.
        Norwegian Directorate of Health, Oslo, Norway2012 (Available at:)
        • National Air Ambulance Service
        Evaluation report following the Oslo Government District Bombing and Utøya Island Shooting July 22, 2011.
        National Air Ambulance Service, Bodø, Norway2012 (Available at:)
        • Østerås Ø
        • Brattebø G
        • Heltne J-K
        Helicopter-based emergency medical services for a sparsely populated region: a study of 42,500 dispatches.
        Acta Anaesthesiol Scand. 2016; 60: 659-667
        • Norwegian Government
        Ministry of Health and Care Services. National Standard Requirements for rescuers in the air ambulance service, the rescue helicopter service and SAR – offshore. [Nasjonal standard for redningsmenn i luftambulansetjenesten, redningshelikoptertjenesten og SAR – offshore].
        Norwegian Government, Oslo, Norway2019 (Available at:)
        • Statistics Norway
        Population data.
        Statistics Norway, Oslo, Norway2020 (Available at:)
        • Central Norway Regional Health Authority
        Coordination of air ambulance helicopters in Central Norway [in Norwegian].
        Central Norway Regional Health Authority, Stjørdal, Norway2018 (Available at:)
        • Donabedian A
        The quality of care. How can it be assessed?.
        JAMA. 1988; 260: 1743-1748
        • Donabedian A
        Evaluating the quality of medical care.
        Milbank Q. 1966; 44: 166-206
        • Rubin HR
        • Pronovost P
        • Diette GB
        From a process of care to a measure: the development and testing of a quality indicator.
        Int J Qual Health Care. 2001; 13: 489-496
        • Haugland H
        • Olkinuora A
        • Rognås L
        • et al.
        Testing quality indicators and proposing benchmarks for physician-staffed emergency medical services: a prospective Nordic multicentre study.
        BMJ Open. 2019; 9e030626
        • Murphy MK
        • Black NA
        • Lamping DL
        • et al.
        Consensus development methods, and their use in clinical guideline development.
        Health Technol Assess. 1998; 2 (i-iv, 1-88)
        • Thangaratinam S
        • Redman CWE
        The Delphi technique.
        Obstet Gynaecol. 2005; 7: 120-125
        • Madsen MM
        • Eiset AH
        • Mackenhauer J
        • et al.
        Selection of quality indicators for hospital-based emergency care in Denmark, informed by a modified-Delphi process.
        Scand J Trauma Resusc Emerg Med. 2016; 24: 11
        • Benson NH
        • Hunt RC
        • Tolson J
        • et al.
        Improved flight following through continuous quality improvement.
        Air Med J. 1994; 13: 163-165
        • Saver BG
        • Martin SA
        • Adler RN
        • et al.
        Care that matters: quality measurement and health care.
        PLoS Med. 2015; 12e1001902
        • Palmer RH
        Using health outcomes data to compare plans, networks and providers.
        Int J Qual Health Care. 1998; 10: 477-483
        • Mainz J
        Defining and classifying clinical indicators for quality improvement. Defining and classifying clinical indicators for quality improvement.
        Int J Qual Health Care. 2003; 15: 523-530
        • Haugland H
        • Rehn M
        • Klepstad P
        • et al.
        Developing quality indicators for physician-staffed emergency medical services: a consensus process.
        Scand J Trauma Resusc Emerg Med. 2017; 25: 14
        • Alstrup K
        • Petersen J
        • Barfod C
        • et al.
        The Danish helicopter emergency medical service database: high quality data with great potential.
        Scand J Trauma Resusc Emerg Med. 2019; 27: 38
        • Commission on Accreditation of Medical Transport Systems
        Accreditation Standards.
        1st ed. CAMTS, Pfaeffikon, Switzerland2017 (Available at:)
        https://www.camtsglobal.org/standards-draft
        Date accessed: October 19, 2020