Advertisement

Development, Implementation, and Assessment of a New Competency and Outcomes-Based Orientation in an Air Medical Transport Program

Published:November 17, 2021DOI:https://doi.org/10.1016/j.amj.2021.07.009

      Abstract

      Objective

      The development and evaluation of new employees in air medical transport has historically lacked standardization and competency-based learning goals. Here we discuss the development, implementation, and assessment of a new competency-based education and evaluation method at Geisinger's Life Flight air medical transport service.

      Methods

      Using Bloom's taxonomy of learning, 14 competencies for flight employees were identified. An electronic database was created to track progress across competencies and serve as an information repository for the identification of goals and the development of individualized learning plans. Ten months after implementation of the new method, 11 preceptors and education team members were surveyed to understand their views on the new program.

      Results

      At the time of survey administration, 20 orientees had completed orientation under the new education and evaluation program in an average of 6.45 weeks, with a range of 3 to 10 weeks. Of the 11 surveyed instructors, 81.1% definitely agree that the new method adequately assesses performance compared with 45.5% with the previous unstandardized method; 81.8% of the instructors rated the overall change as very helpful.

      Conclusion

      The adoption of a competency-based learning model for air medical transport employee education and evaluation improves the assessment of performance and allows for the development of customized learning plans.
      There is a lack of standardization across the health care industry in training and evaluation of new employees during orientation. Nursing and emergency medical services are two professions in which orientation is critical but often unstandardized, creating challenges that impact the learner but also those conducting and evaluating the process. Within the field of nursing, although competency-based orientation has recently gained momentum in conjunction with online nursing education and nursing certification examinations, there is still a disconnect between theory and clinical practice.
      • Gravina EW.
      Competency-based education and its effect on nursing education: a literature review.
      ,
      • Hodges AL
      • Konicki AJ
      • Talley MH
      • et al.
      Competency-based education in transitioning nurse practitioner students from education into practice.
      Specific to emergency medical services, the standard national curriculum outlines competencies that professionals should meet, but these guidelines are broad, open to interpretation, and lack measurable outcomes over time.
      US Department of Transportation
      EMT-paramedic: NSC refresher curriculum.
      This lack of standardization and competency-based education impacts the educational quality and progress of emergency medical services orientees. At our own institution, orientation feedback was historically given every shift by an assigned preceptor in the areas of aircraft safety, aviation knowledge, patient interaction, provider interaction, protocol knowledge, clinical thinking and treatments, and overall performance. Each assessment category was rated on a Likert scale (1, very poor; 2, poor; 3, average; 4, above average; and 5, excellent). This method allowed for a qualitative reflection, but the exact meaning of the value was left open for interpretation by individual preceptors, leading to significant variation in interrater reliability. Analysis of our own orientation feedback demonstrated variation of up to 3 points for evaluation of the same assessment category for the same orientee (Appendix 1). This variation impacted the actionability of feedback and limited the customization of learning plans for new orientees.
      One sector of health care that successfully uses a competency-based education model to track education and training is resident medical education. In 2013, the Accreditation Council of Graduate Medical Education recognized that medical education lacked a framework for analyzing individual performance and took steps to mitigate the issue, resulting in a competency-based framework that was constructed and implemented across all disciplines of residency education. The Milestones Project further defined levels of performance for each competency.,
      • Powell DE
      • Carracio C.
      Toward competency-based medical education.
      This change has demonstrated success in physician competency development and customization of individual learning plans and advancement through medical education.
      • Powell DE
      • Carracio C.
      Toward competency-based medical education.
      Here we describe how an air medical critical care transport program developed and integrated a unique competency-based orientation with predefined levels to support the standardization of training and the customization of individualized learning plans.

      Methods

      Setting

      Geisinger's flight program has been in service since 1981 and celebrated its 40th anniversary in 2021. The program has completed over 60,000 air medical transports and is accredited by the Commission on Accreditation of Medical Transport Systems. The footprint of the program is diversified, covering a large portion of rural Pennsylvania. The program is categorized as a Part 135 hospital-based service and operates 6 air bases and 2 ground bases. There are a total of 9 aircraft (7 EC-145s and 2 BK 117s) in service, as well as 3 critical care ground ambulances. The medical crew configuration typically consists of a flight registered nurse and a flight paramedic. Postgraduate second- and third-year emergency medicine residents regularly comprise a third medical crewmember at 1 of the 6 bases each weekday. Prehospital emergency physicians occasionally replace the flight paramedic. There are specialty transport teams, including neonatal intensive care nurses and extracorporeal membrane oxygenation perfusionists. In all, there are approximately 100 medical personnel across the program's platform.

      Construction of a New Orientation Evaluation Method

      Using Bloom's taxonomy to highlight domain progression, the team was able to define 14 competencies for new orientees as seen in Table 1.
      • Armstrong P.
      Bloom's taxonomy.
      Each individual competency is designed with levels to progress from the lower stages of Bloom's taxonomy, such as knowledge and understanding, through the higher stages, such as application and synthesizing the learned information or skill.
      • Armstrong P.
      Bloom's taxonomy.
      Following Bloom's taxonomy guidance, each of the modules is scored from 1 through 4 to rate orientee performance and follow a progression in the development of the competency.
      Table 1Life Flight New Employee Orientation Competency Rubric
      Level 1Level 2Level 3Level 4
      1. Protocol knowledgeOrientee demonstrates, verbalizes, recalls, and recites 100% of PA state protocolsOrientee is able to verbally recite or demonstrate through practice (scenario/patients) 25% to 50% of the Life Flight protocolsOrientee is able to verbally recite or demonstrate through practice (scenario/patients) 50% to 75% of the Life Flight protocolsOrientee is able to verbally recite or demonstrate through practice (scenario/patients) greater than 75% of Life Flight protocols
      2. Aircraft/CCGT equipmentEasily locates and recites all contents of the primary bag and Propaq MD, Zoll Manufacturing Corporation, Pittsburgh, PA, USAA. Recites equipment in the pediatric bag B. Recalls location of medication pouch, pump bag, ACLS medications, and ventilator/bagAble to locate all equipment in the aircraft and can recall contents of the medication pouch, pump bag, ventilator supply bag, pediatric supply bag, arterial line kit, medication drawer, and pouchesA. Can function during flights when equipment is needed and can logistically place equipment during flight as needed for access B. Excellent knowledge of equipment locations
      3. Airway managementA. Completed OR rotation with positive feedback B. Can perform basic airway skills (jaw thrust, airway adjunct placement, BVM)

      C. Can state location of airway equipment
      A. Readily describes indications for placement of an advanced airway B. Can verbalize LF airway management policy C. Able to identify need for and use ETT cuff manometerA. Demonstrates proficiency in performing endotracheal intubation with both direct and video laryngoscopy B. Can identify patient with potential difficult airwayA. Demonstrates ability to successfully deploy adjunct airway devices on difficult intubations B. Can discern need for and demonstrate technique for emergent cricothyroidotomy
      4. Ventilator managementA. Can properly set up ventilator tubing, including filter B. Can turn on ventilator and perform a leak testA. Can verbalize initial ventilator settings for adult, pediatric, and neonatal populations B. Able to make ventilator adjustments to place patient on BiPap/CPAP policiesA. Assesses alarms and can perform necessary adjustments to clear alarms and ensure proper ventilator functionA. Identifies patients who need ventilator adjustments dependent on clinical condition B. Can execute necessary changes to ventilator dependent on patient assessment
      5. Aircraft/CCGT operationsA. Completed ASAT training B. Understands cockpit communications (helmet microphone, Vox, radio etiquette)A. Demonstrates safe operations around aircraft B. Effectively ingress and egress aircraft C. Competent during aircraft operations (rebuckles seat belts, secures all equipment)A. Operates doors, completes preflight walk around, and can manipulate litter mount without difficulty B. Safely operates during "hot" operationsA. Capable to input GPS coordinates for scene call and hospital identifiers B. Operates cockpit radios
      6. Propaq MD operationsA. Can turn on monitor B. Easily attaches 4-lead ECG C. Can cycle BP D. Acquires snapshot E. Changes paperA. Able to initiate ETCO2 monitoring B. Can change patient settings based on age C. Troubleshoots alarms D. Initiates temperature monitoringA. Demonstrates ability to initiate invasive monitoring (arterial line, CVP, etc.) B. Displays ability to cardiovert, pace, and defibrillateA. Demonstrates competency with all aspects of the Propaq monitor B. Changes setting and correlates waveforms to patient assessment (every 5 min BP, decreased ETCO2 waveform)
      7. Braun infusion pump operationsA. Easily turns on pump B. Checks battery level C. Able to assemble proper equipment (pump dependent)A. Demonstrates ability to draw uA; shows ability to infuse a bolus

      B. Demonstrates ability to run a timed infusion C. Displays ability to edit pump settings, and medications B. Proficient with loading medication into the pump C. Displays ability to begin a basic rate infusion
      A. Shows ability to infuse a bolus B. Demonstrates ability to run a timed infusion C. Displays ability to edit pump settingsA. Determines the proper infusion (mL/h, cg/kg/min, etc.)

      B. Able to use the drug library for dosing

      C. Selects correct concentrations D. Demonstrates ability to use dose rate calculator special functions
      8. Life Flight medicationsA. Up-to-date with all medications on the PA state protocols B. Identifies indications, contraindication, mechanism of action for those medicationsA. Identifies medications on the LF list by both protocol and location B. Basic pharmacologic understanding of frequently used medications (ASA, narcotics, benzo, etc.)A. Recognizes indications for medications in certain patient populations (ie, norepinephrine–sepsis, labetalol–CVA, mannitol–high ICP)A. Applies critical thinking to effectively manage infusions and titrate medications based on patient presentation (ie, changes vasoactive rate to increase BP, titrates propofol for adequate sedation)
      9. Isolette/MVP 10 operationsA. Turns on monitor

      B. Checks gas levels/changes tanks C. Demonstrates knowledge of supplies (ECG, pulse oximetry, oxyhood, etc.)
      A. Connects ventilator circuit B. Can set up and monitor oxygen % with oxyhood C. Attaches monitor and changes settings, properly routes cables into isoletteA. Demonstrates programming ventilator settings (rate, IPAP, EPAP, Fio2) B. Able to program CPAP settings

      C. Sets pressure relief valve
      A. Manages ventilator settings based on patient assessment B. Troubleshoots MVP alarms C. Can easily arrange infant in isolette, can route wires, vent tubing, etc., change from wall gas to tank source
      10. PharmacologyA. States correct doses for ACLS/PALS medications (epinephrine 1:10,000, amiodarone 300 mg vs. 150 mg, adenosine 6 mg, 12 mg, etc.) B. Can verbalize the indications for different classifications of medications (pressors, beta blockers, analgesics, antiplatelets, etc.)A. Calculates and verbally reports proper doses for all RSI medications including sedation and analgesia (ie, etomidate 0.3 mg × weight, fentanyl 1 μg/kg) without use of calculatorA. Verbalizes contraindications to protocolized medications (ie, no succinylcholine with burn/hyperkalemia, metoprolol/ntg with IWMI) B. States the appropriate need for medication titration based on patient assessment (increase pressor dose, redose analgesia, titrate to sedation level)A. When presented with a patient (actual or scenario based) considers Life Flight medication availability to select the appropriate agent based on mechanism of action, intended effect, and anticipates potential adverse side effects; verbalizes proper choices
      11. Patient assessmentA. Recognizes and verbalizes when a patient is stable, unstable, or critical (based on scenarios and/or actual transports) B. Demonstrates the ability to perform head to toe physical examination on patientsA. Effectively listens to patient report from referring provider and records information on the hand off form B. Accurately documents pertinent findings (HPI, PMH, all, labs/studies)A. Verbalizes alteration in patient condition from normal based on assessment findings and disease process or injuries B. Prioritizes relevant information and verbalizes finding on assessment which are concerning or altered from normal)A. Analyzes ascertained medical information to formulate a potential diagnosis and develop a plan of care based on those findings; discusses POC and differential diagnosis with preceptor
      12. Analgesia and sedation managementA. Discusses with crewmembers the need to provide analgesia and/or sedation to patient B. Can verbalize both the appropriate medication and appropriate dose of selected medicationA. Verbalizes the potential side effects and contraindications of medication selectedA. Demonstrates the ability to implement an appropriate pain scale to initially assess and reassess pain level before and after analgesia administration B. Documents in EPIC GMC approved pain scale-initial, after any analgesia administration and at time of handoffA. Verbalizes patients in need of additional therapies if initial measures are unsuccessful (such as additional doses of analgesia, communication with medical command for additional medications) B. Verbalizes technique to monitor sedation/analgesia side effects (respiratory rate, ETCO2, LOC)
      13. Clinical patient managementA. Demonstrates organized history taking to include HPI, medications, HX, all; recognition of abnormal vital signs

      B. Incorporates patient physical examination into decision making C. Does not consistently use secondary sources of information to arrive at patient diagnosis D. Difficulty arriving at differential diagnosis or patient management plan
      A. Displays full examination including level 1 (A and B) and includes secondary sources (referring facility labs, medications, imaging) during the decision-making process B. Discusses potential patient problem with partner and/or medical command and frequently requires assistance to discern patient's differential diagnosis C. Able to verbalize patient management plan but requires assistance on a continued basisA. Demonstrates proficiency regarding information gathering and prioritizes a management plan for critically ill/injured patients B. Requires minimal assistance in developing a plan of care for common patient presentations C. Verbalizes when patient diagnosis or problem is emergent vs. urgent and can adapt care based on informationA. Manages patients with multiple diagnoses based on examination and data synthesis B. Identifies aspects of assessment and data that may be rarely seen C. Systematically and consistently integrates an approach to patient care in team setting for critically ill and complicated patient presentations
      14. Demonstrates professionalism and accountability and incorporates C.I.CARE dailyYes/no
      ACLS = Advanced Cardiac Life Support; ASA = aspirin; BiPap = bilevel positive airway pressure; BP = blood pressure; BVM = bag valve mask; CCGT = Critical Care Ground Truck; C.I.CARE = care, introduction of self, ask/anticipate, respond, end with excellence, communicate, ; CPAP = continuous positive airway pressure; CVA = cerebrovascular accident; CVP = central venous pressure ; ECG = electrocardiogram; EPAP = expiratory positive airway pressure; EPIC GMC = EPIC Geisinger Medical Center electronic health record; ETT = endotracheal tube; HPI = history of pertinent illness; HX = history; ICP = intracranial pressure; IPAP = inspiratory positive airway pressure; LF = Life Flight; LOC = ;level of consciousness; MVP = pediatric/neonatal ventilator; ntg = nitroglycerin; OR = operating room ; PALS = Pediatric Advanced Life Support; PMH = past medical history; RSI = rapid sequence intubation; uA = Change to: up medication.
      To reach consensus on scoring interpretation, the education team compared scores achieved by assessing current employees and comparing their results. An experienced 20-year flight nurse achieved a level 4 on each competency, whereas a flight nurse with 5 years of experience achieved levels between 2.5 and 3.5 on each competency. After this thorough review, the education team concluded that the basic operating level a new employee needed to successfully complete orientation aligned with the objective criteria that were achieved at the second level of each of the 14 competencies.

      Data Collection and Learning Plan Customization

      With the advent of a new organizational employee electronic evaluation system, an electronic format for these new evaluation forms was implemented. Several inherent flaws in a paper evaluation system that impeded data collection such as loss of forms or lack of submission were eliminated. For this study, data were able to be collected electronically in real time (every shift) and were viewable by each member of the education team and the orientee themselves. This was accomplished using a Microsoft (Redmond, WA) Access database.
      At the beginning of each shift, the orientee and preceptor discuss daily goals and objectives and enter them into the database. During the shift, these goals are addressed and either completed or not completed. Near the end of the shift, the orientee and preceptor determine if goals were met and then indicate the status in the database. Next, they review each of the 14 competencies to assess that shift's achievement level (1-4 or not applicable if not addressed during the shift). Patient encounters, patient type, and additional pertinent transport information is entered into the database. Once the entry is completed and closed, an e-mail is autogenerated and forwarded to the clinical nurse educator and paramedic educator with tabulated data.
      At the conclusion of each week of orientation, performance measures are averaged and autotracked via a spreadsheet to evaluate orientee progress through each of the 14 competencies. The averages for each week are calculated as well as the overall averages across each specific competency (Appendix 2).
      To assess the functionality of the new program, 11 preceptors and education team members were anonymously polled using a 6-question electronic survey. This electronic survey was completed at 10 months postimplementation of the new competency evaluation system. The survey asked respondents to report their views regarding the prior and new evaluation method, assessing the following areas: the ability of the method to adequately assess performance, the ability to target knowledge deficits, and the amount of performance information collected by each method. Additionally, respondents rated the overall helpfulness of the new program. Surveys were administered using SurveyMonkey, Momentive Inc., San Mateo, CA, US.

      Results

      At the time of the survey, a total of 20 orientees have completed orientation under the new evaluation program. Each of these employees has demonstrated progression through the modules with an average time frame of 6.45 weeks. Two staff had been prior employees returning to the program who progressed through orientation at 3 weeks each. There were 2 employees identified early to have educational needs and required 10 weeks to reach the required minimum numbers as well as demonstration of proficiency.
      All 11 preceptors and educational team members surveyed on their opinion of the new competency-based education and evaluation method responded to the survey. As seen in Table 2, 81.82% of the surveyed preceptors and members of the education team definitely agreed that the competency-based education and evaluation method adequately assessed performance compared with 45.45% for the prior Likert scale–based method; 63.64% agreed that the new method adequately targeted knowledge deficits compared with 9.09% for the old method.
      Table 2The Assessment of Preceptor and Education Team Perceptions of New Training and Evaluation Method
      ItemNot At All (%)Somewhat (%)Definitely (%)
      Prior method adequately assessed performance9.0945.4545.45
      New method adequately assesses performance018.1881.82
      Prior method adequately targeted knowledge deficits45.4545.459.09
      New method adequately targets knowledge deficits036.3663.64
      ItemPrevious (%)Current (%)
      As a preceptor, which method provides better performance information0100
      ItemNot Helpful (%)Somewhat Helpful (%)Extremely Helpful (%)
      Rate the overall change in orientation assessment from the old system to the new system9.099.0981.82
      One hundred percent of preceptors indicated that the current method provides better performance indication, and 81.82% of those surveyed suggested that the change from the Likert scale–based system to the competency-based education and evaluation method has been extremely helpful.

      Discussion

      Preceptors and members of the education team surveyed indicated the change in education and evaluation model to a unique competency-based method was extremely beneficial. In addition to providing a more standardized evaluation framework, this change allowed for the development of customized learning plans for the new orientees to address personal knowledge gaps. When it was identified that they had already achieved the benchmark for a particular competency, their learning goals were shifted to focus on competencies that needed additional work. Furthermore, when an orientee was assigned to a new preceptor, they did not have to repeat skills and procedures in which they had already demonstrated competency.
      The use of milestones or levels established a credible time line for orientation completion based on the progress made and, more importantly, provided quantitative rather than qualitative feedback. Providing quantitative feedback to preceptors has been demonstrated to better support the identification of areas for improvement.
      • Swihart D.
      How will a unit-specific competency-based orientation program benefit the new nursing staff coming to my unit?.
      Additionally, this process allowed for early identification of a lack of progress and gave preceptors and the education team ample time for remediation. Finally, this new method allowed for increased transparency and open communication across all parties.
      Competency-based education models have demonstrated success in resident medical education, enabling customized learning plans and individualized progression through training.
      • Powell DE
      • Carracio C.
      Toward competency-based medical education.
      This type of training and evaluation method has also been gaining traction in nursing education.
      • Gravina EW.
      Competency-based education and its effect on nursing education: a literature review.
      ,
      • Hodges AL
      • Konicki AJ
      • Talley MH
      • et al.
      Competency-based education in transitioning nurse practitioner students from education into practice.
      Here we demonstrate that this type of model can be successfully applied to emergency medical services orientation, enabling improvement in performance assessment and knowledge deficit remediation.
      There are several limitations to this study. For one, this study reviewed the implementation and assessment of a competency-based education model for emergency medical services at 1 institution. Additionally, because this model has recently been implemented, quality-based outcomes related to care delivery for orientees who trained under this model are not yet available. Nevertheless, this study proposes a competency-based education model that can be applied at other institutions with flight programs to support better assessment of performance and remediation of knowledge and/or skill deficits.
      Future directions include the comparison of senior program staff to new staff; interval competency assessment of all staff; and postorientation assessment at 1 year to test for skill, protocol knowledge, and procedure knowledge retention. Orientation methods such as this can be tailored to meet the needs of many other facets of health care. Nursing and emergency medical services could very well reap the benefits of executing such a program. Early identification of learning needs and knowledge deficits, focusing on recognized gaps rather than on already attained goals, and concrete measured data have proven their benefit for this program.

      Appendix. Supplementary materials

      References

        • Gravina EW.
        Competency-based education and its effect on nursing education: a literature review.
        Teach Learn Nurs. 2017; 12 (Available at:) (Accessed September 24, 2017): 17-121
        • Hodges AL
        • Konicki AJ
        • Talley MH
        • et al.
        Competency-based education in transitioning nurse practitioner students from education into practice.
        J Am Assoc Nurse Pract. 2019; 31: 675-682
        • US Department of Transportation
        EMT-paramedic: NSC refresher curriculum.
        2001 (Available at:) (Accessed September 24, 2017)
        • Holmboe E
        • Edgar L
        • Hamstra S.
        The milestones guidebook.
        2016 (1-41. Available at:) (Accessed September 21, 2017)
        • Powell DE
        • Carracio C.
        Toward competency-based medical education.
        N Engl J Med. 2018; 378: 3-5
        • Armstrong P.
        Bloom's taxonomy.
        Vanderbilt University Center for Teaching, 2017 (Available at:) (Accessed September 21, 2017)
        • University of Central Florida
        Bloom's taxonomy.
        2017 (Available at:) (Accessed October 5, 2017)
        • Swihart D.
        How will a unit-specific competency-based orientation program benefit the new nursing staff coming to my unit?.
        Strategies for Nurse Managers, Inc, 2010 (Available at:) (Published FebruaryAccessed September 22, 2017)