Designing and implementing full immersion simulation as a research tool

Open AccessPublished:February 22, 2016DOI:https://doi.org/10.1016/j.aenj.2016.01.001

      Summary

      Simulation is a valuable research tool used to evaluate the clinical performance of devices, people and systems. The simulated setting may address concerns unique to complex clinical environments such as the Emergency Department, which make the conduct of research challenging. There is limited evidence available to inform the development of simulated clinical scenarios for the purpose of evaluating practice in research studies, with the majority of literature focused on designing simulated clinical scenarios for education and training. Distinct differences exist in scenario design when implemented in education compared with use in clinical research studies. Simulated scenarios used to assess practice in clinical research must not comprise of any purposeful or planned teaching and be developed with a high degree of validity and reliability. A new scenario design template was devised to develop two standardised simulated clinical scenarios for the evaluation of a new assessment framework for emergency nurses. The scenario development and validation processes undertaken are described and provide an evidence-informed guide to scenario development for future clinical research studies.

      Keywords

       What is known

      • Simulation is widely used as an education modality in healthcare, due to its effectiveness in improving learning outcomes and patient care.
      • Full immersion simulation serves as a valuable clinical research tool used to evaluate the clinical performance of devices, people and systems.
      • The literature on simulated design is focused on education and training with limited evidence on the development of simulated clinical scenarios for evaluating clinical practice in research studies.

       What this paper adds?

      • Highlights the differences in scenario design when implemented in education compared with use in clinical research studies.
      • A new scenario design template for the evaluation of clinical performance in clinical research studies.
      • An evidence-informed development and validation process to guide the design of valid and reliable clinical simulated scenarios for use in research.

      Introduction

      Over the past few decades, simulation has become increasingly popular as an education modality in healthcare. Simulation involves the use of a manikin or actor to represent a clinical situation.
      • Jeffries P.
      Using Simulation in Nursing Education. Simulation in Nursing Education: From Conceptualization to Evaluation.
      There has been considerable focus placed on studies about simulation, with studies reporting its effectiveness in improving educational outcomes and patient care.
      • McGaghie W.C.
      • Issenberg B.
      • Petrusa E.R.
      • Scalese R.J.
      A critical review of simulation-based medical education research: 2003−2009.
      • Schmidt E.
      • Goldhaber-Fiebert S.
      • Ho L.
      • McDonald K.
      Simulation exercises as a patient safety strategy.
      • Zendejas B.
      • Brydges R.
      • Wang A.
      • Cook D.
      Patient outcomes in simulation-based medical education: a systematic review.
      The usefulness of simulation is not limited to education, but also serves as a valuable clinical research tool that may be used to investigate other research questions. Simulation enables the study of various aspects of clinical practice that may not otherwise be measureable and conditions may be controlled significantly more in the simulated setting.
      • Dieckmann P.
      • Phero J.C.
      • Issenberg S.B.
      • Kardong-Edgren S.
      • Ostergaard D.
      • Ringsted C.
      The first research consensus Summit of the society for simulation in healthcare: conduction and a synthesis of the results.
      There is also the added benefit of ensuring no harm to patients, resolving ethical constraints of the clinical environment.
      • LeBlanc V.R.
      • Manser T.
      • Weinger M.B.
      • Musson D.
      • Kutzin J.
      • Howard S.K.
      The study of factors affecting human and systems performance in healthcare using simulation.
      Various studies have reported using simulation to assess device performance,
      • Vardi A.
      • Berkenstadt H.
      • Levin I.
      • Bentencur A.
      • Ziv A.
      Intraosseous vascular access in the treatment of chemical warfare casualties assessed by advanced simulation: proposed alteration of treatment protocol.
      technical and non-technical skills of clinicians
      • Gaba D.
      • Howard S.
      • Flanagan B.
      • Smith B.
      • Fish K.
      • Botney R.
      Assessment of clinical performance during simulated crises using both technical and behavioral ratings.
      • Wright M.C.
      • Segall N.
      • Hobbs G.
      • Phillips-Bute B.
      • Maynard L.
      • Taekman J.M.
      Standardized assessment for evaluation of team skills: validity and feasibility.
      • Fletcher G.
      • Flin R.
      • McGeorge P.
      • Glavin R.
      • Maran N.
      • Patey R.
      Anaesthetists’ non-technical skills (ANTS): evaluation of a behavioural marker system.
      • Yule S.
      • Flin R.
      • Maran N.
      • Rowley D.
      • Youngson G.
      • Paterson-Brown S.
      Surgeons’ non-technical skills in the operating room: reliability testing of the NOTSS behavior rating system.
      and human factors affecting clinical performance.
      • Howard S.
      • Gaba D.
      • Smith B.
      • Weinger M.
      • Herndon C.
      • Keshavacharya S.
      • et al.
      Simulation study of rested versus sleep-deprived anesthesiologists.
      • Merry A.
      • Weller J.
      • Robinson B.
      • Warman G.
      • Davies E.
      • Shaw J.
      • et al.
      A simulation design for research evaluating safety innovations in anaesthesia.
      Simulation has also been used to discover potential threats to patient safety such as environmental factors.
      • Hamman W.
      • Beaudin-Seiler B.
      • Beaubien J.
      • Gullickson A.
      • Orizondo-Korotko K.
      • Gross A.
      • et al.
      Using in situ simulation to identify and resolve latent environmental threats to patient safety: case study involving operational changes in a labor and delivery ward.
      The simulated setting may be used to address factors unique to complex clinical environments such as the Emergency Department (ED), which make the conduct of clinical research studies challenging. In the ED, priority must be given to the most urgent patients who present seeking treatment and the delivery of care cannot be influenced by their eligibility for research participation.
      • Worster A.
      • Rowe B.
      • Stiell I.
      • Borgundvaag B.
      • Sivilotti M.
      • Campbell S.
      • et al.
      Clinical research in the emergency department conducted by non-emergency physicians: potential problems and proposed recommendations.
      In addition, emergency clinicians often do not have the time or resources to assist with research activities.
      • Cofield S.
      • Conwit R.
      • Barsan W.
      • Quinn J.
      Recruitment and retention of patients into emergency medicine clinical trials.
      Studies which involve the consumption of ED personnel and resources are often unsuccessful.
      • Worster A.
      • Rowe B.
      • Stiell I.
      • Borgundvaag B.
      • Sivilotti M.
      • Campbell S.
      • et al.
      Clinical research in the emergency department conducted by non-emergency physicians: potential problems and proposed recommendations.
      Simulated clinical scenarios may be designed to reflect the ED setting and used to assess the performance of emergency clinicians without increasing clinician workloads or impeding on patient care.
      The design of simulated clinical scenarios is complex and encompasses a wide range of components to replicate the clinical setting. There are a variety of modalities that may be employed including: part-task trainers designed to provide experience in specific skills;
      • Littlewood K.E.
      High fidelity simulation as a research tool.
      standardised patients which are individuals carefully trained to accurately and consistently role-play a patient with a health concern;
      • Nestel D.
      • Clark S.
      • Tabak D.
      • Ashwell V.
      • Muir E.
      • Paraskevas P.
      • et al.
      Defining responsibilities of simulated patients in medical education.
      and full-body simulators which are computer driven-mannequins with varying levels of fidelity.
      • Littlewood K.E.
      High fidelity simulation as a research tool.
      Literature relating to scenario design appears mostly focused on the development of simulated clinical scenarios for education and training purposes, with little discussion on how to design valid and reliable simulated clinical scenarios for clinical research studies. A systematic evidence-informed approach is needed to develop valid and reliable simulated clinical scenarios to be used as an evaluation method in research studies.
      This paper highlights the differences in the design of simulated clinical scenarios when implemented as an investigative research tool to assess clinical performance compared with education and training. The importance of achieving validity and reliability when designing simulated clinical scenarios for research studies is presented and existing approaches to simulation design are reviewed. We present an evidence-informed approach to simulation design developed to evaluate a new emergency nursing assessment framework which may be used to guide scenario development in future clinical research studies.

      Research versus education design features

      Distinct differences exist between the design of simulated scenarios for clinical research studies compared with education and training. In a teaching and learning context, simulation may be used to facilitate learning and assess clinical competence which usually involves instruction and participant feedback as part of the overall simulation-based learning experience.
      • Gaba D.
      The future vision of simulation in health care.
      When using simulation as a research tool to evaluate participant's behaviours and experiences, the simulation experience must not contain instruction as this may confound study findings. The key differences in designing simulated clinical scenarios for use in teaching and learning versus assessing participant performance in a research setting are outlined in Table 1. The differences are presented using the design constructs as identified in the NLN/Jeffries Simulation Framework including: facilitator; participant; simulation design characteristics; educational practices and outcomes.
      • Jeffries P.R.
      • Rogers K.
      Theoretical framework for simulation design.
      • Durham C.F.
      • Cato M.L.
      • Lasater K.
      NLN/Jeffries simulation framework state of the science project: participant construct.
      The NLN/Jeffries Simulation Framework is a theoretical model intended to inform simulation design for education and training and provides a consistent method for the development and implementation of simulation experiences.
      • Jeffries P.R.
      A framework for designing, implementing, and evaluating simulations Used as teaching strategies in nursing.
      Table 1Characteristics of simulation design when implemented in education and training compared with clinical research studies designed to evaluate clinical performance, using NLN/Jeffries Simulation Framework Constructs
      • Jeffries P.R.
      • Rogers K.
      Theoretical framework for simulation design.
      • McGaghie W.C.
      • Issenberg B.
      • Petrusa E.R.
      • Scalese R.J.
      A critical review of simulation-based medical education research: 2003−2009.
      • Jeffries P.
      Using Simulation in Nursing Education. Simulation in Nursing Education: From Conceptualization to Evaluation.
      ConstructSimulation Framework ExplanationEducation and trainingClinical research studies
      Participant/sIndividuals or groups who partake in the simulation. Characteristics and elements of participant construct include roles and responsibilities, demographics, attributes and values.
      Roles and responsibilitiesStudentsStudy subjects
      DemographicsIdentifies learning needs of studentsDescribes study sample
      Attributes (preparation for simulation, learning styles, motivation, perceptions, anxiety, self-efficacy and vulnerability)Affects how students learnStudy outcomes
      Values (participants’ expectations of the simulation to add to their learning).Informs teaching methodsNot applicable
      FacilitatorEducator responsible for evaluating and teaching participantsTeaches and evaluates studentsEvaluates study subjects/governs scenario operation
      Educational practicesThe processes involved in learning, comprising of active learning, feedback, learning styles and collaborationPromote student learningShould not be implemented to prevent contamination of study findings
      OutcomesKnowledge, skill performance, satisfaction, critical thinking and self-confidence assessed during simulationLearning outcomesStudy outcomes
      Simulation Design CharacteristicsDesign features of simulation including objectives, fidelity, problem solving, student support and cues, and reflective thinking.
      ObjectivesLearning objectivesResearch question/hypothesis
      Fidelity (realism)Must enable development or demonstrate the knowledge or skill the simulation is designed to assess or teachA high degree of fidelity is necessary to optimise validity
      Problem solving (level of complexity in the simulation)Must develop or demonstrate the knowledge or skill the scenario is designed to teach or assessMust reflect what takes place in the clinical setting and facilitate measurement of outcome variables necessary to answer research question
      Student support and cuesMust meet individual learner needs to progress through the scenarioAre necessary for scenario to progress, however must be limited to reflect real life cues to validly measure study outcomes. Must be consistently replicated each time scenario repeated.
      Reflective thinking (debriefing)Conducted in groups to promote learning in all involvedConducted as individuals or in groups to gain insights into individual or group views, experiences, beliefs and motivations

      Evaluation of the new emergency nursing assessment framework ‘HIRAID’

      HIRAID (History; Identify Red flags; Assessment; Interventions; and Diagnostics; reassessment and communication) is an evidence-informed systematic approach developed to guide the initial comprehensive nursing assessment of Emergency Department (ED) patients following triage.
      • Munroe B.
      • Curtis K.
      • Margerat M.
      • Strachan L.
      • Buckley T.
      HIRAID: an evidence-informed emergency nursing assessment framework.
      HIRAID and the associated training course is informed by a strong theoretical foundation, expert opinion, adult learning principles and the available clinical assessment research.
      • Munroe B.
      • Curtis K.
      • Margerat M.
      • Strachan L.
      • Buckley T.
      HIRAID: an evidence-informed emergency nursing assessment framework.
      Prior to the implementation of any new model of care into clinical practice, it should be tested.
      • Graham I.D.
      • Logan J.
      • Harrison M.B.
      • Harrison M.B.
      • Straus S.E.
      • Tetroe J.
      • et al.
      Lost in knowledge translation: time for a map?.
      A research study was designed to test the feasibility and effectiveness of the HIRAID model in advancing early career emergency nurses’ technical and non-technical patient assessment skills and human factors affecting assessment performance, using full immersion high-fidelity patient simulation. Valid and reliable-simulated clinical scenarios needed to be developed for this evaluation. Two high-fidelity simulated clinical scenarios (A & B) were devised to test the generalisability of the HIRAID intervention and participants were randomised to scenario A or B pre-intervention, and the alternate scenario for assessment post-intervention.

      Validity and reliability of simulated clinical scenarios

      Clinical research studies using simulation to measure participant behaviours and skills need to have a high degree of validity and reliability. Validity refers to the degree in which an instrument measures, or in the case of simulation a simulated clinical scenario, is representative of what it is supposed to reflect and measure.
      • Polit D.
      • Beck C.
      Measurement and Data Quality. Essentials of Nursing Research: appraising evidence for nursing practice.
      In the simulated environment validity is achieved through ensuring simulation features closely reflect the real world clinical setting so that findings may be applied to clinical practice.
      • Cioffi J.
      Clinical simulations: development and validation.
      A high degree of fidelity or realism is necessary to assist participants to engage in simulated scenarios and respond to the situation as they would in the actual clinical setting.
      • Gormley G.
      • Sterling M.
      • Menary A.
      • McKeown G.
      Keeping it real! Enhancing realism in standardised patient OSCE stations.
      Failure to identify a relationship between the clinical and simulated environment has been suggested to be an inadequacy of the original research design and indicates a need to refine scenario script to more closely reflect the clinical setting.
      • Kirlik A.
      Brunswikian theory and method as a foundation for simulation-based research on clinical judgment.
      The use of real clinical cases to inform simulated scenario development are recommended to ensure the simulation experience accurately portrays patients in the real world.
      • Black S.A.
      • Nestel D.F.
      • Horrocks E.J.
      • Harrison R.H.
      • Jones N.
      • Wetzel C.M.
      • et al.
      Evaluation of a framework for case development and simulated patient training for complex procedures.
      Scenario designers also often rely on the expert opinions of experienced clinicians and simulation educators/instructors to validate scenarios once developed.
      • Black S.A.
      • Nestel D.F.
      • Horrocks E.J.
      • Harrison R.H.
      • Jones N.
      • Wetzel C.M.
      • et al.
      Evaluation of a framework for case development and simulated patient training for complex procedures.
      • Adler M.D.
      • Trainor J.L.
      • Siddall V.J.
      • McGaghie W.C.
      Development and evaluation of high-fidelity simulation case scenarios for pediatric resident education.
      Reliability refers to the consistency and accuracy of the measurement tool,
      • Polit D.
      • Beck C.
      Measurement and Data Quality. Essentials of Nursing Research: appraising evidence for nursing practice.
      or in the case of scenario development, consistency and accuracy of the scenario developed. To achieve reliability in simulation, scenarios must be designed and planned so they may be replicated consistently to ensure all participants are exposed to the same conditions. Whilst the importance of standardising scenarios for teaching and learning purposes is highlighted in the literature,
      • Issenberg S.B.
      • McGaghie W.C.
      • Petrusa E.R.
      • Gordon D.L.
      • Scalese R.J.
      Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review.
      there has been little discussion on how to achieve reliability in the design and operation of scenarios when evaluating participant performance as part of a research study.
      • Boulet J.R.
      • Jeffries P.R.
      • Hatala R.A.
      • Korndorffer J.R.
      • Feinstein D.M.
      • Roche J.P.
      Research regarding methods of assessing learning outcomes.
      Simulation-based training in aviation and healthcare recommend scenario scripts to ensure simulation conditions are maintained.
      • Fowlkes J.
      • Dwyer D.J.
      • Oser R.L.
      • Salas E.
      Event-based approach to training (EBAT).
      The integration of realistic ‘trigger events’ has also been reported as a valid technique and provides consistent opportunities to observe specific behaviours,
      • Fowlkes J.
      • Salas E.
      • Baker D.
      • Cannon-Bowers J.
      • Stout R.
      The utility of event-based knowledge elicitation.
      which may be integrated into scenario scripts to facilitate answering the research question.
      A scenario template was considered necessary to standardise scenario scripts, promote consistency when replicating scenarios and ensure the simulation answered the research question. A number of templates exist which inform scenario design for education and training purposes and most include information about the participants (target learner audience and cast members), instructional methods (learning objectives and teaching content), design strategies (scripting cues and events) and supporting details relating to the scenario (case summary, equipment/simulation setup and patient data).
      • Benishek L.E.
      • Lazzara E.H.
      • Gaught W.L.
      • Arcaro L.L.
      • Okuda Y.
      • Salas E.
      The template of events for applied and critical healthcare simulation (TEACH Sim): a tool for systematic simulation scenario design.
      • Dieckmann P.
      • Roll M.
      Designing a scenario as a simulated clinical experience: the TuPASS scenario script.
      • Taekman J.M.
      Duke University Template for Simulation Patient Design.
      • Alinier G.
      Developing high-fidelity health care simulation scenarios: a guide for educators and professionals.
      A selection of existing templates and a description of their components are presented in Table 2. An adaption of existing templates was needed to script valid and reliable simulated scenarios for use in clinical research.
      Table 2A summary of existing scenario templates and their components designed to guide simulation design for education and training in healthcare.
      TemplateComponents
      The Template of Events for Applied and Critical Healthcare Simulation (TEACH Sim)
      • Benishek L.E.
      • Lazzara E.H.
      • Gaught W.L.
      • Arcaro L.L.
      • Okuda Y.
      • Salas E.
      The template of events for applied and critical healthcare simulation (TEACH Sim): a tool for systematic simulation scenario design.
      Scenario overview; learner information; learning objectives and associated knowledge, skills and attitudes; clinical context; scenario development and ancillary information including patient information, modality, equipment and props and references.
      The TuPASS Scenario Script
      • Dieckmann P.
      • Roll M.
      Designing a scenario as a simulated clinical experience: the TuPASS scenario script.
      Scenario summary; learning goals and major debriefing points; narrative description of scenario; staffing; case briefings; simulator and room setup; simulator operation during scenario; scenario life savers; and actor instructions.
      Duke University Template for Simulation Patient Design
      • Taekman J.M.
      Duke University Template for Simulation Patient Design.
      Educational rational for simulation; learning objectives; target audience; authors; equipment and supportive objects; set-up; patient background information and baseline simulation state; scenario development states; and parameter adjustment form.
      Simulation Script by Alinier
      • Alinier G.
      Developing high-fidelity health care simulation scenarios: a guide for educators and professionals.
      based on the work of Dieckmann and Rall
      • Dieckmann P.
      • Roll M.
      Designing a scenario as a simulated clinical experience: the TuPASS scenario script.
      Scenario summary; actor instructions; narrative description of scenario including patient and actors scripts, patient/mannequin preparation, room and equipment setup, medical documentation, initial physiological parameters and simulation operation, results and teaching information and references; observation sheet; observed key events; and main points for debriefing.
      California Simulation Alliance Simulation Scenario Template
      • Polit D.F.
      • Beck C.T.
      Data Collection Methods. Essentials of Nursing Research: appraising evidence for nursing practice.
      Scenario overview; learner objectives and pre-learner activities; scenario script (case summary, key contextual details, scenario cast, patient/client profile, baseline simulator/standardised patient state, environment, equipment and essential props, and scenario development); and appendices including healthcare provider orders, digital images of mannequin/setting and debriefing guide.
      Special Interest Group Simulation Scenario Template
      • Graneheim U.H.
      • Lundman B.
      Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness.
      Scenario overview; objectives; competencies being assessed; environment and props; simulation personnel and assigned roles; case narrative including overview of case; patient information; flow diagram of expected interventions and reactions from patient simulator; distracters in case; and trends needed; instructor notes; debrief plan; pilot testing and revisions; authors and their affiliations.
      James Cook University Simulated Patient Template
      NVivo qualitative data analysis Software. Version 10.
      Scenario overview; objectives; patient information for learner; human resources; if/then table for events; debrief questions; simulation lab coordinator staging information.
      A new scenario template was constructed, based on four of the five critical NLN/Jeffries Simulation Framework constructs
      • Jeffries P.R.
      • Rogers K.
      Theoretical framework for simulation design.
      for design of simulation for research purposes (Appendix A). The NLN/Jeffries Simulation Framework was selected as it was developed based on research-evidence and considers all the fundamental components of simulation design which are pertinent to clinical research studies. The new template consists of four main sections: scenario overview; outcomes; participant and faculty information; and simulation design characteristics. The inclusion of the research study objectives, outcomes and data collection methods make this template unique as they have not been included in previous templates. Learning objectives and teaching information consistently present in existing templates were excluded to avoid contamination of study findings. A technical instruction template was also developed in response to feedback received after piloting the scenarios to inform simulation directors and technicians what were required in regards to the patient's physiological parameters, with potential options pre-programed and replicated consistently throughout the scenario (Appendix B). A brief explanation is provided describing what information should be inserted in each template.

      Development and validation process

      A three phase development and validation process were carried out to optimise scenario validity and reliability informed by existing evidence on simulation design. The three phases of development and validation undertaken are summarised in Fig. 1. After reviewing records of real patients, two simulated clinical scenarios were developed using the new scenario template. Design features such as staffing and equipment specified in scenario scripts were selected based on what is available in real EDs to enhance validity. A similar level of complexity, number of critical indicators and treatment needs were integrated into the different scenario scripts to facilitate comparison between the two different scenarios. Walk throughs were conducted during the scripting process to identify equipment required and feasibility of scenarios. Completed scenarios were peer reviewed by a panel of three emergency physicians for face validity. Scenarios were piloted in the simulated setting and refined prior to implementation in the research study. See Supplementary Data Table 1 and 2 for completed versions of the scenario template and technical instruction template used in Scenario A.
      Figure thumbnail gr1
      Figure 1Scenario development and validation process for use in clinical research studies.

      Implementation

      Following ethics approval from the relevant human research and ethics committee, the simulated clinical scenarios developed were implemented in a simulated study to test the HIRAID intervention. Registered nurses currently working in the ED with less than 3 years post graduate experience were invited to participate through placing an expression of interest in 6 Australian Hospital EDs across New South Wales. Thirty eight emergency nurses from 5 of the 6 EDs participated in the study. Participation was voluntary and written consent was obtained prior to commencing study activities.

       Scenario realism

      Following each scenario, participants were required to rate the level of realism achieved in each simulated clinical scenario on an 11-point Likert scale ‘0’ indicating very unrealistic and ‘10’ very realistic. Overall participants reported a moderate-to-high degree of realism was created in the scenarios (median (interquartile range (IQR)) = 7.50 (2.75)), indicating the simulation experience closely portrayed the clinical environment. Comparing scenario A to B in levels of perceived realism there was no significant difference reported by participants, (7.5 (3.00) vs. 7.5 (2.00), p = 0.59) (comparison performed using Wilcoxon-signed rank test).

       Post-simulation debriefing: interviews and focus groups

      Participants were not formally debriefed about the clinical cases presented in each scenario but were provided with an opportunity to express their feelings both as individuals and in groups to a non-clinical person who did not provide any teaching or feedback to participants as to not pollute study findings. These interactions functioned as interviews and focus groups through which participants were asked to describe the simulation experience and difficult or rewarding aspects of applying the HIRAID assessment model in the simulated setting. Individual interviews were conducted immediately post completion of each simulated scenario to alleviate any troubling feelings they may have been experiencing and gain insight into individual thoughts without influence from their peers. Focus groups were conducted at the end of the study to gain further insight into participants’ views by allowing for greater freedom in dialogue and expression of ideas through the interaction of participants.
      • Polit D.F.
      • Beck C.T.
      Data Collection Methods. Essentials of Nursing Research: appraising evidence for nursing practice.
      Seventy six interviews and 6 focus groups were conducted. There were 3 to 8 participates in each focus group.
      Graneheim & Ludman's
      • Graneheim U.H.
      • Lundman B.
      Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness.
      method of inductive content analysis was used to analyse interview and focus group data. Transcripts were transcribed verbatim. After reading transcripts several times to achieve a sense of the whole, the text was separated into content areas. Text pertaining to the simulated clinical scenarios were extracted and brought together into one document prior to being imported into NVivo™ v10
      NVivo qualitative data analysis Software. Version 10.
      for analysis. Meaning units were selected, condensed and labelled with a code. Codes were checked against original data sources to confirm validity of meaning and reviewed by the facilitator who checked the overall data for trustworthiness. The various codes were then sorted and grouped into categories. Categories were divided into two key themes: evaluation through simulation and the scenario design.

       Evaluation through simulation

      Thirteen participants (34%) described the knowledge of being watched or observed during the simulation as anxiety provoking: ‘Knowing being watched heightened everything’ (RN 27). This concern was expressed after both scenario A and scenario B and was also raised in one of the focus groups: ‘it was really hard to try and think straight because you’re really worried what's going on with your patient… and have the knowledge of being observed’ (Focus group 5). Knowing some of the research team was referred to by three participants (in Scenario B only) as adding to their anxieties and fear of failure: ‘Knowing MM, my clinical nurse coordinator & educators adds extra pressure, may be different with someone else’ (RN20). Participants also indicated that acting and pretending during the simulated scenarios was challenging: ‘it's hard in sim setting to act out’ (RN25). Difficulties thinking out loud during the scenarios were also described in one of the focus groups: ‘there's a lot of stuff that you do without even thinking but then in this scenario you’ve got to think out loud. It's just not natural.’ (Focus group 5).

       Scenario design

      Design characteristics were described to either increase or decrease the realism of the scenarios including: the patient, the environment and the scenario duration.

       The patient

      The majority of participants felt that the patients in both scenarios were realistic. Fifteen participants (39%) reported that the use of human actors as patients was helpful and felt that they closely portrayed how a patient would normally present in the real world: ‘it was realistic, the way the patient came in acting…acting the same symptoms’ (RN7). Only four participants (11%) stated that they felt the patient was not realistic in either scenario A or scenario B: ‘Was a bit different to how a real patient presents.’ (RN2).

       The environment

      Fourteen participants (37%) expressed concerns relating to the simulation environment being different to the clinical setting in both scenarios. Participants used words such as ‘strange’, ‘different’ and ‘unfamiliar’. One participant noted that the simulation environment was ‘quiet’ compared with the noisy ED setting (RN12). It was recommended that a more comprehensive orientation would be useful to overcome these concerns and enable participants to ‘get a better sense of where everything is’ (RN41).
      Confusion was reported relating to what was real and what was pretend in both scenarios by three participants (8%) and was also raised as a concern in one of the focus groups. This confusion created uncertainty around what the participant should or shouldn’t do: ‘It was hard, like cos definitely the first thing I do when I get a patient I’m like “Right, out of your clothes!” And this one, I’m like, are we stripping them off?’ (Focus Group 2).
      A lack of support from other clinicians during both scenarios was expressed by 15 participants (40%). This is described in contrast to the normal network of doctors and nurses available in the clinical setting: ‘In real life there is always someone to ask ‘what would you do?” (RN39). The idea of seeking additional staff to assist with patient care and ask questions was also present in two of the focus groups: ‘You’re in this big room all by yourself (laughs)] and you hope one of the nurses will drop around…’ (Focus group 2); ‘… generally you can go and assess your patient and do your initial assessment with two’ (Focus group 4).

       Scenario duration

      The brief duration of the scenarios was a prominent concern of participants throughout the interviews. Seventeen participants (45%) described feeling either ‘rushed’ or ‘pressed for time’, due to not having enough time to complete the assessment and/or clinical documentation during both scenarios. One participant stated the timeframe was ‘ok’ (RN36). In three of the focus groups participants discussed that severe time constraints were characteristic of the clinical ED environment and frequently stated that they often ‘don’t have time’ to complete clinical work.

      Using simulation as a clinical research tool

      The implementation of full immersion simulation as a clinical research tool in the HIRAID study presented a number of new experiences not observed in the educational setting. The structured development and validation process provide researchers with a systematic approach to developing valid and reliable simulated clinical scenarios. The benefits of the synthetic environment were found to provide ideal conditions necessary to evaluate the intervention (new assessment model) without causing harm to patients. Specific triggers realistic of the clinical setting were scripted and carried out consistently in scenarios through the use of the new scenario design template. This enabled investigation of the research question without prompting participants how to behave, whilst achieving a high degree of realism as reported by participants.
      The importance of post-simulation debriefing is evident in the education literature necessary for participants to reflect on the simulated experience and learn from what happened.
      • Fanning R.M.
      • Gaba D.M.
      The role of debriefing in simulation-based learning.
      • Arafeh J.M.
      • Hansen S.S.
      • Nicols A.
      Debriefing in simulated-based learning: facilitating a reflective discussion.
      The value of debriefing in the research study was used as an evaluation tool to gain further insight into the opinions and experiences of study subjects in relation to the usability and relevance of the intervention under investigation. The use of interviews and focus groups also provided insight into participants’ perceptions of the design of the two scenarios presented. There were no recommendations made by participants to indicate any modifications to the new scenario template.
      Whilst the overall realism of the scenarios presented were rated highly by participants, there were some design features of the simulation that were described as lacking in realism. The use of ‘in situ’ simulation has been recommended as an educational tool to overcome differences in the simulated setting, where simulated clinical scenarios are conducted within the actual work environment which they are familiar.
      • Patterson M.D.
      • Blike G.T.
      • Nadkarni V.M.
      In Situ Simulation: challenges and Results.
      A similar model may also be used in clinical research studies, however issues of overcrowding and bed shortages common to EDs
      • Forero R.
      • Hillman K.
      • McCarthy S.
      • Fatovich D.M.
      • Joseph A.P.
      B RD. Access block and ED overcrowding.
      is likely to make access to treatment spaces to conduct simulations challenging. Unfamiliarity with the simulated environment may be reduced through fuller orientation to the simulation environment. The importance of preparing students appropriately for participation in simulation has been raised to optimise learning in performance,
      • Gantt L.T.
      The effect of preparation on anxiety and performance in summative simulations.
      but there has been little discussion in the literature on how to orientate participants to the simulation environment when simulation techniques are used as an evaluation tool in clinical research studies.
      Scenarios were purposely designed with limited clinical support available to participants and with a limited timeframe to enable investigation of the research question. The inclusion of additional clinical staff to help the nurse would have likely prevented evaluation of the individual participant's skills through providing participant with too much support and cues. Timeframes were standardised to make certain participants were exposed to the same conditions and to allow enough time to measure the intended study outcomes.
      Anxiety associated with being observed reported by participants during the simulated clinical scenarios in this study has similarly been reported in other research studies.
      • Pierazzo J.
      Learner Anxiety and Professional Practice Selfefficacy in Nursing Education.
      • Cato M.L.
      Nursing Student Anxiety in Simulation Settings: a Mixed Methods Study.
      • Treadwell I.
      • van Rooyen M.
      • Havenga H.
      • Theron M.
      The effect of an interprofessional clinical simulation on medical students.
      Simulation as an educational tool has been found to increase psychological stress levels among clinicians when compared with traditional teaching methods.
      • Bong C.L.
      • Lightdale J.R.
      • Fredette M.E.
      • Weinstock P.
      Effects of simulation versus traditional tutorial-based training on physiologic stress levels among clinicians: a pilot study.
      Clinicians have specifically expressed concerns that anxiety experienced in the simulated environment reduces their ability to demonstrate knowledge and competence.
      • Bryson E.O.
      • Levine A.I.J.O.C.C.
      The simulation theater: a theoretical discussion of concepts and constructs that enhance learning.
      • Leigh G.T.
      High-fidelity patient simulation and nursing students’ self-efficacy: a review of the literature.
      Further investigation is needed to understand the effects of simulation on participant anxiety and performance whilst being evaluated as part of a clinical research study.

      Conclusion

      The majority of literature on clinical simulation design discusses the use of simulation as part of teaching and learning. We presented a new scenario template designed for use in evaluating performance in clinical research studies as part of a systematic scenario development and validation process. Two standardised simulated clinical scenarios were developed and validated using this organised approach and were both reported to achieve a high degree of realism. The systematic process undertaken to achieve valid and reliable simulated clinical scenarios provides clinicians with a standardised evidence-informed approach to scenario design which may be used in future clinical research studies.

      Provenance and conflict of interest

      Author Kate Curtis is an Associate Editor of the Australasian Emergency Nursing Journal but played no role in the peer review or editorial decision-making of this paper whatsoever. No other conflicts of interest were declared. This paper was not commissioned.

      Funding

      The collection and interpretation of data for this study were funded by the New South Wales (NSW) Emergency Care Institute, NSW Agency for Clinical Innovation, NSW Ministry of Health, Australia.

      Research ethics statement

      This paper reports the findings of a research study that adhered to the National Statement on the Conduct of Human Research by the Australian National Health and Medical Research Council, and has been approved by the Western Sydney Local Health District Human Research and Ethics Committee Approval LNR/13/WMEAD/44.

      Authorship

      All authors contributed to the design, execution and analysis of the work presented, and attest to the accuracy and validity of the contents.

      Acknowledgements

      Thank you to the NSW Emergency Care Institute (ECI) for granting funding for this study (http://www.ecinsw.com.au/). We gratefully acknowledge the emergency physicians who validated the scenarios: Dr Richard Pryor, Dr Mark Newcombe and Dr Serena Ayers (Wollongong Hospital Emergency Department (ED)); the nurses who assisted in piloting the scenarios: Nurse Practitioner Luke Strachan (Blacktown Hospital ED), Clinical Nurse Consultant Margaret Murphy (Westmead Hospital ED), Dr Jennifer Hardy (Sydney Nursing School University of Sydney), Clinical Nurse Educator Amy McCarthy, Nurse Practitioner Wendy Fenton and Nurse Educator Mark Wilson (Wollongong Hospital ED); and Education Consultant Melinda Lewis (University of Sydney) who assisted in data collection procedures. We also wish to acknowledge the staff from Sydney Nursing School (University of Sydney) Simulation Laboratory, Illawarra Shoalhaven Health Education Centre and Westmead Simulation Laboratory for the use of their facilities and technical support.

      Appendix A. Simulated clinical scenario design template for use in research studies devised to inform simulation faculty and facilitate the consistent operation of scenarios

      Appendix B. Technical information template designed to direct simulation technicians of the patient's physiological parameters during simulated clinical scenarios for use in clinical research studies

      Appendix C. Supplementary data

      References

        • Jeffries P.
        Using Simulation in Nursing Education. Simulation in Nursing Education: From Conceptualization to Evaluation.
        2nd ed. The National League for Nursing, New York2012: 3
        • McGaghie W.C.
        • Issenberg B.
        • Petrusa E.R.
        • Scalese R.J.
        A critical review of simulation-based medical education research: 2003−2009.
        Med Educ. 2010; 44: 50-63
        • Schmidt E.
        • Goldhaber-Fiebert S.
        • Ho L.
        • McDonald K.
        Simulation exercises as a patient safety strategy.
        Ann Internal Med. 2013; 158: 426-432
        • Zendejas B.
        • Brydges R.
        • Wang A.
        • Cook D.
        Patient outcomes in simulation-based medical education: a systematic review.
        J Gen Intern Med. 2013; 28: 1078-1089
        • Dieckmann P.
        • Phero J.C.
        • Issenberg S.B.
        • Kardong-Edgren S.
        • Ostergaard D.
        • Ringsted C.
        The first research consensus Summit of the society for simulation in healthcare: conduction and a synthesis of the results.
        Simulat Healthcare. 2011; 6: S1-S9
        • LeBlanc V.R.
        • Manser T.
        • Weinger M.B.
        • Musson D.
        • Kutzin J.
        • Howard S.K.
        The study of factors affecting human and systems performance in healthcare using simulation.
        Simul Healthcare. 2011; 6: S24-S29
        • Vardi A.
        • Berkenstadt H.
        • Levin I.
        • Bentencur A.
        • Ziv A.
        Intraosseous vascular access in the treatment of chemical warfare casualties assessed by advanced simulation: proposed alteration of treatment protocol.
        Anesth Analg. 2004; 98: 1753-1758
        • Gaba D.
        • Howard S.
        • Flanagan B.
        • Smith B.
        • Fish K.
        • Botney R.
        Assessment of clinical performance during simulated crises using both technical and behavioral ratings.
        Anesthesiology. 1998; 89: 8-18
        • Wright M.C.
        • Segall N.
        • Hobbs G.
        • Phillips-Bute B.
        • Maynard L.
        • Taekman J.M.
        Standardized assessment for evaluation of team skills: validity and feasibility.
        Simul Healthcare. 2013; 8: 292-303
        • Fletcher G.
        • Flin R.
        • McGeorge P.
        • Glavin R.
        • Maran N.
        • Patey R.
        Anaesthetists’ non-technical skills (ANTS): evaluation of a behavioural marker system.
        Bri J Anaesth. 2003; 90: 580-588
        • Yule S.
        • Flin R.
        • Maran N.
        • Rowley D.
        • Youngson G.
        • Paterson-Brown S.
        Surgeons’ non-technical skills in the operating room: reliability testing of the NOTSS behavior rating system.
        World J Surg. 2008; 32: 548-556
        • Howard S.
        • Gaba D.
        • Smith B.
        • Weinger M.
        • Herndon C.
        • Keshavacharya S.
        • et al.
        Simulation study of rested versus sleep-deprived anesthesiologists.
        Anesthesiology. 2003; 98: 1345-1355
        • Merry A.
        • Weller J.
        • Robinson B.
        • Warman G.
        • Davies E.
        • Shaw J.
        • et al.
        A simulation design for research evaluating safety innovations in anaesthesia.
        Anaesthesia. 2008; 63: 1349-1357
        • Hamman W.
        • Beaudin-Seiler B.
        • Beaubien J.
        • Gullickson A.
        • Orizondo-Korotko K.
        • Gross A.
        • et al.
        Using in situ simulation to identify and resolve latent environmental threats to patient safety: case study involving operational changes in a labor and delivery ward.
        Qual Manage Health Care. 2010; 19: 226-230
        • Worster A.
        • Rowe B.
        • Stiell I.
        • Borgundvaag B.
        • Sivilotti M.
        • Campbell S.
        • et al.
        Clinical research in the emergency department conducted by non-emergency physicians: potential problems and proposed recommendations.
        Can J Emergen Med. 2005; 7: 241-248
        • Cofield S.
        • Conwit R.
        • Barsan W.
        • Quinn J.
        Recruitment and retention of patients into emergency medicine clinical trials.
        Acad Emergen Med. 2011; 17: 1104-1112
        • Littlewood K.E.
        High fidelity simulation as a research tool.
        Best Pract Res Clin Anaesth. 2011; 25: 473-487
        • Nestel D.
        • Clark S.
        • Tabak D.
        • Ashwell V.
        • Muir E.
        • Paraskevas P.
        • et al.
        Defining responsibilities of simulated patients in medical education.
        Simul Healthcare. 2010; 5: 161-168
        • Gaba D.
        The future vision of simulation in health care.
        Qual Saf Health Care. 2004; 13: i2-i10
        • Jeffries P.R.
        • Rogers K.
        Theoretical framework for simulation design.
        in: Jeffries P.R. Simulations in Nursing Education: from Conceptualization to Evaluation. The National League for Nursing, New York2012: 25-42
        • Durham C.F.
        • Cato M.L.
        • Lasater K.
        NLN/Jeffries simulation framework state of the science project: participant construct.
        Clin Simul Nurs. 2014; 10: 363-372
        • Jeffries P.R.
        A framework for designing, implementing, and evaluating simulations Used as teaching strategies in nursing.
        Nurs Educ Perspect. 2005; 26: 96-103
        • Munroe B.
        • Curtis K.
        • Margerat M.
        • Strachan L.
        • Buckley T.
        HIRAID: an evidence-informed emergency nursing assessment framework.
        Aust Emergen Nurs J. 2015; 18: 83-97
        • Graham I.D.
        • Logan J.
        • Harrison M.B.
        • Harrison M.B.
        • Straus S.E.
        • Tetroe J.
        • et al.
        Lost in knowledge translation: time for a map?.
        J Contin Educ Health Prof. 2006; 26: 13-24
        • Polit D.
        • Beck C.
        Measurement and Data Quality. Essentials of Nursing Research: appraising evidence for nursing practice.
        Lippincott Williams & Wilkins, Philadelphia: Wolters Kluwer2010: 377
        • Cioffi J.
        Clinical simulations: development and validation.
        Nurs Educ. 2001; 21: 477-486
        • Gormley G.
        • Sterling M.
        • Menary A.
        • McKeown G.
        Keeping it real! Enhancing realism in standardised patient OSCE stations.
        Clin Teach. 2012; 9: 382-386
        • Kirlik A.
        Brunswikian theory and method as a foundation for simulation-based research on clinical judgment.
        Simul Healthcare. 2010; 5: 255-259
        • Black S.A.
        • Nestel D.F.
        • Horrocks E.J.
        • Harrison R.H.
        • Jones N.
        • Wetzel C.M.
        • et al.
        Evaluation of a framework for case development and simulated patient training for complex procedures.
        Simul Healthcare. 2006; 1: 66-71
        • Adler M.D.
        • Trainor J.L.
        • Siddall V.J.
        • McGaghie W.C.
        Development and evaluation of high-fidelity simulation case scenarios for pediatric resident education.
        Ambul Pediatr. 2007; 7: 182-186
        • Issenberg S.B.
        • McGaghie W.C.
        • Petrusa E.R.
        • Gordon D.L.
        • Scalese R.J.
        Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review.
        Med Teach. 2005; 27: 10-28
        • Boulet J.R.
        • Jeffries P.R.
        • Hatala R.A.
        • Korndorffer J.R.
        • Feinstein D.M.
        • Roche J.P.
        Research regarding methods of assessing learning outcomes.
        Simul Healthcare. 2011; 6: S48-S51
        • Fowlkes J.
        • Dwyer D.J.
        • Oser R.L.
        • Salas E.
        Event-based approach to training (EBAT).
        Int J Aviat Psychol. 1998; 8: 209-221
        • Fowlkes J.
        • Salas E.
        • Baker D.
        • Cannon-Bowers J.
        • Stout R.
        The utility of event-based knowledge elicitation.
        Hum Factors. 2000; 42: 24-35
        • Benishek L.E.
        • Lazzara E.H.
        • Gaught W.L.
        • Arcaro L.L.
        • Okuda Y.
        • Salas E.
        The template of events for applied and critical healthcare simulation (TEACH Sim): a tool for systematic simulation scenario design.
        Simul Healthcare. 2015; 10: 21-30
        • Dieckmann P.
        • Roll M.
        Designing a scenario as a simulated clinical experience: the TuPASS scenario script.
        in: Kyle R.R. Murray B W Clinical Simulation: Operations, Engineering and Management. Elsevier, London, UK2008: 541
        • Taekman J.M.
        Duke University Template for Simulation Patient Design.
        2003
        • Alinier G.
        Developing high-fidelity health care simulation scenarios: a guide for educators and professionals.
        Simulat Gaming. 2011; 42: 9-26
        • Polit D.F.
        • Beck C.T.
        Data Collection Methods. Essentials of Nursing Research: appraising evidence for nursing practice.
        7th ed. Lippincott Williams & Wilkins, Philadelphia Wolters Kluwer2010: 341
        • Graneheim U.H.
        • Lundman B.
        Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness.
        Nurse Educ Today. 2004; 24: 105-112
      1. NVivo qualitative data analysis Software. Version 10.
        QSR International Pty Ltd, 2012 (http://www.qsrinternational.com/Support/FAQs/How-do-I-cite-NVivo-10-NVivo-9-or-NVivo-8-in-my-wo)
        • Fanning R.M.
        • Gaba D.M.
        The role of debriefing in simulation-based learning.
        Simulat Healthcare. 2007; 2: 114-125
        • Arafeh J.M.
        • Hansen S.S.
        • Nicols A.
        Debriefing in simulated-based learning: facilitating a reflective discussion.
        J Perinat Neonat Nur. 2010; 24: 302-309
        • Patterson M.D.
        • Blike G.T.
        • Nadkarni V.M.
        In Situ Simulation: challenges and Results.
        Agency for Healthcare Research and Quality, Philadelphia2008
        • Forero R.
        • Hillman K.
        • McCarthy S.
        • Fatovich D.M.
        • Joseph A.P.
        B RD. Access block and ED overcrowding.
        Emergen Med Aust. 2010; 22: 119e35
        • Gantt L.T.
        The effect of preparation on anxiety and performance in summative simulations.
        Clin Simulat Nurs. 2013; 9: e25-e33
        • Pierazzo J.
        Learner Anxiety and Professional Practice Selfefficacy in Nursing Education.
        The University of Western Ontario;, 2014
        • Cato M.L.
        Nursing Student Anxiety in Simulation Settings: a Mixed Methods Study.
        Portland State University, 2013
        • Treadwell I.
        • van Rooyen M.
        • Havenga H.
        • Theron M.
        The effect of an interprofessional clinical simulation on medical students.
        Afr J Health Prof Educ. 2014; 6
        • Bong C.L.
        • Lightdale J.R.
        • Fredette M.E.
        • Weinstock P.
        Effects of simulation versus traditional tutorial-based training on physiologic stress levels among clinicians: a pilot study.
        Simulat Healthcare. 2010; 5: 272-278
        • Bryson E.O.
        • Levine A.I.J.O.C.C.
        The simulation theater: a theoretical discussion of concepts and constructs that enhance learning.
        J Crit Care. 2008; 23: 185-187
        • Leigh G.T.
        High-fidelity patient simulation and nursing students’ self-efficacy: a review of the literature.
        Int J Nurs Scholarship. 2008; 5: 1-17