(1091) A Validated Assessment Tool for the Competence of Advanced Practice Providers in Simulation

Introduction: Assessment encompasses a variety of techniques including tests, ratings, and observations designed to assign a score that represents the degree of competence an individual possesses. The debate will continue over the use of simulation as an assessment tool versus an educational one. A review of the literature revealed no valid, reliable instrument to assess competence for the population of APPs within both simulation and clinical settings. Thus, the purpose of this exploratory study was to develop and assess the content validation of a modified Advanced Practice Provider- Queen’s Simulation Assessment Tool (APP-QSAT) based on best evidence and resources available.

Methods: The APP-QSAT consists of the following domains: primary assessment, diagnostic actions, therapeutic actions, communication, and overall performance. Thirty items were analyzed by nine content experts on a content validity index tool. The content experts were a sample of medical professionals who possessed experience in simulation training of medical residents, medical fellows, medical students, nurses, nursing students, nurse practitioners (NPs), NP students, physician associates (PAs), paramedics, pharmacists, and pharmacy students. There were two rounds of content validity implemented.

Results: In the first round, six items were less than 0.78 and modified for the second round. The second round included a subset of five content experts from the original panel. After two rounds of content validation, the APP-QSAT demonstrated having excellent content validity with an item-level content validity index (I-CVI) of 1.00 after revision of items. A modified kappa of 1.0 and probability of 0.41 also demonstrated the exclusion of change agreement among the content experts. Internal consistency serves as an estimate of reliability, which demonstrated an alpha coefficient of 1.00.

Conclusions: The APP-QSAT is a modifiable template, designed to assess competence for APPs within a simulation activity. Educators can modify the instrument in accordance with a chosen simulation scenario and at the level of the learner. The APP-QSAT captures the spectrum of clinical competence, not only technical skills but also critical thinking, decision-making, and communication.