Chapter Authors: Stephanie Marton, MD & Gordon Schutze, MD
There are four areas of assessment and evaluation to consider with GH training: 1) assessment of the trainee; 2) evaluation of the GH training program; 3) evaluation of the GH elective; and 4) evaluation of outcomes of the GH training program.
Assessment of trainee performance during a GH elective can be difficult to obtain due to time limitations of the host supervisor as well as the length of standard stateside assessment forms. Innovative strategies for obtaining assessment data surrounding the GH elective and from the host supervisor are offered.
As with all training programs, assessment and evaluation are fundamental components for continual optimization. With regard to GH training for U.S. residency and fellowship programs, there are four realms of evaluation and assessment to consider:
- Assessment of the trainee, particularly pertaining to performance surrounding the GH elective experience and, if enrolled in a GH track or fellowship, a milestones-based evaluation throughout training
- Evaluation of the GH training program, particularly for institutions that offer tracks, additional years for GH training, GH fellowships, or other GH-specific curricula
- Evaluation of the GH elective, including feedback on the preparation process, the on-site experience, and the debriefing
- Evaluation of outcomes of the GH training program.
A fifth realm is evaluation of the impact of global training partnerships, which is pertinent if trainees are traveling under the auspices of a partnership. This includes assessment of the impact of individual visiting trainees on host institutions, as well as a more comprehensive periodic partnership-specific evaluation. This fifth realm is outside of the scope of this chapter but should be considered for ongoing partnerships.46
Assessment of the Trainee
The assessment of trainees during GH experiences can be done in a manner similar to that which the training program has in place for other electives or programs. There are many different tools available for assessing trainees. Some of the methods that have been utilized by programs offering GH electives are listed below.3
- Knowledge assessments such as a pre-and post-assessment of specific topics
- Competency-based assessments
- A procedure log or patient diagnosis log
- Simulation-based assessments
- Trainee self-assessments
- Qualitative assessments such as essays written during and after the elective
- Mentorship meetings
Although many of the above formats are utilized throughout training, some special considerations should be taken into account when obtaining GH elective evaluations. Competency-based assessments typically require obtaining written feedback from an international site — usually communicated electronically, via facsimile, or via written documentation transported by the trainee. Additionally, there may be different options for who should complete the evaluation; typically, someone who has directly observed the trainee's clinical skills will provide the most useful feedback. Although traditional pediatric competency-based assessments are often used and modified for GH tracks,4 some programs use an abbreviated assessment tool that focuses on key milestones and offers an opportunity for open-ended input.12 In such situations, the home country’s GH faculty must use that abbreviated assessment and translate it into the comprehensive milestones-based assessment that is used at the home training institution.
In situations where the host site evaluator is uncomfortable with the requested assessment tool or when language barriers or lack of familiarity with the pediatric milestones prevent a thorough evaluation,12 trainee self-assessment and stateside mentor assessments can be useful. Pre- and post-elective mentorship meetings offer an opportunity for stateside faculty to assess trainee performance and progress surrounding the GH elective, particularly pertaining to professionalism. Appendix R offers examples of GH-specific self-assessments and learner assessments pertinent to GH electives. Ideally, assessment tools would be created in collaboration with the host supervisors whenever possible, and host supervisors should have access to the trainee goals and objectives to help inform their assessments. Additionally, trainees should give the assessment to the host supervisors at the beginning of the elective so that the supervisors can be aware of assessment priorities (and modify the tool in collaboration with the home institution if warranted).
Assessment tools may also vary depending on the duration of the elective. Qualitative assessments such as personal reflections can provide more in-depth insights only realized by the learner after several weeks on the elective. Procedure logs allow trainees to have a method to validate competencies in new procedures learned. A patient diagnosis log, maintained daily with patient age and primary diagnoses may also be helpful to the trainee once they return, allowing them to document and recall the management of cases that they may not have been exposed to in their home country.
Evaluation of the GH Training Program
If a program offers additional training (other than GH electives), annual evaluations should be conducted. Examples of useful components of a GH training program evaluation include but are not limited to:
- Perceived impression of whether participation in GH training has positively influenced knowledge base, clinical skills, advocacy skills, career trajectory, etc.
- Utility of curriculum and other training modalities (eg, journal clubs, simulation sessions, trainee conferences, pre-departure sessions)
- Availability of local GH experiences
- Availability of international GH electives
- Comprehensiveness of pre-departure training for international electives
- Evaluation of program leader(s) (communication skills, teaching skills, etc.)
- Opportunities for GH-related scholarly work
- Most helpful aspects of program
- Most surprising aspects of program
- Program aspects that should change
- Impact of the program on post-residency career plans.
Evaluation of the GH Elective
Trainees should have the opportunity to evaluate their GH elective site as well as their experiences with preparation and debriefing. The site evaluations are particularly pertinent for institutions with existing partnerships in order to promote continual modifications and improvements34 but are also applicable for non-partnership electives so that other trainees can have insight into their peer experiences. All efforts should be made to return feedback obtained from the trainee to the host faculty in a timely fashion. Elective site evaluations should seek input on details that include the following:
- Host faculty
- Was there sufficient communication prior to the start of the elective?
- Were expectations communicated at the beginning of the elective?
- Were expectations appropriately aligned with trainee scope of expertise?
- Was the level of supervision appropriate?
- Was teaching suitable to the learner’s level of knowledge?
- Did supervising physicians treat the learner with respect?
- Was feedback provided during and after the elective?
- Did the host faculty/trainees extend hospitality?
- Ease of transition to and from international site?
- Availability of safe, affordable, and appropriate housing?
- Opportunities for self-learning outside of clinical hours (eg, community activities, cultural activities)?
- Stateside faculty
- Accessibility during elective?
- Thoroughness of pre-departure meetings?
- Usefulness of post-return debriefing meeting?
Additional components pertinent to the specific partnerships should be periodically assessed for the purposes of quality improvement, which should include seeking evaluations from the host faculty regarding the GH elective and ongoing partnership.
Evaluation of GH Training Outcomes
GH electives have increased in popularity in the past 10 years, and over half of pediatric residency programs now offer GH electives.47 Administration of GH electives can be labor-intensive, and allowing trainees to be away from their home training site may add additional burdens to the training programs. It is therefore critical for programs to show the value of these electives to both an individual trainee as well as to the medical community as a whole.
Immediate trainee outcomes can be obtained through either knowledge-based assessments or trainee self-assessments. Knowledge-based assessments could include review of specific sections of pediatric in-service examinations, review of specific categories of resident PREP question completions, or specific knowledge questions designed for a GH elective. Self-assessment survey questions could include the following:
- Which of the following do you feel more confident in managing after your time abroad (eg, TB, HIV, malnutrition, pneumonia, diarrhea, etc.)?
- Which of the following skills do you feel more confident in managing after your time abroad (eg, intravenous access, withdrawing blood, lumbar punctures, resuscitation, etc.)?
Long-term outcomes of GH training are harder to assess but equally important. It has been shown that trainees who have spent time abroad are more likely to continue their work with the underserved, either at home or abroad.48 Additionally, there is evidence to support that trainees with GH exposure during residency are more likely to pursue careers with a significant GH component.49
To evaluate GH training program outcomes, follow-up trainee surveys can be conducted by either the training program or by larger groups such as the APPD, AAP, and ABP. Example questions include:
- Are you engaged in full or part-time care abroad?
- Are you engaged in caring for local-global populations in the United States (eg, refugees, immigrants)?
- Are you engaged in care of the underserved in your medical practice?
- Are you engaged in policy making at either an international level or at the local level that focuses on the underserved?
- Are you engaged in advocacy regarding care of underserved populations?
Innovative strategies for evaluating outcomes of GH training continue to be explored by the GH education community, and educators are encouraged to collaborate with other programs to measure the impact of both stateside training and GH electives.
For those seeking to employ a GH track director and/or a GH track program coordinator, sample position descriptions are included in Appendices B and C.
Citation: St Clair N, Abdul-Mumin A, Banker S, Condurache T, Crouse H, Helphinstine J, Kazembe P, Lukolyo H, Marton S, McQuilkin P, Pitt M, Rus M, Russ C, Schubert C, Schutze G, Steenhoff A, Uwemedimo O, Watts J, Butteris S. Global health in pediatric education: an implementation guide for program directors. American Board of Pediatrics Global Health Task Force Publication, in collaboration with the American Academy of Pediatrics Section on International Child Health and the Association of Pediatric Program Directors Global Health Learning Community. September, 2018. abp.org/ghpdguidehome