Pediatricians routinely monitor a child’s growth and development at well-child visits. Similar checks are part of a pediatrician’s training. The traditional way to track a resident is through knowledge tests and time in training — usually three years of supervised clinical rotations.
But pediatric program directors, among others, have been searching for a better measure of readiness for unsupervised practice.
“We have to think about the essential activities residents need to perform safely and effectively by the end of training and then make a judgment about whether they are ready to practice unsupervised,” says Carol Carraccio, MD, MA, ABP Vice President for Competency-Based Medical Education. Dr. Carraccio has been implementing and studying competency-based medical education for almost two decades.
While the pediatric community developed milestones and introduced entrustable professional activities (EPAs) about a decade ago, meaningful assessment has been an ongoing struggle. Recently, however, Dr. Carraccio and Daniel Schumacher, MD, MEd, Chair of the Education Research Group at Cincinnati Children’s, along with 22 training programs throughout the country, conducted a three-year study to determine whether general pediatric residents were ready to practice all 17 EPAs without supervision when they finished training, and if not, to identify their training gaps. Results from the study will be published in 2020.
Drs. Carraccio and Schumacher worked with the Association of Pediatric Program Directors’ Longitudinal Educational Assessment Research Network (APPD LEARN) to enroll pediatric residency program directors in the study and to help with data collection and analysis. The participating program directors committed to implementing EPAs and collecting performance data on their residents.
“Program directors [involved in APPD LEARN] are really interested in the assessment of residents,” says Alan Schwartz, PhD, director of APPD LEARN. “They want to understand how to know what residents are good at and what they need to get better at.”
The ultimate goal, he says, is to know “what residents can be allowed to do with minimal supervision, what they need more supervision for, and how that changes as they go through residency and eventually graduate.”
The study followed nearly 2,000 trainees throughout their three years of training. The study found that two of the lowest-performing EPAs were treating behavioral and mental health and understanding the science behind quality improvement (QI).
Dr. Carraccio says several factors might have contributed to the gap in training for the treatment of behavioral and mental health. The number of children with behavioral and mental health problems is escalating, and most pediatric faculty members are not well-equipped to teach students and residents in this area because they themselves were not trained to diagnose and treat these conditions. To make the problem worse, there are not enough child psychiatrists to care for all the patients who need them.
“Training programs are really struggling with how they are going to teach this,” Dr. Carraccio adds. “They say, ‘We need resources to help us do a better job.’”
The other gap identified in the study is the trainees’ understanding of QI science. After pediatricians pass their initial board certification examination, one requirement for continuing certification is to engage in ongoing practice improvement.
“If you don’t know how to do a quality improvement project, it [the QI requirement] becomes a real problem,” says Dr. Carraccio.
Now that the data has shown gaps in training, Dr. Carraccio says the next step is to disseminate the findings to all the programs.
Some program directors who participated in the study are beginning to address these gaps. They have begun to present their ideas at national meetings, sponsored by groups such as the APPD and Pediatric Academic Societies, to share their progress and lessons learned with other training programs.
“Our hope is to partner with even more pediatric organizations to create learning activities for trainees and faculty,” says Dr. Carraccio. “Then we’ll disseminate the learning activities that have been created and look at the impact of those learning activities on performance over time.”