Team 2: Patient Care
Team 3: Professionalism
Team 4: Communication
Team 6: Practice Based Learning
Team 7: Faculty Development
Team 8: Directed Lifelong Learning
Assignment: Click here to open assignment in Microsoft Word
Context |
In your group there are people that attended all five of the tradeshow presentations in the last round of work. You have been provided a list of the knowledge, skills and attitudes developed in the first colloquium mapped to the program requirements – plus any additional knowledge, skills and attitudes added in the round of work on the Patient Perspective. You've also been provided a document that defines the ACGME competencies and a document that highlights the pediatric practice competencies as background information. Your group will focus on the Medical Knowledge competence. |
Objective |
The objective for this round of work is to match the knowledge, attitudes and skills to the assessment tools and modalities you've learned about in the Tradeshows. |
Process |
You have about 90 minutes for this activity including lunch. As a group, first, update each other on what you learned in the tradeshows. Then, develop a model that matches the knowledge, attitudes and skills under your competence ( Medical Knowledge ) to the assessment tools and modalities you learned about today. Use the following questions to stimulate your conversations:
Use the white walls and markers provided to brainstorm and develop your model. |
Exchange |
After you finish this activity you will have an opportunity to share your work with the other groups. |
Our focus was Medical Knowledge. Multiple Choice Questions are a pretty good way to measure this competence. We think that a portfolio could also help us measure this – the supervisor should be able to document learning in real time. The Clinical Skills Exam can assess clinical reasoning, but only as a snapshot. The mini-CSX is a poor tool. The 360-review is a poor tool. The directed lifelong learning is a good tool with an MCQ component. The Global Evaluation Tool would be an interesting tool if the evaluator had a long-term relationship with the student. We would give the oral exam tool a “C” for assessing Medical Knowledge. (Some specialties still believe in the oral exam. None of the tools received an “A” for assessing Medical Knowledge.
The Canadians have done some nice work with Strict Concordance Testing. A patient will present with some information (e.g. vaginal bleeding). The student is asked to report their conclusions. Then they are presented with another piece of information, and asked to revaluate. And so on.
The Triple Jump test asks students to go research an issue in 15 minutes and come back with their analysis and diagnosis. This tests the student's ability to ask the right questions and find information instead of their ability to recall information.
We cannot test everything. Some tests are good indicators of a student's ability to eventually pass the boards.
Assignment: Click here to open assignment in Microsoft Word
Context |
In your group there are people that attended all five of the tradeshow presentations in the last round of work. You have been provided a list of the knowledge, skills and attitudes developed in the first colloquium mapped to the program requirements – plus any additional knowledge, skills and attitudes added in the round of work on the Patient Perspective. You've also been provided a document that defines the ACGME competencies and a document that highlights the pediatric practice competencies as background information. Your group will focus on the Patient Care competence. |
Objective |
The objective for this round of work is to match the knowledge, attitudes and skills to the assessment tools and modalities you've learned about in the Tradeshows. |
Process |
You have about 90 minutes for this activity including lunch. As a group, first, update each other on what you learned in the tradeshows. Then, develop a model that matches the knowledge, attitudes and skills under your competence ( Patient Care ) to the assessment tools and modalities you learned about today. Use the following questions to stimulate your conversations:
Use the white walls and markers provided to brainstorm and develop your model. |
Exchange |
After you finish this activity you will have an opportunity to share your work with the other groups. |
Our focus was Patient Care. We talked about data gathering, clinical reasoning and patient management.
The minimum assessment level for the board will include a global assessment (experience specific) and a direct observation component. We want to move away from a single form for every type of clinical rotation. We want to emphasize narrative comments over numeric scores. Evaluators should only answer questions about which they have direct experience with the resident. The direct observation should use mini-CSX and focus on communication, PE and reasoning.
We recommend several other assessment tools beyond the minimum. We want to evaluate recorded performances. We want to employ simulated scenarios. And we want to use MSF to evaluate students in stressful situations.
To make this evaluation system work, we need to change the culture. We need our evaluators to observe their residents all of the time and write down their observations frequently.
We did not finally agree on whether to implement this system all at once or phase it in over time. A phased approach would allow the grumblers to get used to the idea gradually. An all-at-once approach would mean that in three years, that would be “the way it always was” as far as any of the current residents would know.
Assignment: Click here to open assignment in Microsoft Word
| Context | In your group there are people that attended all five of the tradeshow presentations in the last round of work. You have been provided a list of the knowledge, skills and attitudes developed in the first colloquium mapped to the program requirements – plus any additional knowledge, skills and attitudes added in the round of work on the Patient Perspective. You've also been provided a document that defines the ACGME competencies and a document that highlights the pediatric practice competencies as background information. Your group will focus on the Professionalism competence. |
Objective |
The objective for this round of work is to match the knowledge, attitudes and skills to the assessment tools and modalities you've learned about in the Tradeshows. |
Process |
You have about 90 minutes for this activity including lunch. As a group, first, update each other on what you learned in the tradeshows. Then, develop a model that matches the knowledge, attitudes and skills under your competence ( Professionalism ) to the assessment tools and modalities you learned about today. Use the following questions to stimulate your conversations:
Use the white walls and markers provided to brainstorm and develop your model. |
Exchange |
After you finish this activity you will have an opportunity to share your work with the other groups. |
Our team focused on Professionalism. This is one of the most difficult skills to test for. We included skills like self-assessment, medical and professional ethics, cultural competency, child advocacy, the giving and receiving of feedback, etc.
We can use several tools. The portfolio tool is a good tool for self-assessment, but we are concerned about unmotivated residents and evaluators. The 360 Feedback has some strengths, but we're concerned about the fairness of evaluating only the resident – we are also concerned about a hyper-evaluation environment. The mentorship model is interesting. Parent satisfaction can be used to provide performance feedback. We would like to provide real-time feedback for either positive or negative professional behaviors. We need to do a lot more training for our faculty before we implement these tools.
It seems as though most revocations of licenses come about because of unprofessional behavior. We would like to introduce other ways to encourage proper professional behavior. We could require a certain number of CMEs in ethics, for example.
NBME.org is launching a project focused on professionalism. There is a good resource there.
We have talked about several issues today that spread across several competencies, but I'm not sure that they would show up within any of the competencies. The responsibility for taking care of the patients is one of those issues – perhaps it should fit into the Professionalism category.
Should the portfolio be used to identify professional qualities in med students who are applying to residency programs? There may be a conflict of interest for med schools whose primary interest is getting their students into residency programs.
Who controls the portfolio? The student controls the portfolio, but they assign rights of access to portions of the portfolio to the med schools and residency programs.
Assignment: Click here to open assignment in Microsoft Word
Context |
In your group there are people that attended all five of the tradeshow presentations in the last round of work. You have been provided a list of the knowledge, skills and attitudes developed in the first colloquium mapped to the program requirements – plus any additional knowledge, skills and attitudes added in the round of work on the Patient Perspective. You've also been provided a document that defines the ACGME competencies and a document that highlights the pediatric practice competencies as background information. Your group will focus on the Communication competence. |
Objective |
The objective for this round of work is to match the knowledge, attitudes and skills to the assessment tools and modalities you've learned about in the Tradeshows. |
Process |
You have about 90 minutes for this activity including lunch. As a group, first, update each other on what you learned in the tradeshows. Then, develop a model that matches the knowledge, attitudes and skills under your competence ( Communication ) to the assessment tools and modalities you learned about today. Use the following questions to stimulate your conversations:
Use the white walls and markers provided to brainstorm and develop your model. |
Exchange |
After you finish this activity you will have an opportunity to share your work with the other groups. |
Our task was to look at Communication. Our first question was whether this skill is really broken. We do not teach the science of communication. And we tend to teach “Leadership” (being the boss) and not “leadership” (how to contribute in lots of ways to a team).
We generally like the portfolio tool. Feedback happens in real time, not six months later. Feedback comes from multiple sources, which implicitly reflects the team-based nature of medicine. If we were to use a 360 review process, we should tell residents that the review would take place “sometime this year”, not on a specific date.
We need to introduce a lot more training for faculty. We want to rate the change in residents' skills over time. We need to have standardized assessors, and we should train the faculty in how to do this. We want to introduce a “communication SWAT team” – this does not have to by physicians, this can be a team of observers focused on communication skills. We have some concerns about the security of the data in a portfolio. Work-based assessments are very similar in some ways to the faculty training.
We have used motivational interviewing as a curriculum-based tool. This is a tool that physicians can use to help evaluate whether their patients understand their treatment plan and whether they are motivated to implement it.
It may be valuable to have recording equipment in certain parts of the clinical environment. Everyone should know that the devices may occasionally be on “for quality or training purposes”.
It will be critical to have a cultural shift around communication. Assessment should be done with the residents, not to the residents.
Assignment: Click here to open assignment in Microsoft Word
Context |
In your group there are people that attended all five of the tradeshow presentations in the last round of work. You have been provided a list of the knowledge, skills and attitudes developed in the first colloquium mapped to the program requirements – plus any additional knowledge, skills and attitudes added in the round of work on the Patient Perspective. You've also been provided a document that defines the ACGME competencies and a document that highlights the pediatric practice competencies as background information. Your group will focus on the Systems Based Practice competence. |
Objective |
The objective for this round of work is to match the knowledge, attitudes and skills to the assessment tools and modalities you've learned about in the Tradeshows. |
Process |
You have about 90 minutes for this activity including lunch. As a group, first, update each other on what you learned in the tradeshows. Then, develop a model that matches the knowledge, attitudes and skills under your competence ( Systems Based Practice ) to the assessment tools and modalities you learned about today. Use the following questions to stimulate your conversations:
Use the white walls and markers provided to brainstorm and develop your model. |
Exchange |
After you finish this activity you will have an opportunity to share your work with the other groups. |
We focused on “Systems Based Practice”. We spent a good amount of time trying to define this term. We started with Paul's model of ten components for a system-based practice and made a few changes. Some of the components are:
Identify and prioritize changes
How well does the system work?
Recognize the need and implement a multidisciplinary system
Recognize how the system interacts with others – simulations
Recognize the role of finance
How is medical education conducted and how successfully? There are a variety of tools for this.
There is a systems-based process that we go through every day called “Discharge Planning”. The morbidity and mortality conference is another great tool for assessing the systems.
Assignment: Click here to open assignment in Microsoft Word
Context |
In your group there are people that attended all five of the tradeshow presentations in the last round of work. You have been provided a list of the knowledge, skills and attitudes developed in the first colloquium mapped to the program requirements – plus any additional knowledge, skills and attitudes added in the round of work on the Patient Perspective. You've also been provided a document that defines the ACGME competencies and a document that highlights the pediatric practice competencies as background information. Your group will focus on the Practice Based Learning competence. |
Objective |
The objective for this round of work is to match the knowledge, attitudes and skills to the assessment tools and modalities you've learned about in the Tradeshows. |
Process |
You have about 90 minutes for this activity including lunch. As a group, first, update each other on what you learned in the tradeshows. Then, develop a model that matches the knowledge, attitudes and skills under your competence ( Practice Based Learning ) to the assessment tools and modalities you learned about today. Use the following questions to stimulate your conversations:
Use the white walls and markers provided to brainstorm and develop your model. |
Exchange |
After you finish this activity you will have an opportunity to share your work with the other groups. |
We focused on Practice-Based Learning and Improvement. We tried to match the tools we learned about in our tradeshows with components of practice-based learning. We feel that portfolios would be a very valuable tool in the future, but they are not yet developed to the point of being functional. Lifelong learning would be well served by secure exams, self-reflection tools and multiple choice questions. We need to focus on learning within the system, not just the individual – we would like to find more tools for evaluating the system. Critical Thinking using data and evidence-based medicine would benefit from chart reviews, multiple-source feedback, etc.
We would like to create an IT model that employed a station that tested how well a person could use IT to solve a particular challenge. We want to develop tools to measure the value and quality of collaboration and collaborative improvement.
We added a couple of skills. Transparency is very important, but we are not sure exactly how it would fit in to practice-based learning. We are not sure how to assess leadership training skills.
Assignment: Click here to open assignment in Microsoft Word
Context |
In your group there are people that attended all five of the tradeshow presentations in the last round of work. You have been provided a list of the knowledge, skills and attitudes developed in the first colloquium mapped to the program requirements – plus any additional knowledge, skills and attitudes added in the round of work on the Patient Perspective. You've also been provided a document that defines the ACGME competencies and a document that highlights the pediatric practice competencies as background information. Your group will focus on Faculty Development for implementing new assessment tools and modalities. |
Objective |
The objective for this round of work is to develop a model for faculty development for incorporating and implementing these assessment tools and modalities. |
Process |
You have about 90 minutes for this activity including lunch. As a group, first, update each other on what you learned in the tradeshows. Then, develop a model for faculty development to be able to implement the assessment tools and modalities you learned about to assess the knowledge, attitudes and skills from the Delphi . Use the following questions to stimulate your conversations:
Use the white walls and markers provided to brainstorm and develop your model. |
Exchange |
After you finish this activity you will have an opportunity to share your work with the other groups. |
We focused on Faculty Development. To have an effective program, we need to establish the need for faculty development. Most of our faculty are just slogging their way through their daily responsibilities – we need to build the argument for them to invest their time in development. We could offer them a certain number of credits towards MOC for faculty who engage in faculty development and who use these assessment tools.
It will require several sea-changes for this to happen. It will be a major change to actually use all of these assessment tools. We need to get vastly better at the giving and receiving of effective feedback – we tend to be pretty timid. We need to decide whether we want to distribute tools nationwide from a centralized source or allow each school to experiment on its own. We need to shift towards a centralized source. We are not sure which faculty need to know how much about the development and use of different assessment tools. We want to focus on a core team with advanced education degrees to really champion these changes.
The ABIM has developed a prototype of this very system. This has been a very tough sell for our board of directors. If we are successful, we will pilot it in select programs in 2008.
One of the barriers to engaging faculty in resident observation is their incentive system that pushes them to make money for the program.
Assignment: Click here to open assignment in Microsoft Word
Context |
In your group there are people that attended all five of the tradeshow presentations in the last round of work. You have been provided a list of the knowledge, skills and attitudes developed in the first colloquium mapped to the program requirements – plus any additional knowledge, skills and attitudes added in the round of work on the Patient Perspective. You've also been provided a document that defines the ACGME competencies and a document that highlights the pediatric practice competencies as background information. Your group will focus on implementing new assessment tools and modalities for Directed Lifelong Learning . |
Objective |
The objective for this round of work is to develop a model for directed, monitored, lifelong learning that incorporates and uses the assessment tools and modalities you learned about this morning. |
Process |
You have about 90 minutes for this activity including lunch. As a group, first, update each other on what you learned in the tradeshows. Then, develop a model for directed, monitored, lifelong learning that uses the assessment tools and modalities you learned about this morning to assess the knowledge, attitudes and skills from the Delphi. Use the following questions to stimulate your conversations:
Use the white walls and markers provided to brainstorm and develop your model. |
Exchange |
After you finish this activity you will have an opportunity to share your work with the other groups. |
We believe that assessment and feedback does not need to be as frequent as it is in residency. This kind of assessment needs to begin in residency. We want to take advantage of pediatric offices with in-house residents. Resident pediatricians need to be able to give and receive feedback before they leave residencies. We have not put enough emphasis yet on managing uncertainty – this is a very important part of lifelong learning. Self-assessment is very important, but it also must be credible – this requires an honest outside observation and feedback. We really like the mentor/coach model. We must set up residency to give the residents more insights and control over their own learning from the very beginning.
The coaching program in the UK was set up in 2001. All doctors have been required to meet with a peer once a year to evaluate a portfolio and to set up a personal development plan for the next year. There has been a lot of resistance to this program. It works better in some institutions and with some assessors. It seems to work better in a coaching model, not a punitive model. Participation in the UK increased dramatically when it became required in order to increase your pay scale. Only engagement in the process is required, not positive evaluations. One of the most beneficial features of the program is simply the 1-2 hours set aside to reflect on the previous year.
We need to present all of these assessment tools as a benefit to physicians, not as a punitive tool.
