|
 |
| ORIGINAL ARTICLE |
|
| Year : 2018 | Volume
: 7
| Issue : 6 | Page : 1458-1463 |
|
|
Workplace-based assessment of family medicine competencies using “field note tool” – A pilot study
Asha E Mathew1, Yeshvanth Kumar2, Ruby P Angeline2, Prince Christopher3, Sajitha P Rehman2, S Venkatesan2
1 Department of Family Medicine, Community Health and Development, Christian Medical College, Vellore, Tamil Nadu, India
2 Department of Family Medicine, Low Cost Effective Care Unit, Christian Medical College, Vellore, Tamil Nadu, India
3 Department of Family Medicine, Shalom Family Medicine Clinic, Christian Medical College, Vellore, Tamil Nadu, India
| Date of Web Publication | 30-Nov-2018 |
Correspondence Address:
Yeshvanth Kumar
Department of Family Medicine, Low Cost Effective Care Unit, Schell Eye Campus, Arni Road, Christian Medical College, Vellore - 632 001, Tamil Nadu
India
 Source of Support: None, Conflict of Interest: None  | Check |
DOI: 10.4103/jfmpc.jfmpc_141_18
Background: Department of Family Medicine in a medical college in South India introduced “field note” (FN) as a tool for Work-Place Based Assessment in postgraduate training. FN collects “open-ended” feedback from both resident and faculty and helps them to arrive at an action plan. This study describes our experience of implementing FN and perceptions of learners and faculty. Methods: While precepting the residents in Family Medicine service areas, faculty documented their observations of the resident's clinical work using FN and provided an action plan. Faculty and residents described their experience and provided feedback. Focus group discussions were conducted for faculty and residents. Data were coded and grouped into themes. Results: Four residents and seven faculties participated in the study during 12 weeks period using 17 consultations. Clinical expert (13/17) and communicator (6/17) are the most commonly assessed competencies followed by professionalism (2/17) and collaborator (2/17). Faculty and residents agreed that “FN” was a useful tool and it helped the faculty to give feedback and guide the learner. Residents and faculty arrived at an action plan in 70% of the consultations. Three of four residents perceived the change in their behaviour positively after the use of FN. Both resident and faculty found the rating of the learner using Dreyfus scale as a barrier. Conclusion: FN could be one of the important tools in our “Toolbox of Assessment Methods” for family medicine specialty. There is a need for sensitizing the learners to feedback process and training the faculty in assessment and feedback.
Keywords: Competencies, family medicine, field note tool, formative assessment, postgraduate education
How to cite this article:
Mathew AE, Kumar Y, Angeline RP, Christopher P, Rehman SP, Venkatesan S. Workplace-based assessment of family medicine competencies using “field note tool” – A pilot study. J Family Med Prim Care 2018;7:1458-63 |
How to cite this URL:
Mathew AE, Kumar Y, Angeline RP, Christopher P, Rehman SP, Venkatesan S. Workplace-based assessment of family medicine competencies using “field note tool” – A pilot study. J Family Med Prim Care [serial online] 2018 [cited 2021 Mar 2];7:1458-63. Available from: https://www.jfmpc.com/text.asp?2018/7/6/1458/246454 |
| Introduction | |  |
“Assessment drives student learning” is a known axiom in medical education. Even though assessments use more resources, it improves motivation of the learner and helps the faculty and learner to achieve educational outcomes.[1],[2] While summative assessment is effective in motivating students to improve their performance toward certification, it cannot be used to provide feedback and guide the learner when a deficiency is identified.[3] Work-Place Based Assessment (WPBA) assesses the “does” level of competence in Miller's pyramid. It is done in an authentic environment and in the context where the resident is going to work in future. It is not merely intended to assign a grade to the learner but to provide feedback and guide the learner. This should be an ongoing activity, based on direct observation by the faculty. It clarifies the goals with the learner and helps faculty to diagnose and correct specific mistakes in learning and behavior.[4]
Competency-based education is being introduced in family medicine postgraduate training across the world.[5] Workplace-based assessments are considered more authentic for assessment of competence and performance in real life. There are many tools available for WPBA like Mini-clinical evaluation exercise, video-review, case-based discussion, multi-source feedback, and portfolio.[6],[7] Field note (FN) is a new tool that has been described by Ross et al. They used FN as the primary assessment tool in the Competency-Based Achievement System in the University of Alberta, to provide feedback to the residents.[8]
FN is a paper or electronic document in which both resident and faculty reflect on the consultations done by the resident [Table 1]. Open-ended feedback is collected about the competencies demonstrated by the resident in the clinical encounter. FN helps the resident to arrive at an action plan in discussion with the faculty. In FN, the learner needs to reflect first followed by the faculty and this is documented and stored for later review. | Table 1: Reflective questions in the Field note tool; (created based on the Competency-Based Achievement System, used by University of Alberta in Edmonton[8]
Click here to view |
This study is aimed at introducing “field note” as a tool for workplace-based assessment in family medicine and collecting the perceptions of the learners and faculty.
| Method | | |
After obtaining ethics committee approval and informed consent, four residents and seven faculties were sensitized to the FN tool and the definitions of competencies in family medicine. Over 12-week study period (March to June 2016), the faculty observed and documented the clinical consultations by the resident in the FN and rated the competencies using “Dreyfus five stage model of adult skill acquisition.”[9] The seven competencies that were assessed are the clinical expert, communicator, professional, collaborator, system-based practitioner, scholar, and leader. These competencies were defined by the faculty attending the National Faculty Development Workshop organized by National Board of Examinations in Christian Medical College, Vellore in February 2016 and were adopted from ACGME Family Medicine Milestone project [Table 2]. The ratings used were (ascending from lower to higher level of performance) novice, advanced beginner, competent, proficient, and expert where novice represents the learner without any knowledge about the competency and expert represents the learner who has demonstrated the maximum level of performance in that competency. In our study, we used a paper-based two-page FN document that was simple to use and could be filed for future reference.
Faculty provided feedback about the selected few competencies relevant for each consultation and helped the learner to arrive at an action plan to progress in their learning curve. Faculty and learners were asked to describe their experience of using the FN and rate the usefulness of FN for assessing the learner and for giving appropriate feedback using the Likert scale.
Both faculty and residents suggested changes for improvement and mentioned about action plan they provided or received. Focus group discussions were conducted for faculty and residents to collect their perceptions and challenges in using the FN. The data were coded and grouped into themes. The following questions were asked to focus group: describe your feelings and experiences on using FN, any challenges or barriers you faced while using FN, and any suggestions to make it a better tool.
| Results | | |
Four learners and seven faculties participated in the assessment using FN during 12 weeks study period, using 17 consultations. “Clinical expert” was the commonly assessed competency (13 of 17 consultations) followed by “communicator” (6/17), “collaborator” (4/17), “professional” (2/17), “scholar” (2/17), and “system-based practitioner (2/17).” “Leader”as a competency was never assessed during the study period. In each consultation, the faculty assessed one or more competency, decided by the faculty.
“Advanced beginner”was the most common rating given by the faculty in 7 of 17 consultations, followed by competent (5/17) and novice (2/17). Residents were never rated as “proficient” or “expert” in any of the consultations. In 3 of 17 encounters, there was no rating mentioned at all. In about 70% of (12 of 17) consultations, residents were given action plan. Three of four residents perceived a positive change in their behavior in achieving the family medicine competencies.
All faculty and learners agreed that the FN is a useful tool for providing feedback and guiding the learner. Overall, FN was found to be a useful tool for assessment of the learner's competence. Almost half of the residents were “hesitant” to agree on the usefulness of the FN [Figure 1], [Figure 2], [Figure 3]. [Table 3] and [Table 4] describe the selected quotes from the faculty and learners, respectively. The quotes are grouped into strengths and challenges/barriers.
Suggestions from the faculty and resident
The following are few suggestions from the faculty;
- “There is a need for entry level assessment of the competencies in the PG student”
- “Systems are needed for behavioral change in resident when a problem is identified”
- “Action plan needs to be followed up”
- “If we could make the documentation online and review periodically, it would be better.”
The following are the suggestions from the resident:
- “Give time to reflect with the resident”
- “No need of rating of the student”
- “Field note should be done for the same condition/problem over few weeks if possible to assess whether the learner has improved based on the feedback by the faculty”
- “A learner needs to be followed up. If a learner is Novice in one topic, he could be followed up with multiple field notes till he is competent on that topic”
- “Remove the Novice and advanced beginner. Rather use stage 1, 2, 3, and 4”
- “Is it possible to counter check the action plan? Whether the action plan is achieved?”
- “One faculty could see all the field notes according to the subject and according to the competencies and provide overall feedback to the resident.”
| Discussion | | |
Any assessment method should be aligned with the content of the training program and all the desired competencies of the specialty. Assessment is also expected to have an educational impact.[10] In postgraduate education in India, more emphasis is given for summative assessment using theory examination and practical clinical case discussions. In these high-stakes summative examinations, only a few competencies such as clinical expertise and communication are assessed. But there is less opportunity to assess important competencies like the collaborator, leader, scholar, and system-based practitioner which are needed for the practice of medicine.
This pilot study found that “FN” can be used to assess most of the competencies in family medicine. Leader as a competency was never assessed in our study. That could be explained by the short duration of the study. Leader is a complex competency and it needs multiple observations to make a reliable judgement. Formative assessment using FN is a way forward in implementing competency-based medical education.
FN was accepted as a tool for workplace-based assessment both by learners and faculty, and it is cost-effective. But the learners had difficulty in accepting the negative feedback and the rating as novice or advanced beginner. They had received more negative feedback than positive. This could be one of the reasons for their hesitation in agreeing on the usefulness of FN [Figure 1], [Figure 2], [Figure 3]. The residents have felt the differences in the ways they are assessed by different faculties [Table 4]. There is a need for developing definitions of milestones of competencies and faculty development in assessment and feedback to reduce the observer bias.
Educational impact of the FN was perceived by the residents during the focus group discussion [Table 3] and [Table 4]. The utility of any assessment tool depends on its validity, reliability, educational impact, acceptability, and cost. We did not measure the “validity” of the tool in this study. Validity refers to whether the instrument measures what it is expected to measure. According to Dr. Van der Vleuten, validity depends on the authenticity of measurement and integration of competencies during measurement.[11] FN was used to assess all competencies, in any clinical context: being outpatient, in-patient care, emergency care, and home care. FN can be used to assess the consultation skills, procedural skills, ward rounds, counselling skills, and academic presentations. The fact that FN is used in the authentic workplace and the whole task is assessed without trivialization of the competencies, FN can fulfil the criteria for validity.
“Reliability” is the reproducibility of the scores achieved in assessment.[11] One can argue that assessment by FN has the subjective bias as it is open-ended without objective scores and this may negatively affect reliability. Van der Vleuten says the reliability of an instrument depends on careful sampling across the content and not based on objectivity or standardization. To reduce the observer bias and improve the reliability, we decided to collect the FN of each resident given by different faculties at different context and file it for review by the course coordinator once in 4 months to assess the resident's progress. A feedback will be given to the resident about the performance in the previous 4 months and also suggestions will be given to achieve the desired milestones in the next 4 months.[8]
Limitations
This is a pilot study done over a short period in one institution in India. Findings of this study need to be validated in a larger multicentric study done in multiple settings with many institutions training family medicine residents in India.
| Conclusion | | |
FN could be one of the important tools in our “Toolbox of Assessment Methods” for family medicine specialty when we introduce competency-based postgraduate medical education in India. There are few challenges observed in using the FN. Rating the learner using Dreyfus levels was the most important barrier that needs to be addressed during implementation. There is a need for sensitizing the learners to feedback process and training the faculty in assessment and feedback.[13]
Acknowledgement
The authors would like to thank all the four residents who were pursuing Fellowship in Family Medicine in Christian Medical College, Vellore, and who participated in the study.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
| References | | |
| 1. | Ferris H, Flynn DO. Assessment in medical education; what are we trying to achieve? Int J Higher Educ 2015;4.  |
| 2. | Bennett, RE. Formative assessment: A critical review. Assessment in Education Principles, Policy & Practice 2011;18:5-25. |
| 3. | Raupach T, Brown J, Anders S, Hasenfuss G, Harendza S. Summative assessments are more powerful drivers of student learning than resource intensive teaching formats. BMC Med 2013;11:61. |
| 4. | Thomas JW, Megan SK, Lutana HH. Formative assessment in health care education. Int J Educ Soc Sci 2014;1. |
| 5. | Tannenbaum D, Kerr J, Konkin J, Organek A, Parsons E, Saucier D, et al. Triple C Competency-Based Curriculum. Report of the Working Group on Postgraduate Curriculum Review – Part 1. Mississauga ON: College of Family Physicians of Canada; 2011. |
| 6. | Epstein RM. Assessment in medical education. N Engl J Med 2007;356:387-96. |
| 7. | Yousuf Guraya S. Workplace-based assessment; applications and educational impact; Malays J Med Sci 2015;22:5-10. |
| 8. | Ross S, Poth CN, Donoff M, Humphries P, Steiner I, Schipper S, et al. Competency-based achievement system: Using formative feedback to teach and assess family medicine residents' skills. Can Fam Physician 2011;57:e323-30. |
| 9. | Dreyfus SE. The five-stage model of adult skill acquisition. Bull Sci Technol Soc 2004;24:177. |
| 10. | Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007;29: 855-71. |
| 11. | van der Vleuten CP, Schuwirth LW. Assessing professional competence: From methods to programs. Med Educ 2005;39:309-17. |
| 12. | |
| 13. | |
[Figure 1], [Figure 2], [Figure 3]
[Table 1], [Table 2], [Table 3], [Table 4]
|
Search Pubmed for