A report evaluating the success of the pathfinder pilot in introducing revalidation has been released.
The pilot involved ten sites where over 3,000 doctors undertook strengthened medical appraisal. The evaluation process involved surveying doctors and their employers to establish their experience with the current appraisal system as a baseline, and then conducting a follow-up survey of their experiences with strengthened medical appraisal.
The following are the key findings from this evaluation - which was prepared by Frontline Consultants and Durham University for the Department of Health and the Revalidation Support Team - along with recommendations for the future:
1. There was a statistically significant increase in the levels of agreement with the statement: “I improved the way I deliver care as a result of my appraisal” for acute and mental healthcare. In acute care, the number of appraisees participating in significant event/case reviews increased from 47% to 75% during the pilot.
2. There was confidence to make revalidation decisions on the basis of the information provided, as 58% of the responsible officer respondents agreed with the statement “I would be confident to make revalidation recommendations based on all of the information provided”.
3. Around half of the organisations will be looking to enhance the way that doctors undertake continuing professional development (CPD) including guidance on the most appropriate CPD and monitoring its effectiveness. As CPD is seen by appraisers as one of the most valuable activities for evaluating doctors’ standards of practice and planning how to improve patient care, this is a potential benefit in a key area.
4. Nearly two-thirds of the organisations will be making changes to ensure that remediation is successful, with greater formalisation of the process and evaluating the effectiveness of those processes.
5. A simplified system is needed. Evidence from the evaluation suggests that, whilst some attributes were easier than others, providing supporting information on each of the 12 appraisal attributes was difficult for the appraisees. The attributes that doctors found most difficult to provide supporting information for were: show respect for patients; treat patients and colleagues fairly and without discrimination; and, acting with honesty and integrity.
Work should be undertaken to establish which attributes are necessary for appraisal, and to rationalise the information requirements for those attributes. Definitive guidance is required on what supporting information appraisees should provide for each attribute.
6. The time required to prepare for the appraisal increased. In preparing for the appraisal, doctors were required to search out supporting information and become familiar with both the strengthened medical appraisal system and the accompanying toolkit. This took additional time compared to the previous appraisal system.
Preparation time was shorter at University Hospitals of Leicester NHS Trust pilot site, where an in-house toolkit was used, and this should be investigated further to see what lessons can be learned for the national roll out.
7. Application of common standards to appraisal is valued. Doctors valued a common approach to appraisal that would be the same wherever they practised. The report recommends that any future appraisal system should contain enough specification to ensure commonality of outcomes.
8. Responsible officers welcomed the oversight of appraisal information. The toolkit provided responsible officers with the ability to look at appraisal information, and make sure that appraisers had appraised their appraisees in a suitable manner.
9. The quality of appraisals improved during the pathfinder pilot. Feedback from the evaluation indicates that the requirements for appraisees to prepare and reflect on their practice and for appraisers to review material beforehand had been strengthened. Appraisees’ views on their appraisers have became more positive during the pilot.
10. Pathfinder pilot organisations and responsible officers were very positive about the potential benefits of revalidation, whilst appraisees and appraisers were less so. Understanding why there are such differences in perceptions and addressing them would help in the roll out of both a future appraisal system and revalidation.
11. Working within local networks can reduce the risk of responsible officers having conflicts of interest. Responsible officers felt that they should not act as responsible officers for every doctor in their organisation (e.g. where they work closely with colleagues). They suggested that it could be addressed by having local networks of responsible officers.
12. Responsible officers need continuous access to a range of information to make recommendations - not just an annual appraisal. Information systems should be implemented such that responsible officers receive high quality information on a continuous basis so that they are aware of emerging issues with doctors’ practice.
13. Key areas for evaluating doctors’ standards of practice have been identified. Appraisers considered continuing professional development and audits/informal data review as the most valuable activities in evaluating doctors’ standards of practice. In acute and mental health care, appraisers also considered feedback from colleagues to be valuable for evaluating doctors’ standards of practice.
The future appraisal system should be prioritised and focused on the areas that are most significant for evaluating doctors’ standards of practice.
14. Key areas for planning how doctors can improve patient care have been identified. Appraisers considered audits/informal data review and significant event/case reviews to be the most valuable activities in planning how to improve patient care. In acute and mental health care, appraisers also considered review of complaints to be valuable.
The report recommends that the future appraisal system should be prioritised and focus on the areas that are most significant for planning how doctors can improve patient care.
15. Locum doctors are not always able to access supporting information. Evidence from the evaluation suggests that it is difficult for locum doctors to access supporting information. Systems should be developed such that locum doctors get access to the supporting information required to meet the appraisal requirements.
16. The report concludes that the best practices achieved by organisations in the pathfinder pilot should be identified and the lessons learned shared with organisations nationally.
Read the whole report.
Tags: Revalidation
