Creating and Applying an Evaluation Framework for the National Decision Support Programme in Scotland

Digital Health & Care Institute (DHI); Cresswell, Kathrin and Callaghan, Margaret and Mozaffar, Hajar and Sheikh, Aziz. (2019) Creating and Applying an Evaluation Framework for the National Decision Support Programme in Scotland. Digital Health & Care Institute, Glasgow. (https://doi.org/10.17868/67863)

[thumbnail of Cresswell-etal-DHI-2019-an-evaluation-framework-for-the-national-decision-support-programme-in-scotland]
Preview
 Text. Filename: Cresswell_etal_DHI_2019_an_evaluation_framework_for_the_national_decision_support_programme_in_scotland.pdf
Final Published Version
Download (1MB)| Preview

Abstract

Context: The Scottish Government recognises the importance of decision support to improve knowledge management in health and care settings as a strategic priority. To this end, they funded the 2015 National Decision Support Roadmap. This laid out a plan for procuring and building a Decision Support Platform delivering a range of small-scale demonstrators (including several mobile platforms for specific user groups e.g. polypharmacy and diabetes), and building clinician and policy engagement for further funding. Aims: We were commissioned to undertake a formative evaluation of the National Decision Support Programme to help facilitate the effective roll-out of systems included in the Roadmap more widely. Methods: We collected qualitative data through a series of in-depth interviews and observations of workshops demonstrating technological systems. Participants included policy makers and clinical leads involved in the National Decision Support Programme. As the Programme was in the early stages of strategy development and system implementation at the time of data collection, we focused on exploring expectations and drivers of Cambio (a pilot platform) being tested in primary care. This system delivers an open standards based algorithm editor and engine which is linked with bespoke decision support applications delivered as web and mobile products and integrated into primary care electronic health record systems. The web and mobile solutions linked to the Cambio algorithms platform were developed by Scottish partners (Tactuum and University of West of Scotland). Employing a flexible methodological approach tailored to changing circumstances and need offered important opportunities for realising true impact through ongoing formative feedback to policymakers and active engagement of key clinical stakeholders. Our work was informed by sociotechnical principles and a health information infrastructure perspective. Qualitative data were coded with the help of NVivo software and analysed through a combination of inductive and deductive approaches. Findings: We collected data through 30 interviews and eight non-participant ethnographic observations of early stakeholder engagement workshops. We developed and applied a theoretically-informed evaluation framework, which we refined throughout our analysis. Overall, we observed a strong sense of support from all stakeholders for Cambio as an exemplar of an open standards based, customisable decision support platform, and proposals to roll this model out across NHS Scotland. Strategic drivers included facilitating integration of care, preventative care, patient self-management, shared decision-making, patient engagement, and the availability of information. However, in order to achieve desired benefits, participants highlighted the need for strong national leadership, system usability (which was perceived to be negatively affected by alert fatigue and integration with existing systems), and ongoing monitoring of potential unintended consequences emerging from implementations (e.g. clinical workloads). Conclusions and implications: In order to address potential tensions between national leadership and local usability as well as unintended consequences, there is a need to have overall national ownership to support the implementation of the Roadmap, whilst the implementation of individual applications needs to be devolved. This could be achieved through allowing a degree of local customisation of systems and tailoring of alerts, ongoing system development with continuing stakeholder engagement including “hands-on” experience for clinicians, a limited number of pilots that are carefully evaluated to mitigate emerging risks early, and development of a nuanced benefits realisation framework that combines smaller and locally relevant measures determined by implementing sites with national progress measures.

CORE (COnnecting REpositories)