Table 1 Cross-site table of activities, measures of successes, common elements, and lessons learned across 7 years of implementation experience in Mozambique, Rwanda, and Zambia for promoting data-driven quality improvement

Plan

1. Ensure high-quality routine data exist through facility-level DQA of RHIS data

2. Training in data review, analysis with systematized data matrices

1. Average routine data concordance achieved >80%

2. Demonstrated competence in visualization and analysis; ascension from stage 0 to stage VI of Berwick’s stages (see Fig. 2)

1. Mentoring on DQAs at facility and district level

2. Training on data review, quality improvement, and analysis

3. Determining best ways for performance review and feedback

1. Increased community health worker data quality [43]

2. Demonstrated use of data to drive new change concepts

3. Collaborative development of MESH program [25]; ascension from stage 0 to stage V of Berwick’s stages (see Fig. 2)

1. Collaboratively create new tools for input, visualization, and management of health data, including training in analysis/interpretation

2. Pre-implementation facility assessment around supplies, pilot DQA, and clinic support workers

1. New tools in place, operating, and staff demonstrating competence with new procedures

2. Minimum standards met as evaluated in pre-implementation assessment; ascension from stage 0 to stage V of Berwick’s stages (see Fig. 2).

1. Began activities after known gaps in data quality and use

2. DQA activities were conducted first

3. Trainings in data analysis and interpretation

4. Tools for systematized data visualization, feedback, and supervision

1. Begin with DQAs to improve analysis skills, promote that change is possible, and ensure high-quality data

2. Ensure Ministry of Health engagement and ownership to allow flexibility in health system protocols and ensure sustainability.

3. Progress to “Do” step once achieving stage IV or V on Berwick’s stages

Do

1. Begin district-level data review and feedback meetings

2. Create collaborative and iterative data-driven action plans

3. Supervision from provincial and district teams to follow-up action planning

1. 56 meetings conducted from 2012 to 2015

2. 498 collaborative action plans implemented and iteratively reviewed

3. Supervision conducted through project period to all, and more intensive supervision to low-performing districts and facilities

1. Integrate data-driven performance review and feedback into management meetings at facility, district, and province.

2. Implement MESH program [25]

3. Field-based operations research training and Masters in implementation science

1. Increased use of data feedback to make decisions in meetings

2. Change in performance measures across areas of focus [25]

3. Number of peer-reviewed papers published and degrees awarded

1. Basic infrastructural upgrade and equipment

2. Formally implement clinic support worker tasks

3. Use new data tools to conduct automatic patient tracking and real-time indicator monitoring.

4. Dedicated QI mentoring teams at district level provide supervision and performance review with feedback

1. Functioning equipment and EMR system to provide real-time data collection and feedback

2. Support workers successfully completing tasks

3/4. Performance indicators being tracked in real-time and reviewed during mentorship visit by QI teams

1. Used iterative performance review and feedback from within the system – collaboratively – instead of an external “audit”

2. Used supervisors working within the government system to build sustainability and engender a culture of data-use, change, and shared accountability.

1. Using Ministry of Health supervisors is essential to ensure a culture of collective improvement and performance review instead of negative feelings of external “audit”

2. Relying on external funding for advanced data technology can undermine sustainability and government ownership

Study

1. Re-visit action plans and performance data during next meeting

2. Conducted qualitative evaluation of meetings among 21 facility staff and 21 managers/supervisors

3. Formal evaluation forthcoming

1. Equal number of meetings and action plans each year incorporating feedback from previous cycles, along with data-driven targeting of resources and supervision

2. Need for more systematized review and support, and suggestions to conduct meetings quarterly instead of bi-annually.

3. Evaluation will be published separately Fall 2016.

1. Assess data quality improvements

2. Assess the effects of data-driven QI changes instituted

3. Assess the culture of data use

1. Observed improvements in RHIS data quality [28]

2. Observed improvements in data-driven QI [29, 30]

3. Qualitative evidence of increase in health workers’ value and ownership of data; capacity building efforts to increase use and value of programmatic data [31]

1. Indicators of data timeliness, form completeness, adherence to care protocols tracked

2. District-level QI teams conducted chart reviews to test data quality and validity

3. Monthly targets for supervision visits

1. Median time from consultation to data visible was 40 h; improvements from 8.4% blood pressure measurements to 81.5%, among others.

2/3. QI teams conducted 2.5 monthly chart review/mentoring visits per facility

1. Use performance review and feedback data to feed back into intervention to improve quality of data, quality of QI change concepts, quality of practice, management, and supervision

2. Use key tracer indicators to allow assessment of data quality yet avoid too much burden

1. Future projects should devise implementation measures and easy ways to track success or failure of action plans and build these performance measures into overall program

2. Staff turnover is major challenge which impacts continuity of data review, action planning, and QI change concepts.

3. Engaging higher-level managers and supervisors can help avoid changing higher-level priorities that can eliminate space for change at lower levels

Act

1. Pilot testing structured action plan monitoring tool

2. Meetings expanded to pharmacy, malaria, and tuberculosis

3. Development of new IDEA project to focus on intensive measurement and scale-up to Manica province

1. Data forthcoming on pilot test

2/3. Proposal funded by DDCF to scale-up meetings, including more in-depth monitoring and process/impact evaluation for next 5 years, including an additional 7 districts.

1. Work with district and national program to expand lessons learned on DQAs

2. Analysis of implementation needs for MESH and adaptation for national spread

3. Plans to sustain field research training, plans developed for spread to other settings

1. Ongoing DQA at district-level; routine publication on quality at national level

2. Scale-up of MESH nationally for priority areas; development of toolkit for spread to other countries

3. Spread of implementation research to other PIH-supported sites; increase in implementation research focused degrees; number of new publications and new primary authors

1. Target additional supportive supervision or resources to low performing facilities or clinicians

2. Identify data gaps and create revised tools

3. Replacement of obsolete or non-functional equipment

4. Policy-level feedback on performance of the system

1. Additional support sought from new funder to sustain intervention

2. New forms created for neonatal care, and revisions done to existing tools to enhance efficiency

3. Number of facilities sustaining all QI activities

4. Challenges identified and corrective action plans discussed with leadership

1. Iterative feedback at multiple levels of the health system

2. Continued efforts to mobilize resources in order to improve/sustain intervention

1. Continued and active engagement with the Ministry is critical

2. Ministry may not have resources to sustain entire QI intervention activities – active engagement will help identify “core” components required for sustainability of effects