Table 3 Guideline-based QI development reporting standard items and report of these criteria in the method papers

1a. Guideline selection: criteria

Selection of guidelines was based on topic and

• evidence-based development (n = 18)

• methodological quality of the guideline (n = 2)

• use of GRADE (n = 1)

• structured format (n = 1)

• no other criterion (n = 1)

• unclear (n = 7)

1b. Guideline selection: appraisal of guidelines

• AGREE (n = 8)

• criteria not fully specified (n = 4)

• not reported (n = 18)

2. Selection of guideline recommendations

• based on topic (n = 2)

• impact on patient outcome (n = 4) / burden of illness (n = 1) / clinical utility (n = 1) / available treatment (n = 1)

• relevance (n = 4) / appropriateness (n = 1)

• value for money (n = 1)

• practice variability (n = 1)

• scope for improvement (n = 1) / gap in quality of care (n = 1)

• priority / feasibility for implementation (n = 3)

• validity (n = 2) / reliable (n = 1)

• (high) level of evidence (n = 8) / adequate scientific proof (n = 1)

• direct link to aim of guideline (n = 1)

• common to more than one guideline (n = 1)

• unclear (n = 3)

• no selection (n = 6)^a

3. Selection of performance measures from recommendations

• formal panel method (n = 11)

• other or informal consensus method (n = 13)

• not reported (n = 2)

• unclear (n = 4)

4. Core attributes of performance measures (criteria for selecting QI)

• relevance (n = 4)

• potential for improvement (n = 9) / likely to change current practice (n = 2) / gap in quality of care (n = 2) / importance for health care (n = 4)

• burden of illness (n = 2) / improving patient outcomes (n = 9)

• cost-effectiveness (n = 4)

• influenced by service provider (n = 3)

• appropriateness (n = 1)

• evidence base/scientific soundness (n = 7)

• (strength of) association with patient important outcome (n = 2)

• feasibility (n = 7)

• no risk for unintended consequences (n = 3)

• unambiguous definition (n = 2) / clear (n = 1)

• data routinely collected (n = 1)

• measurable (n = 4) / interpretable (n = 1) / actionable (n = 2)

• applicable (n = 3) / acceptable (n = 1) / adherence (n = 1)

• reliable (n = 6) / face validity (n = 2) / construct validity (n = 1) / content validity (n = 2)

• precision (n = 1)

• minimum bias (n = 1)

• not reported (n = 4)

• unclear (n = 3)^a

5. Specification of performance measures

Numerator and denominator is specified unambiguously and in detail.

• denominator: population eligible to receive the clinical interventions, numerator: desired intervention and subset of population that should receive it (n = 6)

• based on algorithm (n = 1)

• formulation of numerator and denominator in line with formulation of recommendation (n = 1)

• numerator and denominator including risk adjustment factors (n = 3)

• clinical researcher drafted an expanded text for each recommendation, using logical operators (e.g. ‘AND’ and ‘OR’) to link descriptive statements to produce numerators and denominators (n = 1)

• method not specified in detail (n = 15)

• not reported (n = 3)

6. Intended use of performance measure

• quality improvement (n = 10)

• quality of care delivered (n = 2)

• monitoring compliance with guideline (n = 4)

• implementation of care (n = 1)

• clinical audit (n = 1)

• pay for performance program (n = 1)

• not specified (n = 8)

• unclear (n = 1)

• not reported (n = 2)

7. Practice test of performance measures

• planned (n = 18)

• retrospective (n = 2)

• implicit (n = 1) / ad hoc (n = 1)

• not reported (n = 8)

8. Review and evaluation of performance measure

• plan for evaluation and updating (n = 3)

• evaluation including criteria for retiring (n = 1)

• mentioned, but not explained in detail (n = 2)

• evaluation not reported, often because QI were developed but not yet implemented (n = 24)

9a. Composition of the panel

• monodisciplinary (n = 2)

• multidisciplinary (n = 23)

• panel composition not reported (n = 5)

9b. Composition of the panel: patient involvement

• yes (n = 10)

• no (n = 17)

• depends on guideline (n = 1)

• not reported (n = 2)