Table 1 Design decisions made during human-centered design

 a) Summary measures

n patients, n patients by area, n patients by shift, n patients by discharge disposition

To give physicians an overview of the patients they saw in a month. Show the area of care and shifts as it might affect case mix

 b) Length of stay metrics

Provider to decision time, overall LOS^a (median)

Overall LOS is most important for ED, but provider-decision time is easier to influence by provider

‘Provider to decision time’ changed to ‘Room to decision time’ as patients might see another provider in the waiting area (iteration 1)

 c) Utilization of tests

CT, MR, US, lab utilization (%)

Can be affected by physician and is known to affect LOS

 d) Outcomes

72-h return rates, deaths (%), LOS after admission (median)

To address concerns about negative outcomes, return-rates and deaths were included. LOS after admission was included as a proxy measure of appropriateness of admissions

Removed deaths as an outcome as it is not feasible to reliably obtain data from EMR (iteration 1)

 a) Over time

Monthly intervals

Balance between too frequent reports with random variation and too infrequent where physicians don’t remember what happened

 b) To peers

Blinded ranking (e.g. ‘your ranking 46/60’, with outcomes of peers with better or worse numbers shown).

Overall ED medians are shown on separate page

Blinded since all interviewed physicians agreed un-blinded was not desired/needed. A ranking showing neighboring peers was included to give physicians an attainable goal

Changed the ranking to interquartile range of peers instead, since the optimum rate is likely someplace in the middle, outliers in either direction can be a problem (iteration 1)

 a) Ease of access

Monthly email summary with 3 measures that can be selected by ED leadership based on priorities

Easy to access

 b) Drilldown functionality

Option to access full dashboard through a link in monthly email

Drilldown functionality

Added tabs to drill down based on the type of shift (e.g. night) and assigned area. (iteration 1)

 c) Customization

Physicians can select measures to show up on their own favorites page

Leadership can select measures in monthly email

Customization options for individual physicians and leadership based on ED priorities

 a) Adverse consequences on quality of care

Inclusion of outcomes on dashboard

To avoid focus only on throughput and utilization measures, which might result in adverse consequences

 b) Conflicting teaching responsibilities

–

As this was not the goal of the dashboard, no measures related to teaching were included

 c) Data accuracy

–

Extensive validation of data is required

Included definition of the measures on the dashboard.

Made section headers very clear (iteration 2)

 d) Case-mix adjustment

Show total number of patients during different shifts and in different areas of care.

By showing these measures, physicians can put other measures in context.

Added tabs to drilldown by area of care and type of shift (iteration 1)