Evaluation of non-response to the In-Center Hemodialysis Consumer Assessment of Healthcare Providers and Systems (ICH CAHPS) survey

Abstract

Background

The In-Center Hemodialysis Consumer Assessment of Healthcare Providers and Systems (ICH CAHPS) Survey is the first patient reported outcome measure included in the U.S. Medicare End Stage Renal Disease Quality Incentive Program. Administered twice yearly, it assesses in-center dialysis experience and survey responses are tied to dialysis facility payments. Low response rates, currently approximately 35%, raise concern for possible underrepresentation of patient groups.

Methods

Cross-sectional analysis of survey administration in 2012 to all in-center hemodialysis patients in Dialysis Clinic, Inc. (DCI) facilities nationally over 18 years old who received hemodialysis at their facility for at least 3 months. Patient-level covariates included demographic, clinical, laboratory, and functional characteristics. Random effects multivariable logistic regression was used to assess survey non-response.

Results

Among 11,055 eligible patients 6541 (59%) were non-responders. Of the remaining 4514 responders, 549 (14%) surveys were not usable due to presence of proxy help or incomplete responses. Non-responders were more likely to be men, non-white, younger, single, dual Medicare/Medicaid eligible, less educated, non-English speaking, and not active on the transplant list; non-responders had longer ESRD vintage, lower body mass index, lower serum albumin, worse functional status, and more hospitalizations, missed treatments, and shortened treatments. Similar associations were found using more parsimonious multivariable analyses and after imputing missing data.

Conclusions

Non-responders to the ICH CAHPS significantly differed from responders, broadly spanning individuals with fewer socioeconomic advantages and greater illness burden, raising limitations in interpreting facility survey results. Future research should assess reasons for non-response to improve ICH CAHPS generalizability and utility.

Background

Patient experience is an integral part of patient-centered care. Multiple factors influence patient experience, including characteristics of the facility, interactions with care teams, patient expectations, and response to or complications of treatment. Interest in measuring patient experience dates to Health Effectiveness Data and Information Set (HEDIS) measures evaluating this in the early 1990s in the United States [1]. However, non-response bias and low response rates complicate measurement of patient experience [2,3,4].

The US Agency for Healthcare Research and Quality (AHRQ) developed consumer assessment surveys starting in the 1990s. The Centers for Medicare and Medicaid Services (CMS) in conjunction with AHRQ began developing the In-Center Hemodialysis Consumer Assessment of Healthcare Providers and Systems (ICH CAHPS) survey in 2004 [5]. After field testing in 2005, ICH CAHPS was endorsed by the National Quality Forum (NQF) in 2007 and was incorporated into the End Stage Renal Disease Quality Incentive Program (ESRD QIP) as the first patient reported outcome measure in 2014 [6,7,8]. Mandatory twice yearly survey administration began in 2016, and facilities with at least 30 annual responses are subject to financial penalties for lower patient experience scores.

Critically, there may be informative differences among patients who complete and do not complete the ICH CAHPS survey that may result in misrepresentation of overall patient experience at a dialysis facility; however, despite its incorporation into value-based payments several years ago, little is known about characteristics of responders and non-responders. Response rates during development and validation of ICH CAHPS were only 46% [8], despite conditions being optimized during this development process. Response rates have continued to drop since the ICH CAHPS has been implemented in the clinical setting, even while the financial and public reporting importance of this assessment of patient experience has increased [9, 10]. As in other areas of medicine, understanding presence of bias and the subsequent generalizability of a test is of utmost importance when interpreting test results and prior to implementing change. Accordingly, we performed the first step in the evaluation of non-response bias by exploring patient characteristics associated with non-response to the ICH CAHPS survey administered in 2012 to patients treated at Dialysis Clinic, Inc. (DCI) facilities nationally.

Methods

Study population

Per 2012 AHRQ guidelines, ICH CAHPS eligible patients consisted of all in-center HD patients at least 18 years old who had been at their facility for at least 3 months. Responses from eligible patients were deemed usable only if patients indicated receiving no proxy help and at least 50% of pre-defined key questions were answered (Additional file 1: Box 1) [11].

Survey

The ICH CAHPS survey administered in 2012 had 58 questions and was available in English and Spanish. Responses were grouped into three composite scores and three global rating scales. The three composite scores were ‘Nephrologists’ Communication and Caring’, ‘Quality of Dialysis Center Care and Operations’, and ‘Providing Information to Patients’; these composite scores were derived from questions that used either ‘never/sometimes/usually/always’ responses or ‘yes/no’ responses. The three global rating scales rated nephrologists, dialysis center staff, and the dialysis facility on a scale of 0-10 (with 0 being worst and 10 being best). The remaining survey questions asked about demographic characteristics, comorbid medical conditions, and whether or not help was received in answering the survey questions.

Survey administration

Dialysis facilities were required to select third party vendors to administer the ICH CAHPS survey. DCI’s survey vendor followed AHRQ guidelines for survey administration, data collection, and data submission. Before the survey administration period, DCI in-center HD facilities received staff and patient education materials describing AHRQ survey administration requirements. AHRQ requirements did not allow dialysis provider, facility staff or physician involvement in survey administration or in the collection of results. As instructed by the survey vendor, on August 3, 2012 DCI created a data file of eligible patients from its electronic medical information system containing mailing addresses, telephone numbers, and primary language. Approximately 10 days later, the survey vendor mailed patients a pre-notification letter on DCI letterhead, signed by a member of the DCI executive team. The letter informed patients that they would receive a survey regarding the care they received at their dialysis facility and that their responses were very important. One week later, ICH CAHPS surveys were mailed to all potentially eligible patients by the survey vendor. Patients were instructed to mail completed surveys directly back to the survey vendor in pre-paid and addressed envelopes. Two weeks after the first survey mailing, the survey vendor sent a reminder letter to non-responders to the first mailing, and another copy of the survey 30 days after the first survey mailing. In October, the survey vendor contacted patients who had not replied to either of the mailed surveys by telephone up to three times over a 4-week period.

Study design

DCI has over 200 dialysis facilities nationally. Their survey vendor provided patient-level data from the 2012 survey period to DCI exclusively for quality improvement and research purposes under a signed Respondent Identifiable Information Disclosure Agreement. A member of the DCI information technology team who was independent from the research team merged survey data to individual patient DCI electronic medical data and de-identified the dataset. The primary study outcome was non-response to the ICH CAHPS survey. In primary analyses, only surveys meeting AHRQ’s definition of usable (no proxy help and answers to at least 50% of pre-defined key questions) were included (Fig. 1). In secondary analyses, survey response was defined using an “expanded usable” criteria which included AHRQ usable surveys as well as surveys without 50% of pre-defined key questions answered and surveys indicating proxy help (Additional file 1: Figure S1a). For surveys indicating proxy help, we only included surveys where the patient checked off receiving help from a family member or friend and checked off any of the following describing the help they received: “Read the questions to me,” “Wrote down the answers I gave,” or both. This definition is more consistent with current ICH CAHPS scoring rules.

Fig. 1

Flow diagram

To account for missing data, we performed multiple imputation of missing covariates for use in sensitivity analyses. Following imputation, we compared patients with AHRQ usable responses to non-responders (Additional file 1: Figure S1b), and we compared the “expanded usable” group of responders to non-responders (Additional file 1: Figure S1c). The study was approved by the Tufts Medical Center Investigational Review Board.

Clinical characteristics

Patient clinical characteristics ascertained from DCI medical records included patient demographics, medical records, clinical variables, information on HD treatments, and functional assessments. Since the precise date of survey completion is not known, all covariate data were taken from August 2012 (when eligible patients were identified by DCI and information was sent to the survey vendor). All laboratory analyses were performed at the central DCI laboratory in Nashville, TN. For missing August laboratory data, we used the last non-missing value within 3 months prior to August 2012. Specifically for missing vascular access data in August, we used the most frequently used vascular access in May, June and July 2012. Unexcused absence was defined as missing an entire HD treatment that was not rescheduled and for which a reason (e.g. hospitalization) was not available; shortened treatments were defined as at least one treatment being shorter than prescribed by 15 min or more, and hospitalizations were defined as any hospital stay. Body mass index (BMI) was calculated using the last estimated dry weight ordered by the patient’s nephrologist. Data on functional covariates including ability to ambulate, ability to transfer, falls in the past month, activities of daily living (ADL) score, and nursing home residence were obtained from the nursing assessment most proximate to the survey administration period. The ADL score was derived from 8 questions from the nursing assessment evaluating the patient’s ability to bathe, dress, feed, use the toilet, shop for groceries, prepare meals, do housework, and take medications; 1 point was given for each activity that the patient could independently perform, and 0 points were given if assistance of any type was needed.

Statistical analysis

We used a random intercept two-level logistic regression model with patients nested within dialysis facilities to model the probability of non-response using the AHRQ definition. The Intra-class correlation (ICC) was calculated using the latent variable model approach [12]. The unobserved patient variable follows a logistic distribution with individual level variance V_I equal to π²/3. On this basis, the ICC is calculated as ICC=V_A/(V_A + π²/3) where V_A is the facility residual variance on the logistic scale. Models were fitted sequentially starting with a parsiminous model using primarily demographic data; subsequent models added clinical and functional data. Secondary analyses used the same covariates in the “expanded usable” cohort defined above. Multivariate multiple imputation for missing covariates was performed using chained equations, averaging five models using Rubin’s rule [13]. The imputation model included response status and variables listed in Table 1, with the exception of ability to ambulate, ability to transfer, history of falls, ADL score, and nursing home residence. To account for the multilevel nature of the data, the imputation model included dummy variables for each dialysis facility. We checked for functional forms of all continuous variables using restricted cubic splines in the rms package in R. There were no statistically significant deviations from linearity for any continuous variable. To measure the overall importance of each variable in the multivariable model, we plotted the ranked Chi-squared minus degrees of freedom for each variable [14]. Analyses were performed using SAS Enterprise Guide (Version 7.12, Cary, NC) and R language (version 3.3.1, R Foundation for Statistical Computing, Vienna Austria).

Table 1 Baseline characteristics

Results

Study population

There were 11,463 patients initially identified by DCI as meeting AHRQ-defined eligibility for the ICH CAHPS survey. Of these, 22 died during the survey administration period and 386 were deemed ineligible based on responses to eligibility screening questions in the survey meant to confirm ongoing in-center HD treatments at their HD facility for at least 3 months. The latter was probably a combination of incorrect initial identification, modality switch after identification, and inaccurate response from patients to the screening questions. Among 11,055 AHRQ eligible patients, 6541 (59%) did not return the survey or answer phone calls from the vendor. Of these non-responders, an additional 1169 (18%) patients were excluded in our primary analysis because of missing data on at least one covariate (Fig. 1). The response rate per facility ranged from 0 to 100% of eligible patients and narrowed to 15-61% for facilities with at least 30 survey eligible patients (Fig. 2). Of the 4514 (41%) patients who completed the survey, 643 (14%) responses could not be scored because of indicating proxy help or not completing at least 50% of the AHRQ key questions. Of all patients who provided any response, 596 (13%) were excluded in our primary analysis because of missing data on at least one covariate.

Fig. 2

Distribution of Response Rates per Facility

Primary analyses

Non-responders, based on AHRQ criteria, differed from responders and from those who had incomplete responses or proxy help in demographic, clinical, and functional characteristics (Table 1). In adjusted analyses, non-responders as compared to responders were more likely to be men, non-white, younger, single, dual Medicare/Medicaid eligible, less educated, and non-English speaking. Non-responders had longer ESRD vintage, were more likely to be inactive on the kidney transplant list, and had lower BMI and lower serum albumin. Non-responders had worse functional status, more hospitalizations, missed treatments, and shortened treatments (Table 2). Race, serum albumin concentration, and education level were the three most influential variables predicting non-response (Fig. 3).

Table 2 Multivariable logistic regression models predicting non-response

Fig. 3

Ranking of variable contribution for determining non-response. Derived using data from model 2 (without functional covariates) shown in Table 2. ESRD: End stage renal disease; BMI: Body mass index. ESRD: End stage renal disease; ADL: Activities of daily living; BMI: Body mass index

Secondary analyses

We included 549 survey responses with complete covariate data that either indicated receiving proxy help or did not respond to at least 50% of AHRQ pre-defined key questions (Additional file 1: Figure S1a). Overall, these 549 patients differed from AHRQ-defined responders in several demographic, clinical, and functional characteristics (Table 1). By including these surveys, we gained an average of 5% more responses for most demographic covariates (Additional file 1: Figure S2). Factors that predict non-response to the survey were similar to primary analyses when including this expanded response group (Additional file 1: Table S1).

Sensitivity analyses

The majority of missing data was on functional covariates (Additional file 1: Table S2). Patients with missing covariate data were more often black and had shorter ESRD vintage (Additional file 1: Table S3). Models using multiple imputation for missing data had similar results to the primary and secondary analyses (Additional file 1: Tables S4 and S5).

Discussion

In a large national in-center HD population, non-responders to the ICH CAHPS survey differed substantially from responders. Specifically, non-responders were more likely to be men, non-white, younger, single, dual Medicare/Medicaid eligible, less educated, non-English speaking, inactive on the transplant list, and had longer ESRD vintage, lower BMI and lower serum albumin, worse functional status, and more hospitalizations, missed treatments, and shortened treatments. These results demonstrate underrepresentation of important groups of in-center HD patients, broadly spanning individuals with fewer socioeconomic advantages and greater illness burden. It is possible that these results could introduce biases into facility-level ICH CAHPS survey results, particularly given low overall response rates, resulting in missed opportunities to assess and improve patient experience among the most vulnerable hemodialysis patients.

CAHPS surveys are widely used in US medical settings to evaluate patient experience, with other CAHPS surveys targeting hospitals, nursing homes and other settings. The ICH CAHPS is unique as it evaluates facilities with a relatively low number of patients per facility and with longstanding patient-facility relationships rather than discreet episodes. There is limited published literature on characteristics of non-responders to other CAHPS surveys. Most importantly, previous assessments use only limited patient-reported characteristics unlike our study where we use extensive characteristics gathered using reliable data sources rather than patient self-report. Even so, similar to our findings, analysis of Medicare Managed Care (MMC) CAHPS survey from 1997 and 1999 found significantly higher non-response rates in participants who were male and non-white [15]. Likewise, analysis of the Hospital CAHPS (HCAHPS) pilot survey data from 2002 to 2003 also found male sex, younger age, and non-white race to be significantly associated with non-response [16]. Finally, in a large sample of Medicare CAHPS participants from 2007 there were significantly higher non-response rates in participants who were men, non-white, and dual eligible for Medicare and Medicaid [17].

These results have important implications for assessment of dialysis patient experience. The DCI patient population is similar to the broader US dialysis population with the exception that there is a slightly higher proportion of black patients receiving dialysis care at DCI [18]. To our knowledge, this is the only non-CMS dataset linking individual patient-level clinical data to ICH CAHPS responses, and this is the first study assessing the differences in characteristics, including laboratory variables and treatment characteristics, between responders and non-responders. Previous published work on the ICH CAHPS survey is restricted to reports on the development and testing of this survey, where there was a response rate of 46% and there was no published evaluation of non-responders [8, 19]. Within one of these study cohorts, response rates were noted to be higher among those assigned to mail followed by telephone versus telephone only survey administration [20]. Although supervision of ICH CAHPS administration was transferred from AHRQ to CMS in 2014, the current survey remains similar to the one administered in 2012, with the major exception that limited assistance is now allowed, consistent with the ‘expanded usable’ criteria used in secondary analyses in this manuscript.

Payers increasingly are moving towards value based purchasing models, with performance metrics critical to quantify value. Before the addition of ICH CAHPS as a performance metric, the ESRD QIP was composed of only clinical and laboratory measures, most of which were not specifically patient-centered outcomes [21]. Patient experience measures have been widely implemented in other areas of healthcare, and use of the ICH CAHPS survey represents an important milestone for in-center HD; however, attempts to address patient-centered care using a patient-reported outcome measures with low response rates may have limitations. Paradoxically, we found that non-responders tended to be patients who are disproportionately represented in the US ESRD population as compared to the general population (specifically younger, black, male, and diabetics) [18]. These differences in characteristics associated with non-response raise the possibility of non-response bias; however further research is needed in evaluating whether or not these characteristics are also associated with experience scores and will thereby affect facility performance ratings and performance-based payments as well as misrepresent key areas needed for intervention to improve patient experience [22].

The specific reasons for non-response remain unknown. Neither the former AHRQ nor the current CMS administration process collects reasons for non-response unless it is due to incorrect contact information. Comorbid conditions common among dialysis patients include physical, cognitive, and visual impairments that may limit the ability of HD patients to respond to a survey themselves. Accordingly, and particularly in view of the survey’s length (currently 62 questions), the initial decision by AHRQ to not allow any assistance may have had important implications. Using a less restrictive method of classifying survey completion, more consistent with current CMS guidance, we were able to include approximately 5% more responses across most demographic characteristics; notably, inclusion of these surveys did not change the predictors of non-response.

Our results may have substantial implications for dialysis facilities if characteristics associated with non-response are also associated with experience scores. Starting in calendar year 2016 (and reflected in 2018 payments to facilities), survey results are a clinical performance measure within the ESRD QIP and experience scores can impact facility payments from CMS [9].

An important strength of this study is that it documents new information about the real-world administration of the ICH CAHPS survey. Additionally, this study provides information that can no longer be gathered since survey vendors are now barred from providing patient-level data to dialysis facilities. Other strengths include having a large number of survey responses from a national dialysis provider linked to extensive facility gathered patient-level demographic, clinical, and functional data. Limitations include not knowing the precise date of survey completion during the survey administration period, which required the use of proximate covariate data. As with most surveys, we do not have information on reasons for non-response.

ICH CAHPS survey response rates remain low overall (only 33% despite allowing limited assistance with survey completion [10]). Future studies should provide ongoing evaluation into the presence of and reasons for non-response to this survey to inform strategies for engaging populations that have a greater likelihood of non-response, specifically patients with greater illness burden and fewer socioeconomic advantages, in order to improve the generalizability and utility of surveys of patient experience.

Conclusion

There are significant differences between ICH CAHPS survey non-responders and responders that potentially affect facility-level assessment of and efforts to better patient experience. Further work should evaluate causes of non-response and interventions to increase response rates in an attempt to optimize assessment and improve patient experience.