Skip to main content
  • Research article
  • Open access
  • Published:

Assessing value in health care: using an interpretive classification system to understand existing practices based on a systematic review

Abstract

Background

Implementing adequate strategies to assess the value of health services plays a central role in the effort to deal with the financial pressures faced by health care systems worldwide. This study aimed to understand which approaches to value assessment have been used in developed countries.

Methods

We conducted a rapid review and a gray literature search to identify value assessment frameworks. A two-stage screening process was utilized to identify existing approaches and cluster similar frameworks. In addition, we developed an interpretive classification system to make sense of existing approaches.

Results

One thousand one hundred seventy-six references were identified and 38 papers were selected for full-review. Among these 38 articles, 22 distinct approaches to assess value of health care interventions were identified and classified according to four points: 1) use of single or multiple considerations to base value estimates; 2) use of disease-specific or generic criteria; 3) reliance on process-based or outcomes-based consideration; and 4) type of input and evidence considered.

Conclusions

The contextual nature of value assessment in health care becomes evident with the diversity of existing approaches. Despite the predominance of cases relying on the Incremental cost-effectiveness ratio as the measure of value, this approach has not been sufficient to meet the needs of decision-makers. The use of multiple criteria has become more and more important, as well as the consideration of patient-reported measures. Considerations of costs are not always explicit and consistent.

Peer Review reports

Background

The unsustainable growth of health expenditures observed in developed countries over the last decades has sparked a variety of initiatives worldwide envisioning adequate allocations of the scarce resources available. Economic evaluations of care interventions, experiments involving payment schemes, campaigns for appropriateness of care targeting clinicians, novel purchasing strategies, among so many others, there seem to exist initiatives targeting virtually every aspect of the health care system. In recent years, the term value has occupied a central spot in this constellation of efforts toward wiser spending of money.

Assessing the value (often referred to as the ‘value for money’) of health care technologies, in particular the novel ones, has received an enormous attention from researchers, policy-makers and the industry. In countries with single-payer systems, the economic evaluation of technologies acquired a paradigmatic status, although its practice has been often incomplete or inconsistent. In the United States, where co-exist a market-oriented system of private insurers/providers and public programs (distinct among themselves, such as Medicare, Medicaid and the VA system), the notion of value has been yet incorporated but it has gained more and more attention in the latest years, despite the unclear lieu of value-based strategies in general.

Notwithstanding the disseminated use of the term ‘value’, it has not always been employed with the same meaning. The most commonly accepted definition of value in health care is provided by Michael Porter as “health outcomes achieved per dollar spent” [1]. This description is conceptually situated within the realm of technical efficiency, concerning about the maximization of objective gains in health in relation to a given amount of financial resources. In defining value at the ISPOR Special Task Force Report, in turn, Garrison et al. point out that “from an economic perspective: the ‘gross value’ can be thought of what someone would be willing to pay for an economic good or intervention, whereas the ‘net value’ subtracts the opportunity cost incurred to obtain that gross value” [2], p124). In the latter definition, on contrary, value evocates a connotation of allocative efficiency.

Although Garrison et al. argue that the underpinning notion of value does not depend on the nature of the health care system (whether a market-based, a social insurance or a single-payer system), that might be true from a normative perspective, but it does not seem the case considering the diversity of ways the term value is employed and the distinct approaches it motivates. The corollary of this observation is that, apart from the unsurprising idea that the value of health care technologies is contextual for many reasons (the perspectives included in the assessment, the societal values, the relative price of labor and technology, etc.), the proper concept of value seems to be contextually sensitive.

The aim of the present study was to understand what strategies of value assessment have been developed and implemented worldwide. In addition, we sought to comprehend the contexts in which these initiatives emerged and what definition of value underpinned it. In order to address these research questions, we performed a rapid literature review and a gray literature search focusing on existing value assessment frameworks of health care technologies within the context of developed countries.

Methods

Two-stage screening scientific literature review

A comprehensive search of the peer-reviewed literature published between 2007 and 2017 was conducted using Ovid MEDLINE, an extensive database of public health journals with a powerful platform for building searching strategies. The specific search strategy, presented in Additional file 1, resulted in the identification of 1176 references. Following execution of this search, we used a two-stage screening process.

The first stage consisted of selecting articles for full-review and the grouping of similar approaches in preliminary heuristic categories. Titles and abstracts were screened with the following two inclusion criteria: 1) does the paper describe a way to assess value of a treatment, service, intervention or technology? And 2) does the paper describe an applied framework (i.e., was the value assessment approach actually implemented)? In addition, we excluded papers that: 1) deal with animal health; 2) describe means to assess the value of management systems, administrative procedures, payment schemes, data management solutions, training programs, human resources schemes, or solutions for health care supply chains and purchasing contracts; or 3) that are not full articles (i.e., editorial, interview, commentary).

All 1176 titles/abstracts were reviewed by one reviewer and 335/1176 (28%) were also reviewed by a second reviewer. The agreement rate between the two reviewers on the first screening was over 95%, with discrepancies settled by discussion. There were no cases where a third reviewer was required. In total, 157/ 1176 papers were initially screened ‘in’ for further review. In order to make sense of the data and help in the understanding of the big picture, each of these 157 references were placed into one of six initial descriptive categories: a) CEA/CUA; b) simple consideration of costs and outcomes; c) unspecified value approach; d) multiple criteria approach; e) net economic/social value; and f) comparative effectiveness.

The second stage of screening was meant to further refine and hone the initial six categories listed above. In this stage, we were working to saturation, so we sought to include papers that described empirical activity of approaches to value assessment while excluding papers there were simply ‘more of the same’. In addition, we sought to identify any particularly innovative or novel aspects in the implementation of a given approach. This second screening resulted in the selection of 38 articles to which we applied our data extraction tool (see Additional file 2) for classification and evaluation of value assessment approaches.

Gray literature search

To complement our peer review literature search, we also conducted a gray literature search. Two main strategies of environmental scan were used. First, we relied on previous systematic reviews of initiatives around disinvestment and/or reassessment of low-value and/or potentially obsolete health technologies to ensure that we are getting at value in the context of resource management [3,4,5,6,7,8]. A search was conducted on the websites of institutions found in these reviews (see Additional file 3). The terms used for the search on each of the identified websites were: low value, disinvestment, reassessment, de-adoption, decommissioning and delisting. The second search strategy consisted of exploring the websites of highly reputable HTA agencies and other relevant professional organizations for presentations, guidelines, working papers or any other pertinent piece of gray literature. For this purpose, we looked at the following organizations: Health Technology Assessment International (HTAi); International Network of Agencies for Health Technology Assessment (INAHTA); International Society for Pharmacoeconomics and Outcomes Research (ISPOR); European Network for Health Technology Assessment (EUnetHTA); International Health Economics Association (IHEA); Agency for Health Research and Quality (AHRQ); Canadian Agency for Drugs and Technologies in Health (CADTH); Kaiser International Health Group; and Blue Cross Blue Shield Association. A total of 1390 documents were identified, of which 52 qualified for full review.

Results

Figure 1 presents the two-stage screening process. We found 22 distinct approaches to assess the value of health care interventions, which were reported in the 38 papers from the peer-reviewed literature search. The gray literature search returned only methodologies and discussions on processes to identify and prioritize low value or obsolete technologies for further evaluation but no particular approach to assess their value per se was identified.

Fig. 1
figure 1

Literature search process and results

No gold standard value assessment framework was reported despite the supposedly theoretical supremacy of cost-effectiveness analysis. The existing strategies found here are diverse and vary according to their context and specific objectives. Given the large and heterogenous number of frameworks, our narrative synthesis process involved the development of a classification system to categorize the methodologies found. Our analytical scheme is based on the key structuring questions observed across the existing frameworks of value assessment.

The classification we propose is built around four questions: 1) Is the method based on one consideration/criterion or multiple considerations/criteria? 2) Are the considerations/criteria disease-specific or more generic? 3) Are the considerations/criteria process-oriented or are they directly based on patients’ perspective? 4) What input is primarily used to measure performance and what is the acceptable ‘evidence’?

These four questions expand on the two fundamental issues that must be addressed in any value assessment exercise: what is the entity to be measured and how is it to be measured.

Question #1: one consideration or multiple considerations?

Based on the nature of the health care service to be assessed, one has to determine the relevant criteria to guide decisions. Most scenarios, if not all, are complex and involve several components and objectives. However, it is not always feasible and practical to take into consideration all relevant aspects pertaining to an intervention. In these cases, the use of ‘one consideration’ seems the most common choice, having cost-effectiveness/cost-utility analysis (CEA/CUA) as the most prevalent framework. There is a broad evidence base supporting the use of CEA in assessing value. It should be noted that with CEA/CUA, outcomes can be defined in many different ways, from life years gained to symptoms avoided to units of clinical benefit. On the other hand, if multiple considerations are deemed relevant, then a myriad of approaches may be employed. Although the majority of the value assessment frameworks falling within the multiple consideration category draw some elements from the set of tools broadly known as multi-criteria decision analysis (MCDA), it is rare to see a consistent, structured and explicit discussion of these theoretical elements of decision sciences.

It is important to explain the choice of using the term ‘consideration’ instead of ‘criterion’ or ‘criteria’. Even though ‘criteria’ is indeed an appropriate term for most of the value assessment frameworks identified here, which basically consist of judging an intervention according to a principle, we found that the term is not entirely suitable for some approaches. Thus, the word ‘consideration’ was chosen as a way to avoid semantic inadequacies in some cases, but these terms can be used interchangeably.

Question #2: disease-specific or more generic consideration?

The key in addressing this question is determining whether or not there is a need for generalizability across a range of interventions. If the need for a particular assessment of value is for the purpose of informing choices between similar interventions, then disease-specific criteria are preferable because they will by nature be more directly relevant and the results will typically be more precise. If the decision-making applies to a broader context across different areas of care and levels of organization, then more generic criteria are more appropriate. From an economic perspective, this question relates to whether the decision at hand is one of technical efficiency (i.e., given a decision to treat a certain patient group, what is the best treatment option) or of allocative efficiency (i.e., how to allocate resources among interventions aiming different patient groups).

Question #3: process-based or patient-oriented outcomes?

This question relates to the point in the health care system where value is assessed. If one is looking at the entire service continuum, then a patient outcomes perspective seems intuitive. On the other hand, if the focus is on a specific element within that continuum, such as rehabilitation, or post-surgical care, then adherence to clinical guidelines might be a more appropriate way to assess value as it attempts to isolate the given intervention from the influence of other parts of care. Patient reported outcomes have been around for many years but recently have received greater attention across all health systems.

Question #4: what input and what ‘evidence’?

This question refers to the nature of information used to address the previous questions and the types of evidence acceptable to base these answers. The answer to this fourth question depends in part on the choices made in questions two and three. For example, process-oriented considerations are more likely going to be data-driven whereas certain health outcomes might be possibly best assessed by expert opinion, including, where possible, patients’ feedback. In addition, disease specific contexts are more likely to have established measurement tools. In contrast, when looking at value assessment across disease areas there is likely only lower level evidence available as there are going to be fewer ‘head to head’ studies of disparate treatment areas. In these cases, expert opinion may probably come to play a role in the assessment.

Classification summary

In order to illustrate how this classification might work, take for example the well-known ASCO framework. In this case, for the first question, benefit gains are measured against more than one consideration; for the second question, the focus is on cancer so the criteria lean more towards disease-specific than generic, without being so specific as to apply to only one cancer; for the third question, the criteria focus on health outcomes; and finally, for the fourth question, input is from clinical trial results with the addition of expert opinion, as necessary. Table 1 presents the 22 approaches organized as per our classification system. It must be noted that some of the approaches are, by their nature, very specific in their content while others are made to adjust to the context where they are applied. For example, the ASCO framework contains specific criteria with specific weights while ‘MCDA’ – as a more generic approach - covers any set of criteria developed for any given implementation.

Table 1 Classification of existing value assessment frameworks

Our classification is not making any normative statements, rather it is meant as a guide to help in understanding the myriad of value assessment approaches that are currently reported in the literature. Using the four questions to classify the 22 approaches that were identified in the peer-reviewed literature we found that 18/22 use multiple criteria, 12/22 use criteria that are more disease-specific in nature, 17/22 focus on outcomes-oriented criteria and 12/22 have performance measurements that are mostly data-driven.

In addition, explicit process evaluation took place in about one third of the 38 papers reporting on the various approaches. Key evaluation findings across these studies included the need for transparency in both criteria and methods, the relevance of expert panels to support and contextualize data, and in many (but not all) cases, the application of the value assessment approach was found to lead to changes either in resource use or clinical outcomes.

Discussion

All studies found in the literature review identify the depicted methodologies as strategies or frameworks for assessing value of health care technologies. However, only a few explicitly state their underlying concept of value. Some articles that report cost-utility analysis agree with Porter defining value broadly as “health outcomes obtained per dollar spent”. Govaert et al. [39] provide a similar but more specific definition of value in a paper on surgical auditing, understanding value as “the health outcomes achieved that matter to patients, relative to costs of achieving those outcomes”. In the DCE experiment conducted by Green and Gerard [26], they sought to estimate the ‘social value’ of health care technologies, in which construct the notion of ‘value for money expected from the treatment’ is a component. It can be said that it represents an attempt to obtain a measure of allocative efficiency that rely on individuals’ judgement on the evidence around technical efficiency. ASCO defines value “as a combination of clinical benefit, side effects, and improvement in patient symptoms or quality of life in the context of cost”.

Despite the lack of a consensual definition of value and the diversity of frameworks as well as contexts where these emerge, we are able to identify two main constructs across the existing initiatives that represent the key challenges in measuring what is achieved by an intervention: what are the outcomes that are to be considered and how is the level of each outcome measured. We expect that each condition will have a unique set of relevant outcomes and, for any condition, the relative importance of each outcome may vary across patient groups. For example, some patient groups may be more risk averse and for those, the rate of adverse events is a more important outcome than to less risk adverse patients. A further complication in the assessment of the value of an intervention is that, when the focus is on outcomes, the contribution of any specific intervention depends in part on the effectiveness of other interventions related to the condition being addressed [1]. These challenges mean that, as we investigate ways to assess the value of health care, we do not expect that there is only one correct ‘value’ for any intervention, even for a specific condition, i.e. ‘value’ is contextual. Our review reveals the inexistence of a gold standard approach, as the right answer is predicated on the particular context and decision that needs to be addressed.

Seeking to understand the current state of affairs in the realm value assessment, we examined both the peer-reviewed literature and gray literature. These reviews focused on actual cases of value measurement as opposed to theoretical frameworks. It is possible that powerful insights might be obtained from theoretical strategies not covered here. Another limitation of our study is that the search was constrained to the developed world and we acknowledge that interesting initiatives may be taking place elsewhere. And it is also possible that existing value assessment frameworks that are currently used have not been published. In order to address this weakness of a sole focus on the peer-reviewed literature, we also included a gray literature search, seeking to capture broader sources of information.

The fact that we were able to identify 22 different approaches in the peer reviewed literature supports the Sorenson et al. [47] conclusion that there is no single approach that is likely to apply in most situations: “At this time, it is clear that there is no one perfect model or framework for value assessment, or even one that will garner consensus across all stakeholders.” So, what can we learn from these 22 approaches found in our review? A few clear messages emerge.

First, there seems to be an awareness across developed countries, and very much so in the US, of the need to improve the measurement of the value of health care services and there obviously have been serious efforts made to do so. That being said, while the convergence in the results of the different approaches seems to be growing [30], these approaches all seem to have some weaknesses. For example, as Cohen et al. [27] state, “all of the frameworks suffer from varying degrees of arbitrariness, namely, subjectively determined end-points and arbitrary ways of combining scores from multiple dimensions to arrive at a composite health outcome measure. Owing to this arbitrariness, it is unknown the degree to which these frameworks capture value accurately.” The awareness of the need to measure value is related to the financial pressures present in health care. This was confirmed in our gray literature search where we did not find any new approach to value measurement but where we found many instances of efforts to apply value measurement to the search for interventions that should be avoided because they provide low value for their cost. Furthermore, this perhaps speaks to the notion that the practice of value assessment is less developed than the writing and publishing on this subject.

The second message is that while the most prevalent approach cited in the literature is the single criterion CEA/CUA, the vast majority of the 22 approaches found utilize multiple criteria, perhaps reflecting the reality that decision makers typically face a broad set of objectives when assessing value.

Third, the accepted forms of evidence are expanding from the traditional gold standard level of evidence (i.e., RCTs). In fact, about a third of the approaches considered explicitly patient input as part of the process for determining value. This certainly fits with broader trends towards patient-oriented research and the inclusion of patient input in health care decision making [37, 48].

And finally, the focus of most approaches seems to be directly on health outcomes rather than on process measures. The fact that some approaches include disease specific criteria while others are more generic in nature likely reflects decision making practice, i.e., in some cases the focus of the question being addressed is narrower while in other cases the question is broader.

While a comprehensive view of existing approaches for assessing value may be useful to provide insights on further action, the question of what approach should be used in a given context remains unanswered. There has been important work on assessing existing approaches and providing guidance for improvements [30, 47, 49, 50], but our searches did not identify any direct guidance on how to select the best approach for a given situation. Although our classification system was developed as a strategy to scrutinize the diversity of methodologies identified in the literature as value assessment frameworks, one can also rely on our 4-question analytical tool in order to develop novel ways to assess value in contexts hitherto unexplored.

Conclusions

Three main lessons emerge from our literature review on value assessment approaches in health care. First, there is widespread interest amongst health care providers and funders in measuring value due to growing financial pressures. Second, the incremental cost-effectiveness ratio, the traditional value measurement tool, does not seem to meet the needs of decision-makers as demonstrated by the number of approaches being developed. And third, the definition of what is included in value and the methods and inputs involved in assessing performance have expanded significantly and continue to expand. Overall, this paper adds to the understanding of value assessment and should be a helpful guide as academics and decision makers seek to not only unpack the notion of value in health care but also move forward with initiatives to assess value in a given context.

Furthermore, our work reveals several important pathways of future research. First, the inconsistency found in the use of the notion of “value” shows the importance of a careful and robust epistemological reflection about this concept and its practical implications. Originally born within the realm of neoclassical economics with an intrinsic association with the notion of utility and its manifestation in price, value has gained other contours in the health economics literature. The widely cited work [1] by Michael Porter defines value as “health outcomes achieved per dollar spent”, which seems to place value as a maximand that fits well in a welfarist or extra-welfarist paradigm. More recently, the ISPOR’s task force [51] on value assessment presented a complex composite nature of value, although according to them only a few elements have been consistently addressed in the literature. Thus, not only a further reflection on the notion of value must be pursued by the health economics community, but also explicit statements about the underpinning theoretical notions of value should be encouraged by those who delve into empirical initiatives in this realm. Second, as many of the self-described value assessment frameworks did not present any explicit consideration of cost, it would be very important to understand how we can develop more appropriate tools that take into account the costs of health care interventions within each framework. That is related to the fact that the traditional cost-effectiveness approach does not seem to be currently fulfilling the needs of decision-makers and new manners to adequately consider cost may be also necessary. Lastly, just as our interpretive classification scheme, novel approaches to guide decision-makers and managers on how to choose and/or develop adequate practices to assess value of health care technologies need to be further investigated.

Availability of data and materials

Not applicable.

Abbreviations

AHRQ:

Agency for Health Research and Quality

ASCO:

American Society for Clinical Oncology

CADTH:

Canadian Agency for Drugs and Technologies in Health

CEA:

Cost-Effectiveness Analysis

CUA:

Cost-Utility Analysis

DCE:

Discrete Choice Experiment

EUnetHTA:

European Network for Health Technology Assessment

HTA:

Health Technology Assessment

HTAi:

Health Technology Assessment International

ICER:

Institute for Clinical and Economic Review

IHEA:

International Health Economics Association

INAHTA:

International Network of Agencies for Health Technology Assessment

ISPOR:

International Society for Pharmacoeconomics and Outcomes Research

MCDA:

Multi-Criteria Decision Analysis

RCT:

Randomized Controlled Trial

VA:

Veteran Affairs

References

  1. Porter ME. What is value in health care? N Engl J Med. 2010;363(26):2477–81.

    Article  CAS  PubMed  Google Scholar 

  2. Garrison LP, Pauly MV, Willke RJ, Neumann PJ. An overview of value, perspective, and decision context—a health economics approach: an ISPOR special task force report [2]. Value Health. 2018;21(2):124–30.

    Article  PubMed  Google Scholar 

  3. Leggett L, Noseworthy TW, Zarrabi M, Lorenzetti D, Sutherland LR, Clement FM. Health technology reassessment of non-drug technologies: current practices. Int J Technol Assess Health Care. 2012;28(3):220–7.

    Article  PubMed  Google Scholar 

  4. Paprica PA, Culyer AJ, Elshaug AG, Peffer J, Sandoval GA. From talk to action: policy stakeholders, appropriateness, and selective disinvestment. Int J Technol Assess Health Care. 2015;31(4):236–40.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Parkinson B, Sermet C, Clement F, Crausaz S, Godman B, Garner S, et al. Disinvestment and value-based purchasing strategies for pharmaceuticals: an international review. Pharmacoeconomics. 2015;33(9):905–24.

    Article  PubMed  Google Scholar 

  6. Mayer J, Nachtnebel A. Disinvesting from ineffective technologies: lessons learned from current programs. Int J Technol Assess Health Care. 2015;31(6):355–62.

    Article  PubMed  Google Scholar 

  7. Maloney MA, Schwartz L, O'Reilly D, Levine M. Drug disinvestment frameworks: components, challenges, and solutions. Int J Technol Assess Health Care. 2017;33(2):261–9.

    Article  PubMed  Google Scholar 

  8. Chambers JD, Salem MN, D’Cruz BN, Subedi P, Kamal-Bahl SJ, Neumann PJ. A review of empirical analyses of disinvestment initiatives. Value Health. 2017;20(7):909–18.

    Article  PubMed  Google Scholar 

  9. Wilson L, Lin T, Wang L, Patel T, Tran D, Kim S, et al. Evaluation of the ASCO value framework for anticancer drugs at an Academic Medical Center. J Manag Care Spec Pharm. 2017;23(2):163–9.

    PubMed  Google Scholar 

  10. Schnipper LE, Davidson NE, Wollins DS, Blayney DW, Dicker AP, Ganz PA, et al. Updating the American Society of Clinical Oncology value framework: revisions and reflections in response to comments received. J Clin Oncol. 2016;34(24):2925–34.

    Article  PubMed  Google Scholar 

  11. Luce BR, Paramore LC, Parasuraman B, Liljas B, de Lissovoy G. Can managed care organizations partner with manufacturers for comparative effectiveness research? Am J Manag Care. 2008;14(3):149–56.

    PubMed  Google Scholar 

  12. Atkins D, Kupersmith J, Eisen S. The veterans affairs experience: comparative effectiveness research in a large health system. Health Aff. 2010;29(10):1906–12.

    Article  Google Scholar 

  13. Anderson JL, Heidenreich PA, Barnett PG, Creager MA, Fonarow GC, Gibbons RJ, et al. ACC/AHA statement on cost/value methodology in clinical practice guidelines and performance measures: a report of the American College of Cardiology/American Heart Association task force on performance measures and task force on practice guidelines. J Am Coll Cardiol. 2014;63(21):2304–22.

    Article  PubMed  Google Scholar 

  14. Alali AS, Naimark DMJ, Wilson JR, Fowler RA, Scales DC, Golan E, et al. Economic evaluation of decompressive craniectomy versus barbiturate coma for refractory intracranial hypertension following traumatic brain injury. Crit Care Med. 2014;42(10):2235–43.

    Article  PubMed  Google Scholar 

  15. Diamantopoulos A, Sawyer LM, Lip GYH, Witte KK, Reynolds MR, Fauchier L, et al. Cost-effectiveness of an insertable cardiac monitor to detect atrial fibrillation in patients with cryptogenic stroke. Int J Stroke. 2016;11(3):302–12.

    Article  PubMed  Google Scholar 

  16. Gyrd-Hansen D, Kjaer T. Disentangling WTP per QALY data: different analytical approaches, different answers. Health Econ. 2012;21(3):222–37.

    Article  PubMed  Google Scholar 

  17. Lien K, Cheung MC, Chan KKW. Adjusting for drug wastage in economic evaluations of new therapies for hematologic malignancies: a systematic review. J Oncol Pract. 2016;12(4):e369–79.

    Article  PubMed  Google Scholar 

  18. McDonough CM, Tosteson ANA. Measuring preferences for cost-utility analysis: how choice of method may influence decision-making. Pharmacoeconomics. 2007;25(2):93–106.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Standaert B, Ethgen O, Emerson R, Postma M, Mauskopf J. Comparing cost-effectiveness results for a vaccine across different countries worldwide: what can we learn? Adv Ther. 2014;31(10):1095–108.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Stout NK, Lee SJ, Schechter CB, Kerlikowske K, Alagoz O, Berry D, et al. Benefits, harms, and costs for breast cancer screening after US implementation of digital mammography. J Natl Cancer Inst. 2014;106(6):dju092.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Harron K, Mok Q, Dwan K, Ridyard CH, Moitt T, Millar M, et al. CATheter infections in CHildren (CATCH): a randomised controlled trial and economic evaluation comparing impregnated and standard central venous catheters in children. Health Technol Assess. 2016;20(18):vii–xxviii 1–219.

    Article  PubMed  Google Scholar 

  22. Smit F, Lokkerbol J, Riper H, Majo MC, Boon B, Blankers M. Modeling the cost-effectiveness of health care systems for alcohol use disorders: how implementation of eHealth interventions improves cost-effectiveness. J Med Internet Res. 2011;13(3):e56.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Bristow RE, Santillan A, Salani R, Diaz-Montes TP, Giuntoli RL, Meisner BC, et al. Intraperitoneal cisplatin and paclitaxel versus intravenous carboplatin and paclitaxel chemotherapy for stage III ovarian cancer: a cost-effectiveness analysis. Gynecol Oncol. 2007;106(3):476–81.

    Article  CAS  PubMed  Google Scholar 

  24. Kirsch F. Economic evaluations of multicomponent disease management programs with Markov models: a systematic review. Value Health. 2016;19(8):1039–54.

    Article  PubMed  Google Scholar 

  25. Fukuda H, Shimizu S, Ishizaki T. Has the reform of the Japanese healthcare provision system improved the value in healthcare? A cost-consequence analysis of organized Care for hip Fracture Patients. PLoS One. 2015;10(7):e0133694.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  26. Green C, Gerard K. Exploring the social value of health-care interventions: a stated preference discrete choice experiment. Health Econ. 2009;18(8):951–76.

    Article  PubMed  Google Scholar 

  27. Cohen JT, Anderson JE, Neumann PJ. Three sets of case studies suggest logic and consistency challenges with value frameworks. Value Health. 2017;20(2):193–9.

    Article  PubMed  Google Scholar 

  28. Dunt D, Anjou M, Boudville A, Hsueh A, Taylor H. Establishing the value of indigenous eye health programs: health needs, economic priority and performance assessment approaches. Aust Health Rev. 2014;38(1):99–105.

    Article  PubMed  Google Scholar 

  29. Ko A, Murry JS, Hoang DM, Harada MY, Aquino L, Coffey C, et al. High-value care in the surgical intensive care unit: effect on ancillary resources. J Surg Res. 2016;202(2):455–60.

    Article  PubMed  Google Scholar 

  30. Bentley TGK, Cohen JT, Elkin EB, Huynh J, Mukherjea A, Neville TH, et al. Validity and reliability of value assessment frameworks for new Cancer drugs. Value Health. 2017;20(2):200–5.

    Article  PubMed  Google Scholar 

  31. Marsh K, Lanitis T, Neasham D, Orfanos P, Caro J. Assessing the value of healthcare interventions using multi-criteria decision analysis: a review of the literature. Pharmacoeconomics. 2014;32(4):345–65.

    Article  PubMed  Google Scholar 

  32. Thokala P, Devlin N, Marsh K, Baltussen R, Boysen M, Kalo Z, et al. Multiple criteria decision analysis for health care decision making--an introduction: report 1 of the ISPOR MCDA emerging good practices task force. Value Health. 2016;19(1):1–13.

    Article  PubMed  Google Scholar 

  33. Eggleston KN, Shah ND, Smith SA, Wagie AE, Williams AR, Grossman JH, et al. The net value of health care for patients with type 2 diabetes, 1997 to 2005. Ann Intern Med. 2009;151(6):386–93.

    Article  PubMed  Google Scholar 

  34. Roth JA, Etzioni R, Waters TM, Pettinger M, Rossouw JE, Anderson GL, et al. Economic return from the Women’s health initiative estrogen plus progestin clinical trial: a modeling study. Ann Intern Med. 2014;160(9):594.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Bohmer P, Pain C, Watt A, Abernethy P, Sceats J. Maximising health gain within available resources in the New Zealand public health system. Health Policy. 2001;55(1):37–50.

    Article  CAS  PubMed  Google Scholar 

  36. Edwards RT, Charles JM, Thomas S, Bishop J, Cohen D, Groves S, et al. A national Programme budgeting and marginal analysis (PBMA) of health improvement spending across Wales: disinvestment and reinvestment across the life course. BMC Public Health. 2014;14:837.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Mitton C, Dionne F, Damji R, Campbell D, Bryan S. Difficult decisions in times of constraint: criteria based resource allocation in the Vancouver coastal health authority. BMC Health Serv Res. 2011;11:169.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Hung S-H, Wang P-C, Lin H-C, Chen H-Y, Su C-T. Integration of value stream map and healthcare failure mode and effect analysis into six sigma methodology to improve process of surgical specimen handling. J Healthc Eng. 2015;6(3):377–98.

    Article  PubMed  Google Scholar 

  39. Govaert JA, van Bommel ACM, van Dijk WA, van Leersum NJ, Tollenaar RAEM, Wouters MWJM. Reducing healthcare costs facilitated by surgical auditing: a systematic review. World J Surg. 2015;39(7):1672–80.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Watkins KE, Burnam MA, Orlando M, Escarce JJ, Huskamp HA, Goldman HH. The health value and cost of care for major depression. Value Health. 2009;12(1):65–72.

    Article  PubMed  Google Scholar 

  41. Dougherty D, Mistry KB, Lindly O, Desoto M, LLanos K, Chesley F. Systematic evidence-based quality measurement life-cycle approach to measure retirement in CHIPRA. Acad Pediatr. 2014;14(5, Supplement):S97–103.

    Article  PubMed  Google Scholar 

  42. Seigfried RJ, Corbo T, Saltzberg MT, Reitz J, Bennett DA. Deciding which drugs get onto the formulary: a value-based approach. Value Health. 2013;16(5):901–6.

    Article  PubMed  Google Scholar 

  43. Kirkpatrick JR, Smith BM, Zapas JL, Thomas WL. Clinical impact of a value-based decision: a surgical case study. J Am Coll Surg. 2013;216(4):800–11; discussion 811-813.

    Article  PubMed  Google Scholar 

  44. Feeley TW, Fly HS, Albright H, Walters R, Burke TW. A method for defining value in healthcare using cancer care as a model. J Healthc Manag. 2010;55(6):399–411; discussion 411-412.

    Article  PubMed  Google Scholar 

  45. Kawamoto K, Martin CJ, Williams K, Tu M-C, Park CG, Hunter C, et al. Value driven outcomes (VDO): a pragmatic, modular, and extensible software framework for understanding and improving health care costs and outcomes. J Am Med Inform Assoc. 2015;22(1):223–35.

    Article  PubMed  Google Scholar 

  46. Lee VS, Kawamoto K, Hess R, Park C, Young J, Hunter C, et al. Implementation of a value-driven outcomes program to identify high variability in clinical costs and outcomes and association with reduced cost and improved quality. JAMA. 2016;316(10):1061–72.

    Article  PubMed  Google Scholar 

  47. Sorenson C, Lavezzari G, Daniel G, Burkholder R, Boutin M, Pezalla E, et al. Advancing value assessment in the United States: a multistakeholder perspective. Value Health. 2017;20(2):299–307.

    Article  PubMed  Google Scholar 

  48. Wilson MG, Ellen ME, Lavis JN, Grimshaw JM, Moat KA, Shemer J, et al. Processes, contexts, and rationale for disinvestment: a protocol for a critical interpretive synthesis. Syst Rev. 2014;3:143.

    Article  PubMed  PubMed Central  Google Scholar 

  49. Westrich K. Current landscape: value assessment frameworks. Washington, DC: National Pharmaceutical Council; 2016.

  50. Mandelblatt JS, Ramsey SD, Lieu TA, Phelps CE. Evaluating frameworks that provide value measures for health care interventions. Value Health. 2017;20(2):185–92.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Lakdawalla DN, Doshi JA, Garrison LP, Phelps CE, Basu A, Danzon PM. Defining elements of value in health care—a health economics approach: an ISPOR special task force report [3]. Value Health. 2018;21(2):131–9.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

This study was funded by the US National Pharmaceutical Council in a project that sought lessons from other countries to the US regarding value assessment. All the study design and analysis were carried out without any specific guidance from the funding body.

Author information

Authors and Affiliations

Authors

Contributions

BVS conducted the systematic review, data extraction, evidence synthesis and wrote the first manuscript draft. FD developed the study design, supervised the data analysis, participated in the development of the classification system and reviewed the manuscript. TC participated in the data extraction and evidence synthesis. CM developed the study design, established criteria for the literature search and article inclusion, participated in the development of the classification system and reviewed the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Brayan V. Seixas.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

Search strategy. (DOCX 30 kb)

Additional file 2:

Full article data extraction tool. (DOCX 14 kb)

Additional file 3:

Organizations identified in previous literature reviews as having signs of initiative around disinvestment and reassessment. (DOCX 14 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Seixas, B.V., Dionne, F., Conte, T. et al. Assessing value in health care: using an interpretive classification system to understand existing practices based on a systematic review. BMC Health Serv Res 19, 560 (2019). https://0-doi-org.brum.beds.ac.uk/10.1186/s12913-019-4405-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12913-019-4405-6

Keywords