Are hospital quality metrics causal?

That is the question asked by a recent NBER working paper by Chandra et al. (2023).

This question is important for a variety of reasons. First, quality measure data collection is expensive. Saraswathula et al. 2023 found that Johns Hopkins Hospital had to report 162 unique quality metrics, and the cost for collecting these data were over $5.6m dollars ($5.0m plus $0.6m in vendor fees. Moreover, understanding whether a metric is causal is important both for value-based purchasing agreements and also evaluation of for whether hospital mergers result in superior quality or patient satisfaction (e.g., see Beaulieu et al. 2020)

A good quality metric should have at least 2
characteristics:

Predictive validity: Patients have better
outcomes when assigned to hospitals that are higher ratedForecast unbiased: The gain in patient health
outcomes is equal to the value predicted by the quality indicators (i.e., actual
quality differences are as large as quality indicators suggest)

Chandra et al. (2023) aim to test whether hospital quality metrics meet these two criteria.

Methodology

An overview of the Chandra et al. methodology is as follows:

To determine whether hospital quality indicators are causal, we exploit the reassignment of patients as a result of a hospital closure to create quasi-experimental variation in the quality of care that patients receive— if quality indicators are causal, patient outcomes ought to improve when a low-quality hospital closes, and worsen when a high-quality hospital closes (predictive validity). Moreover, the magnitude of the change in patient outcomes should be consistent with the predicted change in hospital quality induced by a closure (forecast unbiasedness).

The authors use 20 years of Medicare claims data for
patients hospitalized between 1992 and 2015 for heart-attacks, hip-fractures,
pneumonia, congestive heart failure or stroke. 
To account for the fact that hospitals may close some units first, before
a full facility closure is completed, the authors define closure date as the last
year for which a hospital had an admission of a given type. The approach for
this population uses a 3-step approach:

estimate quality metrics (30-day mortality,
30-day readmissions, 30-day total inpatient (Part A) costs, and length of stay),
Use hospital closures to predict how
(risk-adjusted) health outcomes would change for patients in each ZIP code of quality
metrics were causal, and Regress actual change in outcomes on the
predicted change in outcomes [from step 2] and test whether the regression
yields a coefficient of 1 (which would happen if the indicators were perfectly
forecast unbiased). 

The econometric approach largely follows the Chetty et al. (2014) empirical
Bayesian methodology, but identification strategy is similar to Doyle et al. (2019) — which
uses ambulance referral patterns to validate the predictive validity for a
range of hospital performance measures and Finkelstein
et al. (2016)—which uses patient migration to validate geographic variation
in health outcomes (mortality) and costs.

Results

Using this approach, the authors find that:

…closures reduced mortality by 0.2 percentage points (off a base of 13%) and reduced readmission by 0.1 percentage points (off a base of 18%), while raising costs and length of stay by about 1%. However, the effect of closure varied widely by zip code depending on the relative quality of the closing hospital. For example, mortality among affected patients in zip codes at the 10th percentile fell by about 1.5 percentage points, while at the 90th percentile they rose by about 1 percentage point

The full paper can be found here.

Further reading:

Beaulieu ND, Dafny LS, Landon BE,
Dalton JB, Kuye I, McWilliams JM. Changes in quality of care after hospital
mergers and acquisitions. New England Journal of Medicine. 2020 Jan
2;382(1):51-9.Chetty, Raj, John N. Friedman, and Jonah E. Rockoff. 2014.
“Measuring the Impacts of Teachers I: Evaluating Bias in Teacher
Value-Added Estimates.” American Economic Review, 104 (9): 2593-2632.Doyle J, Graves J, Gruber J. Evaluating Measures of Hospital
Quality:Evidence from Ambulance Referral Patterns. Rev Econ Stat. 2019
Dec;101(5):841-852.Finkelstein, Amy, Matthew Gentzkow, Heidi Williams, Sources of
Geographic Variation in Health Care: Evidence From Patient Migration, The
Quarterly Journal of Economics, Volume 131, Issue 4,  November 2016, Pages 1681–1726,
https://doi.org/10.1093/qje/qjw023Saraswathula, Anirudh, Samantha J. Merck, Ge Bai, Christine M.
Weston, Elizabeth Ann Skinner, April Taylor, Allen Kachalia, Renee Demski,
Albert W. Wu, and Stephen A. Berry. 2023. “The Volume and Cost of Quality
Metric Reporting.” JAMA: The Journal of the American Medical Association 329 (21):
1840–47.