In this post, I present the case that U.S. News & World Report’s patient safety score, a component of its annual Best Hospitals rankings, has a bias against Maryland hospitals. Two editors at U.S. News respond to my position at the bottom of the post.

Hospitals can get overwhelmed by the array of ratings, rankings and scorecards that gauge the quality of care that they provide. Yet when those reports come out, we still scrutinize them, seeking to understand how to improve. This work is only worthwhile, of course, when these rankings are based on valid measures.

Certainly, few rankings receive as much attention as U.S. News & World Report's annual Best Hospitals list. This year, as we pored over the data, we made a startling discovery: As a whole, Maryland hospitals performed significantly worse on a patient safety metric that counts toward 10 percent of a hospital's overall score. Just three percent of the state's hospitals received the highest U.S. News score in patient safety — 5 out of 5 — compared to 12 percent of the remaining U.S. hospitals. Similarly, nearly 68 percent of Maryland hospitals, including The Johns Hopkins Hospital, received the worst possible mark — 1 out of 5 — while nationally just 21 percent did. This had been a trend for a few years.

What could account for this discrepancy? Could we all really be doing this poorly in my home state and in our hospital, where we take great pride in our efforts to prevent patient harm? After lengthy analysis, it seems quite clear that the answer is no. Instead, the patient safety score appears to have a bias against Maryland hospitals, because the data from our state is incomplete and not consistent with the data reported for hospitals outside of Maryland.

Maryland's Unique Arrangement

The U.S. News patient safety score rates hospitals on their track record for preventing seven health care-associated patient harms, such as punctured lung, hematoma and pressure ulcer. U.S. News derives this score by identifying Medicare billing claims that include the diagnosis codes for these harms. For this year's rankings, this analysis used claims data from October 2010 through September 2013.

The differences between Maryland and other states involves how we account for complications that are "present on admission" and therefore are not the result of poor hospital care. The hematoma that a patient suffered in an auto accident, for example, should not be attributed to the care he or she received in the hospital. Since late 2007, hospitals outside of Maryland have been required to add codes to their Medicare billing claims to indicate such present-on-admission conditions or face financial penalties for not doing so.

But in Maryland, we have a longstanding and unique arrangement with Medicare that has allowed us to participate in our state's pay-for-quality programs instead of the federal program. This essentially requires Maryland hospitals to have two data sets, one we submit to Medicare for billing and one we submit to the state for quality reporting. It wasn't until October 2014 — after the period analyzed for this year's U.S. News patient safety score — that that Medicare program started requiring present-on-admission codes from Maryland hospitals. On the other hand, Medicare required these codes from non-Maryland hospitals starting in 2007.

The result: Many complications that patients actually suffered before they came to our hospitals are being counted against Maryland hospitals in the U.S. News rankings.

The impact can be staggering. For example, when we looked at Medicare data for The Johns Hopkins Hospital in 2012 we found 29 cases of pressure ulcers — the number used by U.S. News. Yet after examining the quality-related data that we sent to the state of Maryland, all but one of those pressure ulcer cases were found to be present on admission.

Other Maryland hospitals mirrored our performance. Nearly 87 percent received the lowest possible score for pressure ulcers, versus 21 percent outside of our state. And Maryland hospitals, on average, performed far worse on safety than any other state. If the public were not aware of these data quality issues, they may mistakenly conclude that Maryland hospitals are significantly less safe than those in other states.

Looking Good vs. Doing Well

If this is too far a journey into the obscure world of health care measures, I feel your pain. Hospital leaders and quality improvement specialists are constantly bombarded with these measures, and spend much time trying to separate true concerns with "noise" from poor measures, poor data quality or random error. Too often, what we find is noise. But if a hospital looks poor in an invalid but high-profile measure, they ignore it at their own peril — even if improving on the score is more about looking good than actually delivering better care.

Over the years, U.S. News has continually sought to make its ranking methodology more fair and robust, and they have made improvements in response to feedback. Given the huge toll of preventable patient harm, it is encouraging that the weight given to patient safety was doubled, from five percent to 10 percent of the overall hospital score, beginning with last year's rankings.

But we must be sure that all hospitals are measured with the same yardstick and that the measures are valid and reliable. This bias wasn't created intentionally by U.S. News. We at Johns Hopkins didn't even realize it existed until this summer, and it appears to be news to other Maryland hospitals as well. It's the result of the uncoordinated, confusing way in which we attempt to measure quality in this country. The federal government, state agencies, insurers, nonprofits and others are creating measures with varying degrees of validity and usefulness. We need to work towards creating a single source of truth for which all hospitals and providers will be judged. In the meantime, I believe the U.S. News methodology for patient safety scores should be re-evaluated and possibly revised or replaced.

 

[themify_hr color="light-blue" width="75%" border_width="3px" ]

The U.S. News Response

–by Avery Comarow, Health Rankings Editor, and Ben Harder, Chief of Health Analysis, U.S. News & World Report

The key question raised by Pronovost is whether higher rates of missing present-on-admission (POA) information in data submitted by Maryland hospitals, and specifically by Johns Hopkins, caused those hospitals to receive lower patient safety scores than they would have otherwise. Pronovost concludes that the clear answer is yes. We feel the answer is complicated.

On our overall patient safety score, it is true that Maryland hospitals as a group did score below hospitals in most other states. It is not clear, however, that this is because of problems with the completeness or quality of POA data for Maryland hospitals, as Pronovost argues. It could reflect Maryland hospitals' actual quality of care. It could reveal deficiencies with software — developed by the federal Agency for Healthcare Research & Quality (AHRQ) — that U.S. News used to adjust for missing POA information about pressure ulcer and other patient safety indicators. Or it could be due to an unknown combination of the three — or to other factors.

Maryland hospitals were not required to submit POA information to the Centers for Medicare & Medicaid Services for payment until October 2014. However, an analysis by U.S. News found at least some POA information in prior years' Medicare records for almost all Maryland hospitals — some, in fact, had less missing data than hospitals in other states. Nevertheless, the typical Maryland hospital had significantly more missing POA data than hospitals elsewhere.

The documentation for the AHRQ software, which was designed specifically to account for missing POA data, made no mention that it had limitations in dealing with large amounts of missing data. Recently, in response to our questions, AHRQ indicated that its software was not designed to fill in the blanks for hospitals with high levels of missing POA data. The agency's scientists told U.S. News and its data contractor RTI International that while this limitation could introduce bias, they could not say how much that problem might skew the results, nor whether any potential bias would favor or disfavor Maryland hospitals.

Further analysis by U.S. News found that on most of the seven complication rates that comprise our overall patient safety score, Maryland hospitals scored similarly to hospitals elsewhere. The overall score and the complication cited in Pronovost's letter — pressure ulcer — were exceptions, not the rule. Reasonable people may differ in how they interpret that observation.

What is clear is the need for U.S. News to give the government's patient safety measures a hard look. We will communicate our findings and describe any planned methodology changes as we determine tem. In the meantime, we soon will annotate the patient safety scores of all Maryland hospitals to reflect the newly understood limitations of the AHRQ software we use.

 

The initial post and the U.S. News response were first printed in the publication's Second Opinion blog.