Skip to content

Voices for Safer Care

Insights from the Armstrong Institute

Voices for Safer Care Home Measurement of Safety and Quality The Surgeon Scorecard and the Need for Measurement Standards

The Surgeon Scorecard and the Need for Measurement Standards

Most of us would agree that there aren't enough valid and meaningful health care quality measures to guide patients' choices of hospitals and physicians. While the federal government has steadily expanded the number of publicly available measures on its Hospital Compare website, it still falls short of what many patients, payers and providers would like. This is particularly true in the realm of outcomes such as infections and mortality rates, and in provider-level ratings.

Journalists and other ratings-making organizations have recently attempted to fill the measurement chasm left by policymakers and health care professionals. In July, nonprofit journalism organization ProPublica unveiled its Surgeon Scorecard, posting the "Adjusted Complication Rates" for more than 16,000 physicians in eight inpatient procedures. The Scorecard’s release set off an intense debate within the health care community about the validity of the measure as well as the requirements of journalists when they function as scientists to create new measures. With the Surgeon Scorecard, ProPublica acted as judge and jury; they defined the measure, deemed it valid, and declared which surgeons were low quality. What assurances does the public have that such "vigilante" measures are scientifically sound? While ProPublica says its work was "guided by experts," that review was informal.

Shortly after the Scorecard was issued, some detractors on social media called for it to undergo peer review, a process that is typical for government-issued measures. That review was delivered on Friday, when several researchers in health care quality measurement, including me, published a critique on the RAND Corporation website. Our conclusion: patients should not consider the Scorecard a valid or reliable predictor of any individual surgeon's outcomes.

Among several concerns raised, we pointed out that the Adjusted Complication Rate, which was based mostly on readmissions, was not a true complication rate. The measure didn't consider complications that occurred during a hospital admission and ignored many complications that are most meaningful to patients. For instance, erectile dysfunction is common after radical prostatectomy (more than 50 percent, according to some estimates), but it was not tracked in the ProPublica measure. We also found problems with the underlying data used by ProPublica: Some surgical cases were attributed to non-surgeons or to surgeons in the wrong specialty — a finding that suggests the existence of other errors that are harder to detect.

Developing and vetting a valid new quality measure can be hard, tedious and controversial. Yet that process unearths weaknesses, improves the final product, and ultimately makes the measure more useful to patients and physicians. No matter who creates a measure — the government, journalists or nonprofit groups — we all have a duty to ensure it receives the highest level of scrutiny before it's issued, not after the fact. When journalists act as scientists, they should be held to the standards of scientists.


Peter Pronovost

One of the world’s leading authorities on patient safety, Peter Pronovost served a the director of the Armstrong Institute, as well as senior vice president for patient safety and quality, at Johns Hopkins Medicine from 2011 until January 2018.

6 thoughts on “The Surgeon Scorecard and the Need for Measurement Standards”

  1. Anytime an organization is based on such a sanctimonious mission statement ("To expose abuses of power and betrayals of the public trust by government, business, and other institutions, using the moral force of investigative journalism to spur reform through the sustained spotlighting of wrongdoing"), you have to be wary.* The implied intent of ProPublica's "moral force" is to produce objective, independent, investigative stories. Presumably these are appropriately informed. But time has shown that ProPublica really is, first and foremost, just all about ProPublica.

    *To that end, I'm reminded of Charles Foster Kane's "Declaration of Principles" to his readers.

  2. Concerns shared by Mr Peter Pronovost, One of the world’s leading authorities on patient safety, Peter Pronovost is the director of the Armstrong Institute, as well as senior vice president for patient safety and quality, at Johns Hopkins Medicine.

    Let us see how Johns Hopkins patient safety efforts came out on Surgeon's scorecard. Looks like, 2 surgeons in what some call "better than average group, 9 Surgeons in what some call the "average group" 2 surgeons in what some call the "worse than average" group. hmm.. interesting, but not stellar.. Now let us see how Consumer Reports scores Johns Hopkins.. Hmm.. not stellar here either... Infection rates not stellar either?
    Just say'en....

  3. Pingback: The Surgeon Scorecard and the Need for Measurement Standards | Doctella

  4. Thanks for picking out the time to discuss this, I feel great about it and love studying more on this topic.
    It is extremely helpful for me. Thanks for such a valuable help again.also visit our websitexanax bars for sale

Leave a Reply

Your email address will not be published. Required fields are marked *