Off the Shelf or Recalibrate? Customizing a Risk Index for Assessing Mortality

Authors

  • James F. Reed, III
  • Stephen A. Olenchock, Jr
  • Sabina A. Murphy
  • Fernando M. Garzia

DOI:

https://doi.org/10.1532/hsf.890

Abstract

Background: Public "report cards" for cardiac surgery have been freely available from a variety of sources. These risk-adjusted indices serve as a means of benchmarking outcomes performances, allowing comparisons of outcomes between surgical programs, and quantifying quality improvement programs. We examined two alternative strategies for using previously developed risk-adjusted mortality models in a community hospital: (1) using the model "off the shelf" (OTS) and (2) recalibrating the existing model (RM) to fit the institution-specific population.

Methods: Six OTS models were used: Parsonnet (PA), Canadian (CA), Cleveland (CL), Northern New England (NNE), New York (NY), and New Jersey (NJ). The RM models were created by each model's independent variables and definitions and adjusting the weighting with logistic regression methods. The accuracy, the C statistic, and the precision of each model were assessed for in-hospital mortality. We compared the OTS version of each model to the RM version with methods detailed by Hanley and McNeil.

Results: The RM C statistic was improved for all risk-adjusted models, most notably in the statistical improvement seen in the PA (0.053 improvement) and NJ (0.052 improvement) indices. Statistical gains in precision were also seen in the RM models for the PA, CL, and NNE indices. Conversely, one model, the CA model, was more poorly calibrated in the RM model compared with the OTS model, despite an improved C statistic (0.062).

Conclusions: The RM strategy provides institution-explicit models that demonstrate a higher degree of accuracy and precision than the OTS models.

References

DeLong ER, Peterson ED, DeLong DM, Muhlbaier LH, Hackett S, Mark DB. 1997. Comparing risk-adjustment methods for provider profiling. Stat Med 16:2645-64.nGeissler HJ, Holzl P, Marohl S, et al. 2000. Risk stratification in heart surgery: comparison of six score systems. Eur J Cardiothorac Surg 17:400-6.nGreen J, Whitfeld N. 1995. Report cards on cardiac surgeons: assessing New York State's approach. N Engl J Med 332:1229-32.nHanley JA, McNeil BJ. 1982. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143:29-36.nHanley JA, McNeil BJ. 1983. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 148:839-43.nHannan EL, Kilburn H Jr, Racz M, Shields E, Chassin MR. 1994. Improving the outcomes of coronary artery bypass surgery in New YorknState. JAMA 271:761-6.nHiggins TL, Estafanous FG, Loop FD, Beck GJ, Blum JM, Paranandi L. 1992. Stratification of morbidity and mortality outcome by preoperative risk factors in coronary artery bypass patients: a clinical severity score. JAMA 267:2344-8.nIezzoni LI, Ash AS, Shwartz M, Landon BE, Mackiernan YD. 1998. Predicting in-hospital death from coronary artery bypass graft surgery: do different severity measures give different predictions? Med Care 36:28-39.nIvanov J, Tu JV, Naylor CD. 1999. Ready-made, recalibrated, or remodeled? Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery. Circulation 99:2098-104.nLawrence DR, Valencia O, Smith EEJ, Murday A, Treasure T. 2000. Parsonnet score is a good predictor of the duration of intensive care unit stay following cardiac surgery. Heart 83:429-32.nLemeshow S, Hosmer DW Jr. 1982. A review of goodness of fit statistics for use in the development of logistic regression models. Am J Epidemiol 115:92-106.nMartinez-Alario J, Tuesta ID, Plasencia E, Santana M, Mora ML. 1999. Mortality prediction in cardiac surgery patients: comparative performance of Parsonnet and general severity systems. Circulation 99:2378-82.n[NJDHSS] New Jersey Department of Health and Senior Services, Division of Health Care Systems Analysis. Coronary artery bypass graft surgery in New Jersey 1998. Available at: http://www.state.nj.us/health/ hcsa/cabgs99/technical.htm. Accessed April 30, 2003.nO'Conner GT, Plume SK, Olmstead EM, et al. 1992. Multivariate prediction of in-hospital mortality associated with coronary artery bypass graft surgery: Northern New England Cardiovascular Disease Study Group. Circulation 85:2110-8.nOrr RK, Maini BS, Sottile FD, Dumas EM, O'Mara P. 1995. A comparison of four severity-adjusted models to predict mortality after coronary artery bypass graft surgery. Arch Surg 130:301-6.nParsonnet V, Dean D, Berstein AD. 1989. A method of uniform stratification of risk for evaluating the results of surgery in acquired adult heart disease. Circulation 79:I3-12.nPeterson ED, DeLong ER, Muhlbaier LH, et al. 2000. Challenges in comparing risk-adjusted bypass surgery mortality results: results from the Cardiovascular Project. J Am Coll Cardiol 36:2174-84.nPliam MB, Shaw RE, Zapolanski A. 1997. Comparative analysis of coronary surgery risk stratification models. J Invasive Cardiol 9:203-22.nPons JM, Granados A, Espinas JA, Borras JM, Martin I, Moreno V. 1997. Assessing open heart surgery mortality in Catalonia (Spain) through a predictive risk model. Eur J Cardiothorac Surg 11:415-23.nRoques F, Nashef SA, Michel P, et al. 1999. Risk factors and outcome in European cardiac surgery: analysis of the EuroSCORE multinational database of 19030 patients. Eur J Cardiothorac Surg 15:816-23.nTu JV, Jaglal SB, Naylor CD. 1995. Multicenter validation of a risk index for mortality, intensive care unit stay, and overall hospital length of stay after cardiac surgery: Steering Committee of the Provincial Adult Cardiac Care Network of Ontario. Circulation 91:677-84.nWeightman WM, Gibbs NM, Sheminant MR, Thackray NM, Newman MA. 1997. Risk prediction in coronary artery surgery: a comparison of four risk scores. Med J Aust 166:408-11.n

Published

2005-02-07

How to Cite

Reed, III, J. F., Olenchock, Jr, S. A., Murphy, S. A., & Garzia, F. M. (2005). Off the Shelf or Recalibrate? Customizing a Risk Index for Assessing Mortality. The Heart Surgery Forum, 6(4), 232-236. https://doi.org/10.1532/hsf.890

Issue

Section

Article