Off the Shelf or Recalibrate? Customizing a Risk Index for Assessing Mortality
Background: Public "report cards" for cardiac surgery have been freely available from a variety of sources. These risk-adjusted indices serve as a means of benchmarking outcomes performances, allowing comparisons of outcomes between surgical programs, and quantifying quality improvement programs. We examined two alternative strategies for using previously developed risk-adjusted mortality models in a community hospital: (1) using the model "off the shelf" (OTS) and (2) recalibrating the existing model (RM) to fit the institution-specific population.
Methods: Six OTS models were used: Parsonnet (PA), Canadian (CA), Cleveland (CL), Northern New England (NNE), New York (NY), and New Jersey (NJ). The RM models were created by each model's independent variables and definitions and adjusting the weighting with logistic regression methods. The accuracy, the C statistic, and the precision of each model were assessed for in-hospital mortality. We compared the OTS version of each model to the RM version with methods detailed by Hanley and McNeil.
Results: The RM C statistic was improved for all risk-adjusted models, most notably in the statistical improvement seen in the PA (0.053 improvement) and NJ (0.052 improvement) indices. Statistical gains in precision were also seen in the RM models for the PA, CL, and NNE indices. Conversely, one model, the CA model, was more poorly calibrated in the RM model compared with the OTS model, despite an improved C statistic (0.062).
Conclusions: The RM strategy provides institution-explicit models that demonstrate a higher degree of accuracy and precision than the OTS models.
DeLong ER, Peterson ED, DeLong DM, Muhlbaier LH, Hackett S, Mark DB. 1997. Comparing risk-adjustment methods for provider profiling. Stat Med 16:2645-64.nGeissler HJ, Holzl P, Marohl S, et al. 2000. Risk stratification in heart surgery: comparison of six score systems. Eur J Cardiothorac Surg 17:400-6.nGreen J, Whitfeld N. 1995. Report cards on cardiac surgeons: assessing New York State's approach. N Engl J Med 332:1229-32.nHanley JA, McNeil BJ. 1982. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143:29-36.nHanley JA, McNeil BJ. 1983. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 148:839-43.nHannan EL, Kilburn H Jr, Racz M, Shields E, Chassin MR. 1994. Improving the outcomes of coronary artery bypass surgery in New YorknState. JAMA 271:761-6.nHiggins TL, Estafanous FG, Loop FD, Beck GJ, Blum JM, Paranandi L. 1992. Stratification of morbidity and mortality outcome by preoperative risk factors in coronary artery bypass patients: a clinical severity score. JAMA 267:2344-8.nIezzoni LI, Ash AS, Shwartz M, Landon BE, Mackiernan YD. 1998. Predicting in-hospital death from coronary artery bypass graft surgery: do different severity measures give different predictions? Med Care 36:28-39.nIvanov J, Tu JV, Naylor CD. 1999. Ready-made, recalibrated, or remodeled? Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery. Circulation 99:2098-104.nLawrence DR, Valencia O, Smith EEJ, Murday A, Treasure T. 2000. Parsonnet score is a good predictor of the duration of intensive care unit stay following cardiac surgery. Heart 83:429-32.nLemeshow S, Hosmer DW Jr. 1982. A review of goodness of fit statistics for use in the development of logistic regression models. Am J Epidemiol 115:92-106.nMartinez-Alario J, Tuesta ID, Plasencia E, Santana M, Mora ML. 1999. Mortality prediction in cardiac surgery patients: comparative performance of Parsonnet and general severity systems. Circulation 99:2378-82.n[NJDHSS] New Jersey Department of Health and Senior Services, Division of Health Care Systems Analysis. Coronary artery bypass graft surgery in New Jersey 1998. Available at: http://www.state.nj.us/health/ hcsa/cabgs99/technical.htm. Accessed April 30, 2003.nO'Conner GT, Plume SK, Olmstead EM, et al. 1992. Multivariate prediction of in-hospital mortality associated with coronary artery bypass graft surgery: Northern New England Cardiovascular Disease Study Group. Circulation 85:2110-8.nOrr RK, Maini BS, Sottile FD, Dumas EM, O'Mara P. 1995. A comparison of four severity-adjusted models to predict mortality after coronary artery bypass graft surgery. Arch Surg 130:301-6.nParsonnet V, Dean D, Berstein AD. 1989. A method of uniform stratification of risk for evaluating the results of surgery in acquired adult heart disease. Circulation 79:I3-12.nPeterson ED, DeLong ER, Muhlbaier LH, et al. 2000. Challenges in comparing risk-adjusted bypass surgery mortality results: results from the Cardiovascular Project. J Am Coll Cardiol 36:2174-84.nPliam MB, Shaw RE, Zapolanski A. 1997. Comparative analysis of coronary surgery risk stratification models. J Invasive Cardiol 9:203-22.nPons JM, Granados A, Espinas JA, Borras JM, Martin I, Moreno V. 1997. Assessing open heart surgery mortality in Catalonia (Spain) through a predictive risk model. Eur J Cardiothorac Surg 11:415-23.nRoques F, Nashef SA, Michel P, et al. 1999. Risk factors and outcome in European cardiac surgery: analysis of the EuroSCORE multinational database of 19030 patients. Eur J Cardiothorac Surg 15:816-23.nTu JV, Jaglal SB, Naylor CD. 1995. Multicenter validation of a risk index for mortality, intensive care unit stay, and overall hospital length of stay after cardiac surgery: Steering Committee of the Provincial Adult Cardiac Care Network of Ontario. Circulation 91:677-84.nWeightman WM, Gibbs NM, Sheminant MR, Thackray NM, Newman MA. 1997. Risk prediction in coronary artery surgery: a comparison of four risk scores. Med J Aust 166:408-11.n
How to Cite
Author Disclosure & Copyright Transfer Agreement
In order to publish the original work of another person(s), The Heart Surgery Forum® must receive an acknowledgment of the Author Agreement and Copyright Transfer Statement transferring to Forum Multimedia Publishing, L.L.C., a subsidiary of Carden Jennings Publishing Co., Ltd. the exclusive rights to print and distribute the author(s) work in all media forms. Failure to check Copyright Transfer agreement box below will delay publication of the manuscript.
A current form follows:
The author(s) hereby transfer(s), assign(s), or otherwise convey(s) all copyright ownership of the manuscript submitted to Forum Multimedia Publishing, LLC (Publisher). The copyright transfer covers the exclusive rights to reproduce and distribute the article and the material contained therein throughout the world in all languages and in all media of expression now known or later developed, including but not limited to reprints, photographic reproduction, microfilm, electronic data processing (including programming, storage, and transmission to other electronic data record(s), or any other reproductions of similar nature), and translations.
However, Publisher grants back to the author(s) the following:
- The right to make and distribute copies of all or part of this work for use of the author(s) in teaching;
- The right to use, after publication in The Heart Surgery Forum, all or part of the material from this work in a book by the author(s), or in a collection of work by the author(s);
- The royalty-free right to make copies of this work for internal distribution within the institution/company that employs the author(s) subject to the provisions below for a work-made-for-hire;
- The right to use figures and tables from this work, and up to 250 words of text, for any purpose;
- The right to make oral presentations of material from this work.
Publisher reserves the right to grant or refuse permission to third parties to republish all or part of the article or translations thereof. To republish, such third parties must obtain written permission from the Publisher. (This is in accordance with the Copyright Statute, United States Code, Title 17. Exception: If all authors were bona fide officers or employees of the U.S. Government at the time the paper was prepared, the work is a “work of the US Government” (prepared by an officer or employee of the US Government as part of official duties), and therefore is not subject to US copyright; such exception should be indicated on signature lines. If this work was prepared under US Government contract or grant, the US Government may reproduce, royalty-free, all or portions of this work and may authorize others to do so, for official US Government purposes only, if the US Government contract or grant so requires.
I have participated in the conception and design of this work and in the writing of the manuscript and take public responsibility for it. Neither this manuscript nor one with substantially similar content under my authorship has been published, has been submitted for publication elsewhere, or will be submitted for publication elsewhere while under consideration by The Heart Surgery Forum, except as described in an attachment. I have reviewed this manuscript (original version) and approve its submission. If I am listed above as corresponding author, I will provide all authors with information regarding this manuscript and will obtain their approval before submitting any revision. I attest to the validity, accuracy, and legitimacy of the content of the manuscript and understand that Publisher assumes no responsibility for the validity, accuracy, and legitimacy of its content. I warrant that this manuscript is original with me and that I have full power to make this Agreement. I warrant that it contains no matter that is libelous or otherwise unlawful or that invades individual privacy or infringes any copyright or other proprietary right. I agree to indemnify and hold Publisher harmless of and from any claim made against Publisher that relates to or arises out of the publication of the manuscript and agree that this indemnification shall include payment of all costs and expenses relating to the defense of any such claim, including all reasonable attorney’s fees.
I warrant that I have no financial interest in the drugs, devices, or procedures described in the manuscript (except as disclosed in the attached statement).
I state that the institutional Human Subjects Committee and/or the Ethics Committee approved the clinical protocol reported in this manuscript for the use of experimental techniques, drugs, or devices in human subjects and appropriate informed consent documents were utilized.
Furthermore, I state that any and all animals used for experimental purposes received humane care in USDA registered facilities in compliance with the “Principles of Laboratory Animal Care” formulated by the National Society for Medical Research and the “Guide for the Care and Use of Laboratory Animals” prepared by the Institute of Laboratory Animal Resources and published by the National Institutes of Health (NIH Publication No. 85-23, revised 1985).