Interact CardioVasc Thorac Surg 2007;6:628-631. doi:10.1510/icvts.2007.157917
© 2007 European Association of Cardio-Thoracic Surgery
Institutional report - Congenital |
Predictive value of paediatric risk of mortality score and risk adjustment for congenital heart surgery score after paediatric open-heart surgery
Leena Mildha,*,
Ville Pettiläb,
Heikki Sairanenc and
Paula Rautiainena
a Department of Anaesthesiology and Intensive Care Medicine, Helsinki University Hospital, Hospital for Children and Adolescents, PO Box 281, 00029 HUS, Helsinki, Finland
b Department of Anaesthesiology and Intensive Care Medicine, Division of Intensive Care, Helsinki University Hospital, Helsinki, Finland
c Department of Cardiac Surgery, Helsinki University Hospital, Hospital for Children and Adolescents, Helsinki, Finland
Received 17 April 2007;
received in revised form 2 July 2007;
accepted 2 July 2007
 This study was supported in part by Helsinki University Central Hospital, Finland (EVO grant).
*Corresponding author. Tel.: +358-50-4271641; fax: +358-9-47174701.
E-mail address: leena.mildh{at}hus.fi (L. Mildh).
 |
Abstract
|
|---|
This study compared the performance of risk adjustment for congenital heart surgery (RACHS-1) score with paediatric risk of mortality (PRISM) score in operative risk prediction after open-heart surgery in children. This was a retrospective analysis of a non-selected patient population from the paediatric intensive care unit of Helsinki University Hospital. All consecutive congenital open-heart surgery patients operated in Finland between the years 2000 and 2004, who were under 18 years of age, were included in this retrospective analysis. Predicted probability of mortality was calculated using the published algorithms for RACHS-1 and PRISM. Those were compared with observed mortality at day 30 postoperatively. Of the 1001 patients, 42 patients died (4.2%) within 30 days of open-heart surgery. The discrimination power, evaluated by AUC (area under curve) for RACHS-1 was moderate: 0.74 (95% CI 0.66–0.82). The AUC-value for PRISM was poor, namely 0.66 (95% CI 0.57–0.75). Both risk scoring systems overestimated the mortality with calculated standardised mortality ratios (SMR) of 0.48 for PRISM and 0.39 for RACHS-1. With only a moderate discriminating AUC, RACHS-1 failed to adequately predict death after paediatric open-heart surgery. The predictive power of PRISM in this patient group was poor. Both scores overestimated the actual mortality rate.
Key Words: PRISM; RACHS-1; Congenital; Outcome; Intensive care
|
1. Introduction
|
|---|
Risk-adjustment tools that predict death in paediatric intensive care units (PICUs) have become established in the past 20 years. The paediatric risk of mortality (PRISM), first published by Pollack et al. in 1988, is a generic severity-scoring system that measures the probability of death on the basis of the hypothesis that physiologic instability reflects mortality risk [1]. Recently, a more accurate risk scoring system has been developed for postoperative cardiac patients. The operative risk scoring according to RACHS-1 (risk adjustment for congenital heart surgery) has been previously shown to predict outcome after paediatric heart surgery in many studies [2–6].
Most risk prediction tools are not made to predict risk in an individual patient, but for overall comparison for different centres. Prediction tools should discriminate well between deaths and survivors and be well calibrated before they can be applied usefully to assess or standardise comparisons of PICUs or to correct for case-mix differences between groups in observational studies. In recent years, the mortality rate of paediatric cardiac surgery has decreased below 5% in most centres. Many of these risk prediction tools were standardised either in the 1980s or 1990s, when the mortality rate with congenital heart surgery was higher. The aim of this study was to evaluate if PRISM as a general PICU prediction tool and RACHS-1 as a specific cardiac risk stratification may be used in prediction of day-30 mortality in consecutive and unselected paediatric open-heart surgery patients.
|
2. Materials and methods
|
|---|
This retrospective cohort study was performed in Helsinki University Hospital for Children and Adolescents, where all paediatric heart surgery is performed in Finland (population 5.2 million) without any patient selection. No patient referral from Finland to other countries was done during the study period. The study was reviewed and approved by the local Human Investigation Committee, which waived the need to obtain informed consent.
2.1. Subjects and data collection
Data from all patients under 18 years having heart surgery necessitating cardiopulmonary bypass (CPB) between 1 January 2000 and 31 December 2004 were included in the study. Primary outcome measure was death within the first 30 days (D30) after surgery. Data were obtained from two different electronic databases linked together by means of patients' social security number. Information describing patients' survival at D30 postoperatively were recorded in a specific paediatric heart surgery database (ProCardio, Melba Group, Finland), which obtains its information from the an Finnish Population Registry Centre. PRISM scores were obtained automatically from an Intensive Care database (Centricity Critical Care Clinisoft, GE Healthcare, Helsinki, Finland). This system calculates the score automatically using continuously monitored variables, laboratory tests and manual entries [1]. P-PRISM, the probability of ICU death is calculated using the equation P-PRISM= exp (r)/1+exp (r). All patients were also placed into operative risk category groups according to RACHS-1 scoring system [3, 4]. This was done retrospectively before analysing the final data.
2.2. Clinical outcome
Primary outcome measure was D30 all-cause mortality.
2.3. Statistical methods
Data are presented as median (interquartile range, IQR). Statistical analyses were performed with SPSS 10.1.3 for Windows (SPSS Inc., Chicago, IL). Differences in continuous variables between groups were compared using the non-parametric Mann–Whitney test. A P-value<0.05 was considered significant. The discriminative power of independent predictors regarding day-30 mortality underwent evaluation by producing receiver operating characteristic curves (ROCs) and by calculating areas under the curves (AUCs) with 95% CIs. The calibration, i.e. the agreement between expected and observed outcomes according to P-PRISM, PRISM and RACHS-1 was planned to be determined with goodness-of-fit statistics applied to dichotomous predictions, calculated by a Hosmer–Lemeshow modification of  2-test. However, due to the small numbers of deaths (under 5) in several risk groups, these statistics could not be performed. Instead of this, standardised mortality ratios (SMRs) were calculated for each RACHS-1 and P-PRISM class.
|
3. Results
|
|---|
A total of 1027 patients had heart surgery necessitating CPB between 1 January 2000 and 31 December 2004. Data from 1001 cardiac surgery patients under 18 years of age were entered in this study. If one patient had multiple operations during the same intensive care period, the risk stratification for the first operation only was included in this study. Forty-two (4.2%) of these patients died within 30 days from operation. Patient characteristics according to surgical procedure are shown in Table 1.
PRISM data were available in 972 patients. Twenty-two patients could not be placed to any RACHS-1 category group. The median PRISM-score was significantly lower for surviving patients 11 (7–15) than for those not surviving 13 (10–20) (P<0.001). Also, the probability of death calculated with P-PRISM was lower for survivors 0.046 (0.019–0.102) than for non-survivors 0.091 (0.050–0.188) (P<0.001). The risk category value of RACHS-1 was lower for surviving patients 3 (2–3) than for those who did not survive 4 (3–6) (P<0.001). A PRISM-value over 21, P-PRISM over 0.294 and a RACHS-1 value of 6.0 predicted death with a specificity of at least 95%.
The AUC value (95% CI) for RACHS-1 was 0.740 (0.658–0.822), for P-PRISM 0.709 (0.632–0.787), and for PRISM 0.661 (0.574–0.748) (Fig. 1). Both P-PRISM and RACHS-1 showed poor overall accuracy with standardised mortality ratios (SMRs) of 0.48 for P-PRISM and 0.39 for RACHS-1. Observed mortality and predicted mortality according RACHS-1 is depicted in Table 2. The SMR of all quintiles of P-PRISM differed from 0.34 to 0.95 (Table 3).
View larger version (28K):
[in this window]
[in a new window]
|
Fig. 1. Receiver operating characteristic (ROC) curves for paediatric risk of mortality (PRISM), P-PRISM the probability of death calculated from PRISM score and for risk adjustment for congenital heart surgery (RACHS-1). The area under curve value (95% CI) for RACHS-1 was 0.740 (0.658–0.822), for P-PRISM 0.709 (0.632–0.787) and for PRISM 0.661 (0.574–0.748).
|
|
View this table:
[in this window]
[in a new window]
|
Table 2 Mortality related to RACHS-1 categories. The predicted mortality is derived from the original publication of Jenkins et al. [2]
|
|
View this table:
[in this window]
[in a new window]
|
Table 3 Comparison of the observed and predicted mortality rates in different quintiles according to P-PRISM scores
|
|
|
4. Discussion
|
|---|
The performance of PRISM predicting outcome in this large, non-selected paediatric open-heart surgery patient population was considered poor. RACHS-1, being a more simple classification of surgical procedures, performed somewhat better in this patient cohort with an acceptable discriminating power, but it also failed to predict fatal outcome with low SMR.
Scoring systems were originally developed in order to quantitate clinical states that are difficult to summarise by other objective or subjective means [7]. Much work has been emphasised on the development of better tools to evaluate the quality of care of paediatric cardiac patients. Clinical scoring systems are used to control for case-mix variables, for example cardiac diagnoses, and thus allow for standardised comparisons between institutions. The Aristotle score is the most recent one in paediatric cardiac surgery patients. It is based on the complexity of the surgical procedures and it adjusts the complexity of a given procedure and is calculated as the sum of potential for mortality, potential for morbidity and anticipated technical difficulty [8]. However, in studies comparing the basic Aristotle's score to RACHS-1, no clear additional value of Aristotle's score over RACHS-1 was noted [9, 10].
PRISM was originally published by Pollack et al. in 1988 [1]. It has been widely used and found accurate in all patient groups of paediatric ICUs. However, the performance of PRISM in paediatric cardiac patients has been questioned in some studies, with an SMR as low as 0.5 [11–13]. PRISM was validated in the 1980s and this might make it out of date for present paediatric cardiac ICU populations with increased survival rates. Also, these patients are returned to the ICU from the cardiac operating room with their physiologic abnormalities treated with interventions and thus extremes in physiologic variables may not be present on admission to the ICU. The reason why the discrimination power of PRISM, measured both as PRISM itself and as P-PRISM, was so poor in our patient cohort compared to other units, remains unclear. The calibration value of these two variables was, however, similarly poor in our study as it was in many previous ones.
A different approach to risk stratification focusing specifically in cardiac surgery patients is RACHS-1. It has been shown to be an accurate risk predictor in paediatric open-heart surgery both in the United States as well as in Europe [2–6, 14]. The downsides of RACHS-1 are that many patients cannot be placed into any risk groups, as in our patient cohort of 22 patients (2.2% from total number of patients). For example, cardiac transplantation patients and patients with ventricular assist device cannot be placed into RACHS-1. One of RACHS-1 risk groups (group 5 with the second highest risk) includes usually only a few patients if any and is, therefore, not useful. In our patient material only two patients of 1001 were eligible for RACHS-1 group 5. The discriminating power of RACHS-1 in our patient material as depicted by ROC was 0.74, which is very similar to previous studies, where it was shown to vary from 0.741 [4] and 0.755 [5] to 0.81 [6]. In the original work of Jenkins et al. the ROC value was 0.784 for the PCCC dataset [2]. All these values are considered at least acceptable in risk stratification. However, the ability of RACHS-1 to predict death was quite low with a total SMR of 0.39. The SMR also varied quite largely in different risk groups as seen in Table 2. In the study of Jenkins and Gauvreau, total SMR varied in different US centres between 0.43 and 2.11 [3]. In European studies, SMR of each risk group varied from 0.52 to 3.6 [4, 5]. The highest SMR in our patient population was 0.55 in risk group 2. In the highest risk group, i.e. 6, SMR was only 0.30. In a recent, large study of Welke et al. [14], the mortality rates of 29 different institutions from years 2001–2004 were lower in most RACHS-1 categories than predicted by the scoring system. These results resemble those of ours.
There are some limitations in this study. First, the mortality defined in this study was all-cause mortality at 30 days postoperatively. In the original publication of PRISM mortality was defined as ICU mortality. However, this would leave at least some definitely surgery related deaths out. In all previous RACHS-1 studies mortality was defined as in-hospital mortality. This also leaves out those who died at home or at a referral hospital shortly after surgery. The 30-day mortality is a definite time point of survival with no exclusions of any deaths and evaluation of this was made possible by the Finnish Population Registry Centre. Secondly, the data analysed in this study are retrospective and thus automatically historical. Whether the same degree of risk prediction applies to patients operated presently, remains to be seen. Thirdly, not all patients had complete PRISM data. However, the PRISM data were gathered as routine clinical management, and this strategy served as a means of assessing the feasibility of data collection as part of standard clinical practice.
In this non-selected, population based patient group of paediatric open-heart surgery, PRISM proved to be a poor tool for risk stratification with low discrimination and calibration values. The discrimination power of RACHS-1 was good and in accordance to other studies published recently. However, RACHS-1 failed to accurately predict death after paediatric open-heart surgery patients. A different and more precise approach in predicting the outcome of these patients is needed in the future.
|
Acknowledgements
|
|---|
We would like to thank Sirpa Savolainen, RN for her expertise and help in collecting the data from electronic data bases.
|
References
|
|---|
- Pollack MM, Ruttimann UE, Getson PR. Pediatric risk of mortality (PRISM) score. Crit Care Med 1988; 16:1110–1116.[Medline]
- Jenkins KJ, Gauvreau K, Newburger JW, Spray TL, Moller JH, Iezzoni LI. Consensus-based method for risk adjustment for surgery for congenital heart disease. J Thorac Cardiovasc Surg 2002; 123:110–118.[Abstract/Free Full Text]
- Jenkins KJ, Gauvreau K. Center-specific differences in mortality: preliminary analyses using the Risk Adjustment in Congenital Heart Surgery (RACHS-1) method. J Thorac Cardiovasc Surg 2002; 124:97–104.[Abstract/Free Full Text]
- Larsen HS, Pedersen J, Jacobsen J, Johnsen SP, Hansen OK, Hjortdal V. The RACHS-1 risk categories reflect mortality and length of stay in a Danish population of children operated for congenital heart disease. Eur J Cardiothorac Surg 2005; 28:877–881.[Abstract/Free Full Text]
- Boething D, Jenkins KJ, Hecker H, Thies W-R, Breymann T. The RACHS-1 risk categories reflect mortality and length of hospital stay in large German pediatric cardiac surgery population. Eur J Cardiothorac Surg 2004; 26:12–17.[Abstract/Free Full Text]
- Kang N, Cole T, Tsang V, Elliot M, deLeval M. Risk stratification in paediatric open-heart surgery. Eur J Cardio-thorac Surg 2004; 26:3–11.[Abstract/Free Full Text]
- Marcin JP, Pollack MM. Review of the methodologies and applications of scoring systems in neonatal and pediatric intensive care. Pediatr Crit Care Med 2000; 1:20–27.[CrossRef][Medline]
- Lacour-Gayet F, Clarke DR. Aristotle Committee. The Aristotle method: a new concept to evaluate quality of care based on complexity. Curr Opin Pediatr 2005; 17:412–417.[CrossRef][Medline]
- Al-Radi OO, Harrell FE, Caldarone CA, McCrindle BW, Jacobs JP, Williams MG, Van Arsdell GS, Williams WG. Case complexity scores in congenital heart surgery: a comparative study of Aristotles Basic Complexity score and Risk Adjustment in Congenital Heart Surgery (RACHS-1) system. J Thorac Cardiovasc Surg 2007; 133:865–875.[Abstract/Free Full Text]
- Kang N, Tsang VT, Elliott MJ, de Leval MR, Cole TJ. Does the Aristotles score predict outcome in congenital heart surgery? Eur J Cardiothorac Surg 2006; 29:986–990.[Abstract/Free Full Text]
- Proulx F, Gauthier M, Nadeau D, Lacroix J, Farrell CA. Timing and predictors of death in pediatric patients with multiple organ system failure. Crit Care Med 1994; 22:1025–1031.[Medline]
- Jones GD, Thorburn K, Tigg A, Murdoch IA. Preliminary data: PIM vs. PRISM in infants and children post cardiac surgery in a UK PICU. Intensive Care Med 2000; 26:145.[CrossRef][Medline]
- Slater A, Shann F. The suitability of the Pediatric Index of Mortality (PIM), PIM2, the Pediatric Risk of Mortality (PRISM), and PRISM III for monitoring the quality of pediatric intensive care in Australia and New Zealand. Pediatr Crit Care Med 2004; 5:447–454.[CrossRef][Medline]
- Welke KF, Shen I, Ungerleider RM. Current assessment of mortality rates in congenital cardiac surgery. Ann Thorac Surg 2006; 82:164–171.[Abstract/Free Full Text]
- Lacour-Gayet F, Maruszewski B, Mavroudis C, Jones JP, Elliot MJ. Presentation of the International Nomenclature for Congenital Heart Surgery. The long way from nomenclature to collection of validated data at the EACTS. Eur J Cardiothorac Surg 2000; 18:128–135.[Abstract/Free Full Text]
|
|
Top
Abstract
1. Introduction
