Evaluating the Inaccuracy of the NSQIP Surgical Risk Calculator in Predicting 30-Day Complications in Plastic Surgery
Marten N. Basta MD1, Vinay Rao, MD1, Marcelo Paiva, MPP1, Paul Y. Liu, MD FACS1, Albert S. Woo, MD FACS1, John P. Fischer, MD MPH2, Karl H. Breuing, MD FACS1
1 Brown University Plastic Surgery Department, Providence, RI, USA 2 University of Pennsylvania, Plastic Surgery Division, Philadelphia, PA, USA.
Background
Preoperative surgical risk assessment is a major component of clinical decision-making. The ability to provide accurate, individualized risk estimates has become critical due to growing emphasis on quality metrics benchmarks. The ACS NSQIP Universal Risk Calculator was designed to quantify patient-specific risk across various surgeries. Its applicability to plastic surgery is unclear, however, with multiple studies reporting inaccuracies among certain patient populations. This study utilizes meta-analysis to evaluate the NSQIP Risk Calculator's ability to predict complications among patients having plastic surgery.
Methods
OVID Medline and PubMed were searched for all studies evaluating the predictive accuracy of the NSQIP Risk Calculator in plastic/reconstructive surgery, including oncologic defect reconstruction, ventral hernia repair, and body contouring. Only studies directly compared Risk Calculator predicted to observed complication rates were included. The primary outcome was Area Under the Curve (AUC), which measures the ability of the Surgical Risk Calculator to predict 30-day complications, ranging from 0.50 (prediction no better than random chance) to 1.0 (perfect prediction). Risk Calculator accuracy was assessed for each complication via DerSimonian and Laird random-effects analytic model. Data heterogeneity was evaluated with the I2 statistic, judged low (I2<50%) or borderline/unacceptably high (I2>50%). All analyses were conducted in StataSE 16.1 (StataCorp LP, College Station, Texas).
Results
10 of the 296 studies screened met criteria for inclusion (2,416 patients). Studies were classified as follows: head and neck oncologic/reconstruction (head & neck: N=5, breast: N=1, extremity: N=1), open ventral hernia repair (N=2), panniculectomy (N=1). Sufficient data was reported to meta-analyze 8 NSQIP-defined complications. Predictive accuracy was poor for medical complications (pulmonary AUC=0.67 [0.48-0.87], cardiac AUC=0.66 [0.20-0.99], venous thromboembolism AUC=0.55 [0.47-0.63]). Similarly, predictive accuracy for surgical complications was unsatisfactory (surgical site infection AUC=0.55 [0.46-0.63), reoperation AUC=0.54 [0.49-0.58), serious complication AUC=0.58 [0.43-0.73]. Finally, any complication was poorly predicted by the NSQIP Risk Calculator (AUC=0.60 [0.57-0.64]). Although mortality was accurately predicted in 2 studies (AUC=0.87 [0.54-0.99], heterogeneity was high with I2=68%. Otherwise, heterogeneity was minimal (I2=0%) or acceptably low (I2<50%) for all other outcomes.
Conclusions
The NSQIP Universal Surgical Risk Calculator, aimed at offering individualized quantifiable risk estimates for surgical complications, consistently demonstrated poor risk discrimination in this plastic surgery-focused meta-analysis. The limitations of the Risk Calculator are perhaps most pronounced where complex, multidisciplinary reconstructions are needed. Future efforts should identify targets for improving Risk Calculator reliability in order to better counsel patients in the perioperative setting and guide appropriate healthcare resource allocation.
Back to 2021 Abstracts