External longitudinal validation of metabolic syndrome risk scores for cardiovascular, diabetes, and all-cause mortality in the US NHANES Linked Mortality File

Pre-registered observational cohort study, NHANES 1999 to 2018 linked to the NHANES Linked Mortality File 2019 release.

Pre-registered Manuscript draft on origin/master Target: JAMA Network Open / npj Digital Medicine Author: Jay Hemnani

Headline finding

For cardiovascular and all-cause mortality, the clinical risk equations (Framingham 2008, PCE) clearly dominate the metabolic-syndrome-derived scores. The XGBoost ceiling on the MetS inputs alone does not close the gap.
For diabetes-related mortality, RMRS competes head-to-head with FINDRISC on point discrimination (delta-AUC +0.014 favoring RMRS, 95% bootstrap CI -0.051 to +0.090) and gives modest incremental value when stacked on FINDRISC (IDI 0.0042, continuous NRI 0.139, both CIs exclude zero). The FINDRISC comparator now uses NHANES-derived family history, prediabetes, and physical-activity items rather than placeholders.
The B9 decision tree is consistently outperformed by the continuous RMRS on every outcome, and the XGBoost ceiling on the same inputs exceeds both, so the MetS signal is not the constraint. A CART refit on a held-out NHANES split recovers RMRS-level discrimination (diabetes-related AUC 0.783), placing the B9 gap on the transportability of the Korean-calibrated splits rather than on the tree method itself.

Pre-registered H3 read. The bootstrap 95% percentile lower bound at -0.051 sits 0.001 AUC units below the registered -0.05 non-inferiority margin, so the test does not formally establish non-inferiority, though the miss is smaller than the bootstrap resampling resolution. The DeLong-based CI was similar at -0.053 to +0.080. Both intervals locate RMRS and FINDRISC within a narrow band, and the point estimate favors RMRS.

Study at a glance

Cohort. Pooled NHANES 1999 to 2018 fasting subsample, adults 20 to 79 years, with non-missing values on the five metabolic syndrome components and eligible mortality linkage. N = 17,031 (all-cause); cardiovascular and diabetes-related analyses use the 13,836 participants from the 1999 to 2014 cycles with full leading-cause coding.

Outcomes. All-cause mortality (10-year cap, 806 deaths, full cohort), cardiovascular mortality (10-year cap, 173 deaths), and diabetes-related mortality under a broadened definition combining underlying-cause diabetes mellitus, underlying-cause nephritis, and the LMF DIABETES contributing-cause flag (15-year cap, 77 deaths). Cardiovascular and diabetes counts are within the cause-coded subcohort.

Scores compared. RMRS (continuous metabolic-syndrome risk score via triangular areal similarity); B9 (decision-tree formulation of the same five components); ACC/AHA Pooled Cohort Equations (primary cardiovascular baseline); Framingham 2008 (secondary cardiovascular baseline); FINDRISC (type 2 diabetes screening baseline).

Methods. Survey-weighted Fine-Gray competing-risks (cardiovascular, diabetes) and Cox proportional hazards (all-cause). Time-dependent IPCW AUC at off-cap horizons. Competing-risks decision curve analysis with Fine-Gray cumulative-incidence recalibration and Aalen-Johansen net benefit. DeLong-based delta-AUC plus 500-rep PSU-cluster bootstrap CIs. Continuous NRI and IDI. XGBoost ML ceiling with PSU x stratum grouped cross-validation.

Diagrams

Pipeline diagram: data flow from raw NHANES and LMF inputs through scripts 01 to 14, ending in manuscript and OSF prereg outputs. — Pipeline. Raw NHANES and LMF inputs flow through cohort assembly, score computation, three parallel survival analyses, decision curve analysis, pairwise comparisons, and the XGBoost ML ceiling, with the 500-rep PSU-cluster bootstrap as the registered sensitivity.

Scoring diagram: 5 risk scores map from NHANES inputs to 3 mortality outcomes, with H2 and H3 contrast roles marked. — Scoring system. The five risk scores draw on overlapping NHANES inputs and are tested against the three mortality outcomes. H2 is PCE vs Framingham on cardiovascular mortality; H3 is the RMRS non-inferiority test against FINDRISC on diabetes-related mortality at the -0.05 AUC margin.

Repo directory tree: scripts, R/scores, R/utils, tests, data, results, manuscript, prereg, diagrams. — Repository structure. Code lives in `scripts/`, `R/scores/`, and `R/utils/`. Outputs land in `results/`. The OSF pre-registration draft is in `prereg/` and the manuscript LaTeX source plus rendered PDF in `manuscript/`.

Headline numbers

Time-dependent AUC at primary horizons (500-rep PSU-cluster bootstrap 95% CIs)

Score	Outcome	Horizon	AUC (95% CI)
Framingham 2008	All-cause	9.5y	0.810 (0.795, 0.825)
PCE	All-cause	9.5y	0.758 (0.736, 0.778)
FINDRISC	All-cause	9.5y	0.672 (0.651, 0.695)
RMRS	All-cause	9.5y	0.600 (0.578, 0.622)
B9 tree	All-cause	9.5y	0.525 (0.505, 0.542)
Framingham 2008	Cardiovascular	9.5y	0.858 (0.831, 0.884)
PCE	Cardiovascular	9.5y	0.810 (0.772, 0.841)
RMRS	Cardiovascular	9.5y	0.660 (0.625, 0.693)
B9 tree	Cardiovascular	9.5y	0.560 (0.519, 0.593)
FINDRISC	Diabetes-related	14.5y	0.770 (0.705, 0.826)
RMRS	Diabetes-related	14.5y	0.752 (0.696, 0.804)
B9 tree	Diabetes-related	14.5y	0.582 (0.532, 0.631)

Registered pairwise delta-AUC contrasts

Contrast	Outcome (horizon)	Delta-AUC (95% CI)
RMRS vs FINDRISC (H3 non-inferiority)	Diabetes-related (14.5y)	+0.014 (-0.051, +0.090)
RMRS vs B9	All-cause (9.5y)	+0.071 (+0.049, +0.094)
RMRS vs B9	Cardiovascular (9.5y)	+0.093 (+0.050, +0.135)
RMRS vs B9	Diabetes-related (14.5y)	+0.158 (+0.088, +0.220)
PCE vs Framingham (H2)	Cardiovascular (9.5y)	+0.007 (-0.010, +0.022)

XGBoost ML ceiling on the metabolic syndrome inputs

Outcome (horizon)	ML AUC	Top clinical / MetS score
All-cause (9.5y)	0.813	Framingham 0.810 (no headroom)
Cardiovascular (9.5y)	0.817	Framingham 0.858 (clinical above ML)
Diabetes-related (14.5y)	0.828	FINDRISC 0.770 (6 to 8 AUC pts of headroom)

Code, draft, and pre-registration

Source repository github.com/jayhemnani9910/longitudinal-mets-validation
Manuscript PDF manuscript/main.pdf
Pre-registration draft prereg/osf-preregistration-draft.md
OSF registration pending submission
Result CSVs results/ (allcause, cv, dm, pairwise, dca, xgboost, bootstrap)
Pipeline source scripts/01 to 14

Source papers

The two metabolic syndrome risk scoring methods evaluated here come from the Shin, Oh, and Shim research line.

Shin HJ, Oh J, Shim SS. A robust metabolic syndrome risk score using triangular areal similarity. PeerJ Computer Science, 2024. (RMRS, score B8.)
Shin HJ, Oh J, Shim SS. Machine learning predictive model for metabolic syndrome prevention. PLoS ONE, 2023. (B9 decision tree.)

The clinical risk equation baselines come from the standard literature: Goff et al. (ACC/AHA 2014) for PCE; D'Agostino et al. (Circulation 2008) for Framingham; Lindstrom and Tuomilehto (Diabetes Care 2003) for FINDRISC.