ROC curves comparison

Very often the aim of studies is the comparison of the size of the area under the ROC curve ($AUC_1$) with the area under another ROC curve ($AUC_2$). The ROC curve with a greater area usually allows a more precise classification of objects. Methods for comparing the areas depend on the model of the study.

Hypotheses:

\begin{array}{cl} \mathcal{H}_0: & AUC_1=AUC_2, \\ \mathcal{H}_1: & AUC_1\neq AUC_2. \end{array}

The test statistics has the form presented below:

\begin{displaymath} Z=\frac{|AUC_1-AUC_2|}{SE_{AUC_1-AUC_2}}, \end{displaymath}

where:

$AUC_1$, $AUC_2$ and the standard error of the difference in areas $SE_{AUC_1-AUC_2}$ are calculated on the basis of the nonparametric method proposed by DeLong (DeLong E.R. et al., 19881), Hanley J.A., and Hajian-Tilaki K.O. 19972))

Statistics $Z$ has (for large sizes) asymptotic normal distribution.

The p-value, designated on the basis of the test statistic, is compared with the significance level $\alpha$:

\begin{array}{ccl} $ if $ p \le \alpha & \Longrightarrow & $ reject $ \mathcal{H}_0 $ and accept $ \mathcal{H}_1, \\ $ if $ p > \alpha & \Longrightarrow & $ there is no reason to reject $ \mathcal{H}_0. \\ \end{array}

The window with settings for comparing dependent ROC curves is accessed via the menu Advanced statistics → Diagnostic tests → Dependent ROC Curves – comparison.

Hypotheses:

\begin{array}{cl} \mathcal{H}_0: & AUC_1=AUC_2, \\ \mathcal{H}_1: & AUC_1\neq AUC_2. \end{array}

Test statistics (Hanley J.A. and McNeil M.D. 19833)) has the form: \begin{displaymath} Z=\frac{|AUC_1-AUC_2|}{\sqrt{SE_{AUC_1}^2-SE_{AUC_2}^2}}, \end{displaymath}

where:

$AUC_1$, $AUC_2$ and standard errors of areas $SE_{AUC_1}$, $SE_{AUC_2}$ are calculated on the basis of:

  • nonparametric method DeLong (DeLong E.R. et al. 1988(DeLong E.R., DeLong D.M., Clarke-Pearson D.L., (1988), Comparing the areas under two or more correlated receiver operating curves: A nonparametric approach. Biometrics 44:837-845)), Hanley J.A., and Hajian-Tilaki K.O. 19974)) - recommended,
  • nonparametric Hanley-McNeil method (Hanley J.A. and McNeil M.D. 19825)),
  • method which presumes double negative exponential distribution (Hanley J.A. and McNeil M.D. 1982\cite{hanley_mcn}) - computed only when groups (+) and (–) are equinumerous.

Statistics $Z$ has (for large sizes) asymptotic normal distribution.

The p-value, designated on the basis of the test statistic, is compared with the significance level $\alpha$:

\begin{array}{ccl} $ if $ p \le \alpha & \Longrightarrow & $ reject $ \mathcal{H}_0 $ and accept $ \mathcal{H}_1, \\ $ if $ p > \alpha & \Longrightarrow & $ there is no reason to reject $ \mathcal{H}_0. \\ \end{array}

The window with settings for comparing independent ROC curves is accessed via the menu Advanced statistics→Diagnostic tests→Independent ROC Curves – comparison

EXAMPLE cont. (bacteriemia.pqs file)

We will make 2 comparisons:

  • 1) We will construct 2 ROC curves to compare the diagnostic value of parameters WBC and PCT;
  • 2) We will construct 2 ROC curves to compare the diagnostic value of PCT parameter for boys and girls.
  • ad1)

Both parameters, WBC and PCT, are stimulants (in bacteremia their values are high). In the course of the comparison of the diagnostic value of those parameters we verify the following hypotheses:

\begin{array}{cl} \mathcal{H}_0: & $the area under ROC curve for WBC $=$ the area under the ROC curve for PCT$, \\ \mathcal{H}_1: & $the area under ROC curve for WBC $\neq $ the area under the ROC curve for PCT$. \end{array}

The calculated ares are $AUC_{WBC}=086$, $AUC_{PCT}=0.90$. On the basis of the adopted level $\alpha=0.05$, based on the obtained value $p$=0.13032 we conclude that we cannot determine which of the parameters: WBC or PCT is better for diagnosing bacteremia.

  • ad2) PCT parameter is a stimulant (its value is high in bacteremia). In the course of the comparison of its diagnostic value for girls and boys we verify the following hypotheses:

\begin{array}{cl} \mathcal{H}_0: & $the area under ROC curve for $PCT_f$ $=$ the area under ROC curve for $PCT_m$$, \\ \mathcal{H}_1: & $the area under ROC curve for $PCT_f$ $\neq $ the area under ROC curve for $PCT_m$$. \end{array}

The calculated areas are $AUC_f=0.86$, $AUC_m=0.91$. Therefore, on the basis of the adopted level $\alpha=0.05$, based on the obtained value $p$=0.6372 we conclude that we cannot select the sex for which PCT parameter is better for diagnosing bacteremia.

1)
DeLong E.R., DeLong D.M., Clarke-Pearson D.L., (1988), Comparing the areas under two or more correlated receiver operating curves: A nonparametric approach. Biometrics 44:837-845
2) , 4)
Hanley J.A. i Hajian-Tilaki K.O. (1997), Sampling variability of nonparametric estimates of the areas under receiver operating characteristic curves: an update. Academic radiology 4(1):49-58
3)
Hanley J.A. i McNeil M.D. (1983), A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 148: 839-843
5)
Hanley J.A. i McNeil M.D. (1982), The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143(1):29-36