Talk:Receiver operating characteristic

Can anyone add information about the use of this method in clinical scenarios (eg. examination of risk factors for disease outcomes)?— Preceding unsigned comment added by 131.104.10.194 (talk • contribs) 23:08, 28 September 2006 (UTC)

The Guyatt et al. paper on iron-deficiency anemia is a classic.

Guyatt G, Patterson C, Ali M, Singer J, Levine M, Turpie I, Meyer R (1990). "Diagnosis of iron-deficiency anemia in the elderly.". Am J Med 88 (3): 205-9. PMID 2178409.

You're welcome to add an example you've come across (this is the encyclopedia anyone can edit)... or write a section after digesting Guyatt's paper -- which makes use of the concept. Nephron  T|C 19:00, 14 December 2006 (UTC)

Merging of articles

I suggest that the articles "Receiver operating characteristic" and "Detection theory" (Signal detection theory) should be merged. The merged article should seek a middle path in terms of technical formality and jargon (as simple as possible, but not simpler). Merged or not, the two articles should be more clearly compatible, if not entirely consistent, since they are basically trying to explain the same underlying thing. Rbfuld (talk) 23:02, 27 January 2008 (UTC)rbfuld

I agree with the suggestion that the articles ROC and "Reciever operator characteristic" should be merged. Wicked Maven 20:00, 28 January 2007 (UTC)

disappointed

For whom was this written? The author displays considerable erudition, but no desire to make his subject palatable to beginners. The second paragraph was enough to choke and die on. I'm not here to be stymied by a pedant - I need to understand ROC curves. I swear - if I ever learn enough about the subject to do so, I will join Wikipedia and rewrite this #$&* entry.64.59.144.85 02:55, 8 February 2007 (UTC)

ROC example misleading or wrong

I think that the example plot w/ points A, B, C, C' is misleading or wrong. C' is intended (I think) to be an example of the effects of inverting the output of the worse-than-random classifier C. If this is actually what it's meant to represent, the plot is wrong: inverting the output of the classifier doesn't correspond to a mirror reflection across the diagonal, but to a mirroring through the point (0.5,0.5). (Inverting the test set labels correponds to a mirror reflection across the diagonal.) I don't have the file used to create the diagram, or I would fix it myself, so I leave this to whoever posted the diagram. 128.62.104.34 22:27, 23 April 2007 (UTC)

Hmm, mirroring with the point (0.5, 0.5) is not the same as mirroring with the diagonal line. I believe the example means that the output of worse than random classifier can be simply mirrored with the diagonal line to get point above the diagonal line. In the table you can see that C' is an invert classification of C. Perhaps you can read the source here: [1]. — Indon (reply) — 08:09, 24 April 2007 (UTC)

I believe the critique above is correct. Reading your cited source confirms that if you read carefully. Fawcett has the mirroring wrong when he says it is across the diagonal, though his explanation of reversing the decision is correct. that is, true positives become false negatives and vice versa. The contingency table on the wiki page for C' is not an inverted classification of C Because the inversion must occur on the columns in the example and not the rows due to the conjunctive equations. Reversing all decisions would then swap the values in the first row with the values in the second row. Therefore, the mirroring is through the point (0.5,0.5), and a true C' which is a reversed decision of C would be at the point (.12,.76) which is still "better than" pt. A. The explanation after the words about mirroring are what is causing confusion, but it would be useful to inform the reader how the mirroring really takes place. Snthor 15:19, 9 August 2007 (UTC)

No matter whether we have to mirror with the point (0.5, 0.5) or at the diagonal line, if we agree that any point under the diagonal line can be mirrored onto the other side then the lower right corner also represents a perfect classification. Therefore IMHO the two arrows labeled "better" and "worse" are misleading too. The closer towards the diagonal line, the worse; the closer towards either the left top edge or the right bottom edge, the better. One solution might be making both arrows two-headed. The heads pointing towards the edges should be labeled "better", the heads pointing towards the diagonal line should be labeled "worse". I am just afraid that people new to the topic will still be confused. Different approach: points under the diagonal line must be mirrored first before they can be compared. Put only one two-headed arrow in the upper triangle. Stevemiller 03:58, 9 October 2007 (UTC)

The lower right triangle corresponds to worse-than-random classifiers, so imho the arrow labels are correct. One might add arrows labeled "higher classification power" pointing away from the diagonal. I agree that the presented contingency table for C' is incorrect, instead of interchanging the columns one has to interchange the rows, since the row index represents the suggested classification. When calculating the TPR and FPR for the modified matrix, one finds that TPR changes to 1-TPR and FPR changes to 1-FPR, so also the presented numbers below the matrix are wrong, as well as the position of C' in the plot. The replacement of (x,y) by (1-x,1-y) corresponds to a mirroring at (0.5, 0.5). Kero6581 (talk) 08:12, 6 May 2009 (UTC)

ok, I corrected the text so that the squares are now correct. The only thing left is to correct the diagram such that the point C' is at (.12,.76). I hope Indon still has his original file, that would make it much easier to correct the figure. Greetings --hroest 12:08, 10 July 2009 (UTC)

Inconsistent Notation

The notation used in the figure ("How a ROC curve can be interpreted") in the fourth section ("Further interpretations") is inconsistent with the notation introduced in the first two sections ("Basic concept" and "ROC Space"). The figure uses TP, FP, TN and FN instead of TPR, FPR, TNR and FNR. Aside from being confusing, it is actually misleading since TP, FP, etc. were already introduced in the earlier sections as having different meanings than TPR, FPR, etc. What is more, the figure's notation isn't even internally consistent. The axis labels on the ROC graph should be "TP" and "FP", not "P(TP)" and "P(FP)". Alternatively, to show explicit dependence of the true positive rate and false positive rate on the threshold value, the axis labels could be "TP(θ)" and "FP(θ)", where the threshold value θ needs then to be introduced in the graph of the probability density curves for the detection statistic. And while I'm picking nits, why aren't the axes of the probability density graph labelled, and for that matter, why don't any of the three subfigures in this image have titles?

Don't get me wrong, I don't want to get rid of this image. For me it is the one illustration that allowed me to "get" what the ROC curve actually quantifies. Which is why I think it is important that it be brought into conformance with the rest of the article. I can see from the article's history that the image itself predates the discussion in the "Basic concept" and "ROC space" sections, so I imagine that the original creator of the image (Kku?) might be resistant to having it replaced with an updated version. However, our collective goal is for the overall article to be as clear as possible, and the best way that I can see to do that is to maintain the notation of the "Basic concept" section and to update the figure accordingly.

Here are the changes that I would propose to the figure:

- Titles for each of the three subfigures (these could be placed in the figure caption as long 
  as the subfigures are labelled with a), b) and c))
- Axis labels where appropriate
- Replace TP, FP, etc. with TPR, FPR, etc.
- Introduce θ as the threshold value and replace P(TP) and P(FP) with TPR(θ) and FPR(θ)

One other thing worth mentioning on the topic of consistency, is that the confusion matrix in the figure is of a different form than that introduced in the "Basic concept" section, having its columns sum to 1 rather than to the respective probabilities of the underlying event occurring or not. I don't think that this should be changed in the name of consistency, however, because as it is, it provides a direct link between the two other subfigures in the image. If the confusion matrix were altered so that the columns sum respectively to P and N, then this link would be lost and the subfigure would only serve to introduce (dare I say it?) confusion.

Personally, if effort will be taken to update this figure, I think it might be wortwhile to introduce one more subfigure at the top showing the information flow (underlying two-state process --> observable data --> detection statistic --> decision), but this may not be the best choice of language if my stated goal is to enforce consistency with the rest of the article.

JanRu 20:26, 26 April 2007 (UTC)

ROC space and metrics

In the section, "ROC Space", the info-box is referenced as containing evaluation metrics. Perhaps inadvertently, the word metric is hyperlinked to the wiki page on metrics, as in metric space distances. The reader may be inclined to believe from this that the info-box contains metrics. This is not the case. None of the "evaluation" metrics listed are true metrics in the mathematical sense. I would recommend deleting the hyperlink.

In a similar discussion, the notion of a ROC space is incorrect. What is meant by space? It is neither a vector space nor a topological space and so the verbiage is abused, even though it appears in some of the cited literature. A ROC graph is what is presented and its limitations are made clear, but the notion of a space is ill advised. I recommend titling the section as ROC graphs.

Snthor 14:21, 9 August 2007 (UTC)

d' (d-prime)

The article says about d' "... under the assumption that both these distributions are normal with the SAME standard deviation" (my emphasis). But the article about d' uses "the standard deviation of the noise distribution". Stevemiller 04:46, 10 October 2007 (UTC)

Number of observations - irrelevant?

Does the number of observations affect the ROC curve at all? With only one observation isn't it possible to have 100% sensitivity (no false negatives) and 100% specificity (no false positives)? Presumably I'm missing something because if that's the case having a good point on a ROC curve doesn't guarantee a good classifier. pgr94 (talk) 18:55, 24 January 2009 (UTC)

You'd have a good ROC curve, but no statistical reason to believe that this curve is representative of actual behaviour. --60.234.219.72 (talk) 01:27, 25 August 2009 (UTC)

Terminology and derivations from a confusion matrix

This is an excellent addition to this article - very helpful for people wanting to dive deeper. Thanks so much. 128.220.160.6 (talk) 00:48, 9 March 2009 (UTC) what does eqv. stand for????

Math Parser Error

From Revision #314192656 by 151.148.122.100 "Failed to parse (unknown function\MCC): \MCC = (TPTN - FPFN)/ \sqrt{P N P' N'}" ... reverted to working formula, however, the formula is rendered as a PNG - anyone who knows how to enforce text rendering, please be my guest. - Dlefree-loc-work (talk) 08:55, 21 September 2009 (UTC)