Confusion matrix

		Predicted condition		Sources:
	Total population = P + N	Predicted Positive (PP)	Predicted Negative (PN)	Informedness, bookmaker informedness (BM) = TPR + TNR − 1	Prevalence threshold (PT) = √TPR × FPR - FPR/TPR - FPR
Actual condition	Positive (P)	True positive (TP), hit	False negative (FN), miss, underestimation	True positive rate (TPR), recall, sensitivity (SEN), probability of detection, hit rate, power = TP/P = 1 − FNR	False negative rate (FNR), miss rate type II error = FN/P = 1 − TPR
	Negative (N)	False positive (FP), false alarm, overestimation	True negative (TN), correct rejection	False positive rate (FPR), probability of false alarm, fall-out type I error = FP/N = 1 − TNR	True negative rate (TNR), specificity (SPC), selectivity = TN/N = 1 − FPR
	Prevalence = P/P + N	Positive predictive value (PPV), precision = TP/PP = 1 − FDR	False omission rate (FOR) = FN/PN = 1 − NPV	Positive likelihood ratio (LR+) = TPR/FPR	Negative likelihood ratio (LR−) = FNR/TNR
	Accuracy (ACC) = TP + TN/P + N	False discovery rate (FDR) = FP/PP = 1 − PPV	Negative predictive value (NPV) = TN/PN = 1 − FOR	Markedness (MK), deltaP (Δp) = PPV + NPV − 1	Diagnostic odds ratio (DOR) = LR+/LR−
	Balanced accuracy (BA) = TPR + TNR/2	F1 score = 2 PPV × TPR/PPV + TPR = 2 TP/2 TP + FP + FN	Fowlkes–Mallows index (FM) = √PPV × TPR	Matthews correlation coefficient (MCC) = √TPR × TNR × PPV × NPV - √FNR × FPR × FOR × FDR	Threat score (TS), critical success index (CSI), Jaccard index = TP/TP + FN + FP

In the field of machine learning and specifically the problem of statistical classification, a confusion matrix, also known as an error matrix, is a specific table layout that allows visualization of the performance of an algorithm, typically a supervised learning one (in unsupervised learning it is usually called a matching matrix). Each row of the matrix represents the instances in a predicted class while each column represents the instances in an actual class (or vice versa). The name stems from the fact that it makes it easy to see if the system is confusing two classes (i.e. commonly mislabeling one as another).

It is a special kind of contingency table, with two dimensions ("actual" and "predicted"), and identical sets of "classes" in both dimensions (each combination of dimension and class is a variable in the contingency table).

Example

Given a sample of 13 pictures, 8 of cats and 5 of dogs, where cats belong to class 1 and dogs belong to class 0,

actual = [1,1,1,1,1,1,1,1,0,0,0,0,0],

assume that a classifier that distinguishes between cats and dogs is trained, and we take the 13 pictures and run them through the classifier, and the classifier makes 8 accurate predictions and misses 5: 3 cats wrongly predicted as dogs (first 3 predictions) and 2 dogs wrongly predicted as cats (last 2 predictions).

prediction = [0,0,0,1,1,1,1,1,0,0,0,1,1]

With these two labelled sets (actual and predictions) we can create a confusion matrix that will summarize the results of testing the classifier:

Actual class Cat Dog Predicted class Cat 5 2 Dog 3 3

In this confusion matrix, of the 8 cat pictures, the system judged that 3 were dogs, and of the 5 dog pictures, it predicted that 2 were cats. All correct predictions are located in the diagonal of the table (highlighted in bold), so it is easy to visually inspect the table for prediction errors, as they will be represented by values outside the diagonal.

In abstract terms, the confusion matrix is as follows:

Actual class P N Predicted class P TP FP N FN TN

where: P = Positive; N = Negative; TP = True Positive; FP = False Positive; TN = True Negative; FN = False Negative.

Table of confusion

In predictive analytics, a table of confusion (sometimes also called a confusion matrix) is a table with two rows and two columns that reports the number of false positives, false negatives, true positives, and true negatives. This allows more detailed analysis than mere proportion of correct classifications (accuracy). Accuracy will yield misleading results if the data set is unbalanced; that is, when the numbers of observations in different classes vary greatly. For example, if there were 95 cats and only 5 dogs in the data, a particular classifier might classify all the observations as cats. The overall accuracy would be 95%, but in more detail the classifier would have a 100% recognition rate (sensitivity) for the cat class but a 0% recognition rate for the dog class. F1 score is even more unreliable in such cases, and here would yield over 97.4%, whereas informedness removes such bias and yields 0 as the probability of an informed decision for any form of guessing (here always guessing cat).

According to Davide Chicco and Giuseppe Jurman, the most informative metric to evaluate a confusion matrix is the Matthews correlation coefficient (MCC).

Assuming the confusion matrix above, its corresponding table of confusion, for the cat class, would be:

		Actual class
		Cat	Non-cat
Predicted class	Cat	5 True Positives	2 False Positives
	Non-cat	3 False Negatives	3 True Negatives

The final table of confusion would contain the average values for all classes combined.

Let us define an experiment from P positive instances and N negative instances for some condition. The four outcomes can be formulated in a 2×2 confusion matrix, as follows:

		True condition
	Total population	Condition positive	Condition negative	Prevalence = Σ Condition positive/Σ Total population		Accuracy (ACC) = Σ True positive + Σ True negative/Σ Total population
Predicted condition	Predicted condition positive	True positive	False positive, Type I error	Positive predictive value (PPV), Precision = Σ True positive/Σ Predicted condition positive		False discovery rate (FDR) = Σ False positive/Σ Predicted condition positive
	Predicted condition negative	False negative, Type II error	True negative	False omission rate (FOR) = Σ False negative/Σ Predicted condition negative		Negative predictive value (NPV) = Σ True negative/Σ Predicted condition negative
		True positive rate (TPR), Recall, Sensitivity (SEN), probability of detection, Power = Σ True positive/Σ Condition positive	False positive rate (FPR), Fall-out, probability of false alarm = Σ False positive/Σ Condition negative	Positive likelihood ratio (LR+) = TPR/FPR	Diagnostic odds ratio (DOR) = LR+/LR−	Matthews correlation coefficient (MCC) = √TPR·TNR·PPV·NPV − √FNR·FPR·FOR·FDR	F1 score = 2 · PPV · TPR/PPV + TPR = 2 · Precision · Recall/Precision + Recall
		False negative rate (FNR), Miss rate = Σ False negative/Σ Condition positive	Specificity (SPC), Selectivity, True negative rate (TNR) = Σ True negative/Σ Condition negative	Negative likelihood ratio (LR−) = FNR/TNR