Confusion matrix

Terminology and derivations
from a confusion matrix

condition positive (P) the number of real positive cases in the data condition negative (N) the number of real negative cases in the data true positive (TP) A test result that correctly indicates the presence of a condition or characteristic true negative (TN) A test result that correctly indicates the absence of a condition or characteristic false positive (FP) A test result which wrongly indicates that a particular condition or attribute is present false negative (FN) A test result which wrongly indicates that a particular condition or attribute is absent sensitivity, recall, hit rate, or true positive rate (TPR) $\mathrm {TPR} ={\frac {\mathrm {TP} }{\mathrm {P} }}={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FN} }}=1-\mathrm {FNR}$ specificity, selectivity or true negative rate (TNR) $\mathrm {TNR} ={\frac {\mathrm {TN} }{\mathrm {N} }}={\frac {\mathrm {TN} }{\mathrm {TN} +\mathrm {FP} }}=1-\mathrm {FPR}$ precision or positive predictive value (PPV) $\mathrm {PPV} ={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FP} }}=1-\mathrm {FDR}$ negative predictive value (NPV) $\mathrm {NPV} ={\frac {\mathrm {TN} }{\mathrm {TN} +\mathrm {FN} }}=1-\mathrm {FOR}$ miss rate or false negative rate (FNR) $\mathrm {FNR} ={\frac {\mathrm {FN} }{\mathrm {P} }}={\frac {\mathrm {FN} }{\mathrm {FN} +\mathrm {TP} }}=1-\mathrm {TPR}$ fall-out or false positive rate (FPR) $\mathrm {FPR} ={\frac {\mathrm {FP} }{\mathrm {N} }}={\frac {\mathrm {FP} }{\mathrm {FP} +\mathrm {TN} }}=1-\mathrm {TNR}$ false discovery rate (FDR) $\mathrm {FDR} ={\frac {\mathrm {FP} }{\mathrm {FP} +\mathrm {TP} }}=1-\mathrm {PPV}$ false omission rate (FOR) $\mathrm {FOR} ={\frac {\mathrm {FN} }{\mathrm {FN} +\mathrm {TN} }}=1-\mathrm {NPV}$ Positive likelihood ratio (LR+) $\mathrm {LR+} ={\frac {\mathrm {TPR} }{\mathrm {FPR} }}$ Negative likelihood ratio (LR-) $\mathrm {LR-} ={\frac {\mathrm {FNR} }{\mathrm {TNR} }}$ prevalence threshold (PT) $\mathrm {PT} ={\frac {\sqrt {\mathrm {FPR} }}{{\sqrt {\mathrm {TPR} }}+{\sqrt {\mathrm {FPR} }}}}$ threat score (TS) or critical success index (CSI) $\mathrm {TS} ={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FN} +\mathrm {FP} }}$ Prevalence ${\frac {\mathrm {P} }{\mathrm {P} +\mathrm {N} }}$ accuracy (ACC) $\mathrm {ACC} ={\frac {\mathrm {TP} +\mathrm {TN} }{\mathrm {P} +\mathrm {N} }}={\frac {\mathrm {TP} +\mathrm {TN} }{\mathrm {TP} +\mathrm {TN} +\mathrm {FP} +\mathrm {FN} }}$ balanced accuracy (BA) $\mathrm {BA} ={\frac {TPR+TNR}{2}}$ F1 score is the harmonic mean of precision and sensitivity: $\mathrm {F} _{1}=2\times {\frac {\mathrm {PPV} \times \mathrm {TPR} }{\mathrm {PPV} +\mathrm {TPR} }}={\frac {2\mathrm {TP} }{2\mathrm {TP} +\mathrm {FP} +\mathrm {FN} }}$ phi coefficient (φ or rφ) or Matthews correlation coefficient (MCC) $\mathrm {MCC} ={\frac {\mathrm {TP} \times \mathrm {TN} -\mathrm {FP} \times \mathrm {FN} }{\sqrt {(\mathrm {TP} +\mathrm {FP} )(\mathrm {TP} +\mathrm {FN} )(\mathrm {TN} +\mathrm {FP} )(\mathrm {TN} +\mathrm {FN} )}}}$ Fowlkes–Mallows index (FM) $\mathrm {FM} ={\sqrt {{\frac {TP}{TP+FP}}\times {\frac {TP}{TP+FN}}}}={\sqrt {PPV\times TPR}}$ informedness or bookmaker informedness (BM) $\mathrm {BM} =\mathrm {TPR} +\mathrm {TNR} -1$ markedness (MK) or deltaP (Δp) $\mathrm {MK} =\mathrm {PPV} +\mathrm {NPV} -1$ Diagnostic odds ratio (DOR) $\mathrm {DOR} ={\frac {\mathrm {LR+} }{\mathrm {LR-} }}$ Sources: Fawcett (2006), Piryonesi and El-Diraby (2020), Powers (2011), Ting (2011), CAWCR, D. Chicco & G. Jurman (2020, 2021), Tharwat (2018). Balayla (2020)

In the field of machine learning and specifically the problem of statistical classification, a confusion matrix, also known as an error matrix, is a specific table layout that allows visualization of the performance of an algorithm, typically a supervised learning one (in unsupervised learning it is usually called a matching matrix). Each row of the matrix represents the instances in an actual class while each column represents the instances in a predicted class, or vice versa – both variants are found in the literature. The name stems from the fact that it makes it easy to see whether the system is confusing two classes (i.e. commonly mislabeling one as another).

It is a special kind of contingency table, with two dimensions ("actual" and "predicted"), and identical sets of "classes" in both dimensions (each combination of dimension and class is a variable in the contingency table).

Example

Given a sample of 12 individuals, 8 that have been diagnosed with cancer and 4 that are cancer-free, where individuals with cancer belong to class 1 (positive) and non-cancer individuals belong to class 0 (negative), we can display that data as follows:

Individual Number	1	2	3	4	5	6	7	8	9	10	11	12
Actual Classification	1	1	1	1	1	1	1	1	0	0	0	0

Assume that we have a classifier that distinguishes between individuals with and without cancer in some way, we can take the 12 individuals and run them through the classifier. The classifier then makes 9 accurate predictions and misses 3: 2 individuals with cancer wrongly predicted as being cancer-free (sample 1 and 2), and 1 person without cancer that is wrongly predicted to have cancer (sample 9).

Individual Number	1	2	3	4	5	6	7	8	9	10	11	12
Actual Classification	1	1	1	1	1	1	1	1	0	0	0	0
Predicted Classification	0	0	1	1	1	1	1	1	1	0	0	0

Notice, that if we compare the actual classification set to the predicted classification set, there are 4 different outcomes that could result in any particular column. One, if the actual classification is positive and the predicted classification is positive (1,1), this is called a true positive result because the positive sample was correctly identified by the classifier. Two, if the actual classification is positive and the predicted classification is negative (1,0), this is called a false negative result because the positive sample is incorrectly identified by the classifier as being negative. Third, if the actual classification is negative and the predicted classification is positive (0,1), this is called a false positive result because the negative sample is incorrectly identified by the classifier as being positive. Fourth, if the actual classification is negative and the predicted classification is negative (0,0), this is called a true negative result because the negative sample gets correctly identified by the classifier.

We can then perform the comparison between actual and predicted classifications and add this information to the table, making correct results appear in green so they are more easily identifiable.

Individual Number	1	2	3	4	5	6	7	8	9	10	11	12
Actual Classification	1	1	1	1	1	1	1	1	0	0	0	0
Predicted Classification	0	0	1	1	1	1	1	1	1	0	0	0
Result	FN	FN	TP	TP	TP	TP	TP	TP	FP	TN	TN	TN

The template for any binary confusion matrix uses the four kinds of results discussed above (true positives, false negatives, false positives, and true negatives) along with the positive and negative classifications. The four outcomes can be formulated in a 2×2 confusion matrix, as follows:

		Predicted condition		Sources:
	Total population = P + N	Positive (PP)	Negative (PN)
Actual condition	Positive (P)	True positive (TP)	False negative (FN)
	Negative (N)	False positive (FP)	True negative (TN)

The color convention of the three data tables above were picked to match this confusion matrix, in order to easily differentiate the data.

Now, we can simply total up each type of result, substitute into the template, and create a confusion matrix that will concisely summarize the results of testing the classifier:

		Predicted condition
	Total 8 + 4 = 12	Cancer 7	Non-cancer 5
Actual condition	Cancer 8	6	2
	Non-cancer 4	1	3

In this confusion matrix, of the 8 samples with cancer, the system judged that 2 were cancer-free, and of the 4 samples without cancer, it predicted that 1 did have cancer. All correct predictions are located in the diagonal of the table (highlighted in green), so it is easy to visually inspect the table for prediction errors, as values outside the diagonal will represent them. By summing up the 2 rows of the confusion matrix, one can also deduce the total number of positive (P) and negative (N) samples in the original dataset, i.e. $P=TP+FN$ and $N=FP+TN$.

Table of confusion

In predictive analytics, a table of confusion (sometimes also called a confusion matrix) is a table with two rows and two columns that reports the number of true positives, false negatives, false positives, and true negatives. This allows more detailed analysis than simply observing the proportion of correct classifications (accuracy). Accuracy will yield misleading results if the data set is unbalanced; that is, when the numbers of observations in different classes vary greatly.

For example, if there were 95 cancer samples and only 5 non-cancer samples in the data, a particular classifier might classify all the observations as having cancer. The overall accuracy would be 95%, but in more detail the classifier would have a 100% recognition rate (sensitivity) for the cancer class but a 0% recognition rate for the non-cancer class. F1 score is even more unreliable in such cases, and here would yield over 97.4%, whereas informedness removes such bias and yields 0 as the probability of an informed decision for any form of guessing (here always guessing cancer).

According to Davide Chicco and Giuseppe Jurman, the most informative metric to evaluate a confusion matrix is the Matthews correlation coefficient (MCC).

Other metrics can be included in a confusion matrix, each of them having their significance and use.

		Predicted condition		Sources:
	Total population = P + N	Positive (PP)	Negative (PN)	Informedness, bookmaker informedness (BM) = TPR + TNR − 1	Prevalence threshold (PT) =${\mathsf {\tfrac {{\sqrt {{\text{TPR}}\times {\text{FPR}}}}-{\text{FPR}}}{{\text{TPR}}-{\text{FPR}}}}}$
Actual condition	Positive (P)	True positive (TP), hit	False negative (FN), type II error, miss, underestimation	True positive rate (TPR), recall, sensitivity (SEN), probability of detection, hit rate, power = TP/P = 1 − FNR	False negative rate (FNR), miss rate = FN/P = 1 − TPR
	Negative (N)	False positive (FP), type I error, false alarm, overestimation	True negative (TN), correct rejection	False positive rate (FPR), probability of false alarm, fall-out = FP/N = 1 − TNR	True negative rate (TNR), specificity (SPC), selectivity = TN/N = 1 − FPR
	Prevalence = P/P + N	Positive predictive value (PPV), precision = TP/PP = 1 − FDR	False omission rate (FOR) = FN/PN = 1 − NPV	Positive likelihood ratio (LR+) = TPR/FPR	Negative likelihood ratio (LR−) = FNR/TNR
	Accuracy (ACC) = TP + TN/P + N	False discovery rate (FDR) = FP/PP = 1 − PPV	Negative predictive value (NPV) = TN/PN = 1 − FOR	Markedness (MK), deltaP (Δp) = PPV + NPV − 1	Diagnostic odds ratio (DOR) = LR+/LR−
	Balanced accuracy (BA) = TPR + TNR/2	F1 score = 2 PPV × TPR/PPV + TPR = 2 TP/2 TP + FP + FN	Fowlkes–Mallows index (FM) = $\scriptstyle {\mathsf {\sqrt {{\text{PPV}}\times {\text{TPR}}}}}$	Matthews correlation coefficient (MCC) =$\scriptstyle {\mathsf {\sqrt {{\text{TPR}}\times {\text{TNR}}\times {\text{PPV}}\times {\text{NPV}}}}}$$\scriptstyle -{\mathsf {\sqrt {{\text{FNR}}\times {\text{FPR}}\times {\text{FOR}}\times {\text{FDR}}}}}$	Threat score (TS), critical success index (CSI), Jaccard index = TP/TP + FN + FP

Confusion matrices with more than two categories

Confusion matrix is not limited to binary classification and can be used in multi-class classifiers as well. The confusion matrices discussed above have only two conditions: positive and negative. For example, the table below summarizes communication of a whistled language between two speakers, zero values omitted for clarity.

Perceived vowel Vowel produced	i	e	a	o	u
i	15		1
e	1		1
a			79	5
o			4	15	3
u				2	2

See also

• Positive and negative predictive values