{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Validation of Classifiers\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Text classification is part of supervised Machine Learning (ML). As depicted in the picture above, supervised ML relies on labeled data - not only for training, but also for model testing. It is important that training- and test-datasets are disjoint.\n", "\n", "Once a model is trained, it is applied on test-data and calculates predictions for the test-data input. Since for test-data also the true label (output) is known, these true labels can be compared with the predicted labels. Based on this comparison, different **metrics** for classifier evaluations can be calculated. The most important classifier metrics are described below." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Assume that for 10 test-data samples the true and predicted labels (class-indeces) are as listed in the table below:\n", "\n", "| True Label | Predicted Label |\n", "|:----------:|:---------------:|\n", "| 0 | 0 |\n", "| 0 | 1 |\n", "| 1 | 1 |\n", "| 1 | 1 |\n", "| 1 | 0 |\n", "| 0 | 0 |\n", "| 1 | 0 |\n", "| 0 | 0 |\n", "| 0 | 0 |\n", "| 0 | 0 |\n", "\n", "All of the metrics described below, can be calculated from this comparison. \n", "\n", "## Confusion matrix\n", "\n", "The confusion matrix contains for each pair of classes $i$ and $j$, the number of class $i$ elements, which have been predicted to be class $j$. Usually, each row corresponds to a true-class label and each column corresponds to a predicted class label. \n", "\n", "In the general 2-class confusion matrix, depicted below, the class labels are $P$ (positive) and $N$. The matrix entries are then\n", "\n", "* **TP (True Positives):** Number of samples, which belong to class $P$\n", " and have correctly been predicted to be class $P$\n", " \n", "* **TN (True Negative):** Number of samples, which belong to class $N$\n", " and have correctly been predicted to be class $N$\n", " \n", "* **FP (False Positives):** Number of samples, which belong to class $N$\n", " but have falsely been predicted to be class $P$\n", " \n", "* **FN (False Negatives):** Number of samples, which belong to class $P$\n", " but have falsely been predicted to be class $N$\n", " \n", "\n", "\n", "For the given example of predicted and true class labels the confusion matrix is:\n", "\n", "\n", "\n", "\n", "## Accuracy\n", "\n", "Accuracy is the ratio of correct predictions among all predictions. For the 2-class problem and the labels $P$ and $N$, accuracy can be calculated from the entries of the confusion matrix:\n", "\n", "$$\n", "Acc =\\frac{TP+TN}{TP+TN+FP+FN}\n", "$$\n", "\n", "\n", "In the example the accuracy is\n", "$$\n", "Acc =\\frac{5+2}{5+2+1+2}=0.7\n", "$$\n", "\n", "\n", "## Recall\n", "The recall of class $i$ is the ratio of correctly predicted class $i$ elements, among all elements, which truly belong to class $i$. The recall of class $P$ is:\n", "\n", "$$\n", "Rec_P =\\frac{TP}{TP+FN}\n", "$$\n", "\n", "and for class $N$:\n", "\n", "$$\n", "Rec_N =\\frac{TN}{TN+FP}\n", "$$\n", "\n", "In the example:\n", "\n", "$$\n", "Rec_1=\\frac{2}{4} \\quad \\mbox{ and } \\quad Rec_0=\\frac{5}{6}\n", "$$\n", "\n", "\n", "## Precision \n", "\n", "The precision of class $i$ is the ratio of true class $i$ elements, among all elements, which have been predicted to be class $i$. The precision of class $P$ is:\n", "\n", "\n", "$$\n", "Pre_P =\\frac{TP}{TP+FP}\n", "$$\n", "\n", "and for class $N$:\n", "\n", "$$\n", "Pre_N =\\frac{TN}{TN+FN}\n", "$$\n", "\n", "In the example:\n", "\n", "$$\n", "Pre_1=\\frac{2}{3} \\quad \\mbox{ and } \\quad Pre_0=\\frac{5}{7}\n", "$$\n", "\n", "## F1-Score\n", "\n", "The F1-score of a class $i$ is the harmonic mean of this class' precision and recall:\n", "\n", "$$\n", "F1_i = 2 \\cdot \\frac{Pre_i \\cdot Rec_i}{Pre_i + Rec_i}\n", "$$\n", "\n", "In the example: \n", "\n", "$$\n", "F1_1= 2 \\cdot \\frac{\\frac{2}{4} \\frac{2}{3}}{\\frac{2}{4} + \\frac{2}{3}} \\quad \\mbox{ and } \\quad F1_0= 2 \\cdot \\frac{\\frac{5}{6} \\frac{5}{7}}{\\frac{5}{6} + \\frac{5}{7}} \n", "$$\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.5" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 4 }