Machine Learning Model Evaluation Explained: Accuracy, Precision, Recall, F1 Score and ROC-AUC

A classifier doesn't have one universal score. A fraud detector can achieve 99% accuracy by predicting that nothing is ever fraud. It would look brilliant in a slide deck and be completely useless in production.

This guide explains each common metric, what question it answers, and how to pick the right one, using a fraud example you can run with scikit-learn.

The 99% accurate model that catches nothing

Imagine 1,000 card transactions, of which 10 are fraud. Here's a "model" that always says "not fraud":

import numpy as np
from sklearn.metrics import accuracy_score, recall_score
 
y_true = np.array([0] * 990 + [1] * 10)   # 1 = fraud
y_lazy = np.zeros(1000, dtype=int)         # always predicts 0
 
print(accuracy_score(y_true, y_lazy))  # 0.99
print(recall_score(y_true, y_lazy))    # 0.0

99% accuracy, 0% of fraud caught. Accuracy only counts how often you're right overall, and when 99% of cases are boring, being boring scores 99%.

It's like a goalkeeper in a match where the opponent only shoots once. Stand still, the shot goes in, and you can still claim you "handled 99% of the game".

The confusion matrix: four boxes that explain everything

Every metric is built from four counts. For fraud:

• True positive (TP): fraud, and the model flagged it. Good.
• False positive (FP): not fraud, but flagged. A customer's card gets blocked for no reason.
• False negative (FN): fraud, but missed. Money is stolen.
• True negative (TN): not fraud, not flagged. Good.

Let's train a real model on an imbalanced dataset (about 5% positives) and look at its four boxes:

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
 
X, y = make_classification(n_samples=5000, n_features=10, n_informative=4,
                           weights=[0.95, 0.05], random_state=7)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=7)
 
model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
proba = model.predict_proba(X_test)[:, 1]   # probability of fraud
pred = (proba >= 0.5).astype(int)
 
print(confusion_matrix(y_test, pred))

[[1411    6]
 [  47   36]]

Read it as rows = truth, columns = prediction. Top-left: 1,411 true negatives. Top-right: 6 false alarms. Bottom-left: 47 frauds missed. Bottom-right: 36 frauds caught.

stratify=y matters here: it keeps the same 5% fraud rate in the training and test sets, so the test isn't accidentally missing the rare class.

The metrics, one question each

from sklearn.metrics import precision_score, recall_score, f1_score, roc_auc_score
 
print(f"accuracy  {accuracy_score(y_test, pred):.3f}")
print(f"precision {precision_score(y_test, pred):.3f}")
print(f"recall    {recall_score(y_test, pred):.3f}")
print(f"f1        {f1_score(y_test, pred):.3f}")
print(f"roc_auc   {roc_auc_score(y_test, proba):.3f}")

accuracy  0.965
precision 0.857
recall    0.434
f1        0.576
roc_auc   0.915

Accuracy: how often is it right?

(TP + TN) / everything = 96.5%. Sounds great, but the lazy model would score about 94.5% on this test set. The real story is elsewhere.

Precision: when it shouts "fraud", should I believe it?

TP / (TP + FP) = 36 / 42 = 0.857. When this model flags a transaction, it's right 86% of the time. High precision means few blocked innocent customers.

Recall: of all the fraud, how much did it find?

TP / (TP + FN) = 36 / 83 = 0.434. It caught only 43% of the fraud. More than half slipped through. For a bank, that's the number that hurts.

F1: one number balancing both

The harmonic mean of precision and recall: 0.576. The harmonic mean punishes imbalance, so a model can't hide terrible recall behind great precision. Use F1 to compare models when both kinds of mistake matter.

ROC-AUC: how well does it rank?

ROC-AUC ignores the 0.5 threshold and asks a different question: if I pick one random fraud and one random normal transaction, how often does the model give the fraud a higher score? 0.915 means 91.5% of the time.

0.5 is coin-flipping and 1.0 is perfect. A high AUC with low recall (like ours) is good news: the model ranks well, and we're just using a bad threshold.

The threshold is a business decision

Precision and recall come from turning probabilities into yes/no at some cut-off. Move the cut-off and watch the trade:

for t in (0.5, 0.3, 0.15):
    p = (proba >= t).astype(int)
    print(t, f"precision {precision_score(y_test, p):.2f}",
             f"recall {recall_score(y_test, p):.2f}", f"flagged {p.sum()}")

0.5 precision 0.86  recall 0.43  flagged 42
0.3 precision 0.68  recall 0.54  flagged 66
0.15 precision 0.49  recall 0.72  flagged 122

Lower the threshold and you catch more fraud (recall up) but bother more innocent customers (precision down). There's no "correct" threshold in maths. It depends on costs:

• Missing fraud is expensive, a review call is cheap → lower threshold, favour recall.
• Spam filter, where losing a real email is worse than seeing some spam → higher threshold, favour precision.

Think of a Marvel security checkpoint. Tony Stark's scanner can flag every suspicious visitor (high recall) and annoy half of Avengers Tower, or only flag certain threats (high precision) and occasionally let Loki through. Someone has to choose which mistake is worse.

Which metric should you report?

That last row: with rare positives, ROC-AUC can look rosy because there are so many easy negatives. Our model's precision-recall AUC is 0.658, a more sober view of the same model.

What goes wrong

• Reporting only accuracy on imbalanced data. Always show the confusion matrix alongside it.
• Tuning the threshold on the test set. Pick it on a validation set, then report the test result once, or you're grading your own exam.
• Forgetting stratify when splitting. With rare classes, a random split can leave very few positives in the test set, and every metric becomes noise.
• Comparing F1 across datasets with different positive rates. The same model scores differently when the class balance changes.

Want the numbers to mean something? Start from clean data, and remember that a good score doesn't mean a feature causes the outcome.

In one line: accuracy asks "how often right", precision asks "can I trust a positive", recall asks "how much did I find", F1 balances them, ROC-AUC measures ranking, and the threshold is where you choose which mistake you can afford.