Classification Evaluation: ROC and AUC calcuation
Published:
ROC AUC is a key evaluation metric for binary classification models. It measures how well a model distinguishes between positive and negative classes.
ROC (Receiver Operating Characteristic) Curve:
- Plots True Positive Rate (TPR) vs. False Positive Rate (FPR).
- Shows the trade-off between sensitivity (recall) and specificity.
- x_axis: FPR = (FP/negative)
- y_axis: TPR = (TP/positive)
AUC (Area Under the Curve):
- Represents the overall performance of the classifier.
- AUC = 1: Perfect classifier.
AUC = 0.5: Random classifier.
2. Why Sort Predictions in Descending Order?
Key Idea: Rank-based Evaluation
- Sorting ensures we check the most confident predictions first.
- Better discrimination between positive and negative classes.
- Preserves order when computing TPR and FPR.
Step-by-Step Calculation:
1. Sort Samples by Predicted Probability
- Higher probabilities first.
- Keeps thresholds in decreasing order.
2. Traverse the Sorted List
- If ground truth = 1 → Increase True Positives (TP).
- If ground truth = 0 → Increase False Positives (FP).
- Compute TPR and FPR at each step.
3. Compute AUC Using Trapezoidal Rule
- Integrates area under ROC curve.
- Approximates the integral using discrete TPR-FPR points.
3. Learn with Code: Implementing ROC AUC from Scratch
import matplotlib.pyplot as plt
import numpy as np
def calculate_auc(y_true, y_pred_probs):
"""
Calculate the AUC (Area Under the Curve) and plot the Receiver Operating Characteristic (ROC) curve manually.
Args:
y_true (list or np.array): True binary labels (0s and 1s).
y_pred_probs (list or np.array): Predicted probabilities for the positive class.
Returns:
float: The calculated AUC value.
"""
# 1. Sort by predicted probabilities in descending order
sorted_indices = np.argsort(-np.array(y_pred_probs))
y_true = np.array(y_true)[sorted_indices]
y_pred_probs = np.array(y_pred_probs)[sorted_indices]
# 2. Initialize True Positive and False Positive counts
tp, fp = 0, 0
tpr, fpr = [], []
# 3. Total positives and negatives in the data ground truth
positives = sum(y_true)
negatives = len(y_true) - positives
# 4. Calculate TPR and FPR at each threshold
for i in range(len(y_true)):
if y_true[i] == 1:
tp += 1 # True Positive
else:
fp += 1 # False Positive
tpr.append(tp / positives) # True Positive Rate
fpr.append(fp / negatives) # False Positive Rate
# 5. Calculate AUC using the trapezoidal rule
auc_value = 0.0
for i in range(1, len(tpr)):
auc_value += (fpr[i] - fpr[i - 1]) * (tpr[i] + tpr[i - 1]) / 2
plot_roc(fpr, tpr, auc_value)
return auc_value
def plot_roc(fpr, tpr, auc_value):
"""
Plot the ROC curve.
"""
plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, color='blue', label=f"AUC = {auc_value:.2f}")
plt.plot([0, 1], [0, 1], color='gray', linestyle='--', label="Random Classifier")
plt.title("ROC Curve")
plt.xlabel("False Positive Rate (FPR = FP/negative)")
plt.ylabel("True Positive Rate (TPR = TP/positive)")
plt.legend(loc="lower right")
plt.grid(alpha=0.5)
plt.show()
# Example Usage:
if __name__ == "__main__":
# True labels (0 or 1)
y_true = [0, 0, 1, 1, 0, 1, 0, 1, 1, 0]
# Predicted probabilities for the positive class
y_pred_probs = [0.1, 0.4, 0.35, 0.8, 0.2, 0.85, 0.05, 0.9, 0.7, 0.3]
# Calculate and plot the AUC curve
auc_score = calculate_auc(y_true, y_pred_probs)
print(f"AUC Score: {auc_score:.2f}")
4. Key Takeaways
- ROC AUC measures a model’s ability to rank predictions correctly.
- Sorting predictions ensures we evaluate high-confidence samples first.
The trapezoidal rule approximates the AUC as an integral.
- More to read: [Google post] (https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc)
🤖 Disclaimer: This post is inspired by Educative.io AI learning course, and generated with AI-assisted but reviewed and refined by Dr. Rebecca Li, blending AI efficiency with human expertise for a balanced perspective.