F-Scores and Balanced Metrics

Topic — Because Precision and Recall exist in a continuous mechanical tradeoff, comparing competing models requires a unified scalar benchmark. This guide explores the $F_1$-Score, derives why the harmonic mean is mathematically mandatory over the arithmetic mean, explores domain weighting with $F_\beta$ ($F_{0.5}$ and $F_2$), and demonstrates why Balanced Accuracy and the Matthews Correlation Coefficient (MCC) are superior on imbalanced data.

---

1. The Core Concept

When comparing models across validation folds, evaluating two separate numbers (e.g. Model A: $P=90\%, R=50\%$ vs. Model B: $P=70\%, R=75\%$) creates ambiguity.

The $F_1$-Score resolves this by computing the harmonic mean of Precision and Recall:

$F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$

Why the Harmonic Mean?

The arithmetic mean $\frac{P + R}{2}$ treats errors additively and permits compensatory masking: a model with $100\%$ precision that catches only $1$ out of $1,000$ actual positives ($R = 0.1\%$) receives an arithmetic mean of $\approx 50\%$.

The harmonic mean is defined as the reciprocal of the arithmetic mean of reciprocals:

$\frac{1}{F_1} = \frac{1}{2} \left( \frac{1}{\text{Precision}} + \frac{1}{\text{Recall}} \right)$

Because reciprocals explode as values approach zero ($\lim_{x \to 0} \frac{1}{x} = \infty$), the harmonic mean is dominated by the minimum value. If either Precision or Recall collapses toward zero, $F_1$ crashes to zero.

tip

If you only take one thing from this page

The arithmetic mean rewards models that sacrifice one metric to inflate the other. The Harmonic Mean ($F_1$) penalizes imbalance, strictly demanding that a model perform well across both Precision and Recall to achieve a high score.

---

2. Interactive Harmonic Mean & F-Beta Calculator

Adjust Precision and Recall below to observe how the Harmonic Mean ($F_1$) penalizes divergence compared to the Arithmetic Mean, and test custom weighting with $\beta$:

Interactive Harmonic Mean and F-Beta Explorer

Move the Precision and Recall sliders to observe how the harmonic penalty reacts to extreme divergence:

Precision (P):85%

Recall (R):25%

F-Beta Weighting (β):

Arithmetic Mean

55.0%

(P + R) / 2

Harmonic Mean (F1)

38.6%

2·P·R / (P + R)

Weighted F-Beta

38.6%

β = 1.0 priority

Harmonic Penalty

-16.4%

Penalty for imbalance

Metric Imbalance Trap: The Arithmetic Mean (55%) deceptively flatters the model, but the Harmonic Mean F1 (39%) correctly penalizes the lagging metric by 16 percentage points!

---

3. Weighting Asymmetric Priorities with F-Beta

When operational costs are asymmetric, the parameter $\beta$ in the generalized $F_\beta$-Score allows engineers to tilt evaluation weight toward Precision or Recall:

$F_\beta = (1 + \beta^2) \cdot \frac{\text{Precision} \cdot \text{Recall}}{\beta^2 \cdot \text{Precision} + \text{Recall}}$

1. F1-Score (β = 1.0)

• Weighting: Precision and Recall are weighted equally.
• Formula: $F_1 = 2 \cdot \frac{P \cdot R}{P + R}$.
• Optimal For: General benchmark ranking when false positive and false negative costs are comparable.

2. F2-Score (β = 2.0)

• Weighting: Weights Recall twice as heavily as Precision ($\beta^2 = 4$).
• Formula: $F_2 = 5 \cdot \frac{P \cdot R}{4P + R}$.
• Optimal For:
  • Medical cancer detection (missing disease is unacceptable).
  • Wire fraud monitoring (catching theft outweighs transaction reviews).

3. F0.5-Score (β = 0.5)

• Weighting: Weights Precision twice as heavily as Recall ($\beta^2 = 0.25$).
• Formula: $F_{0.5} = 1.25 \cdot \frac{P \cdot R}{0.25P + R}$.
• Optimal For:
  • Automated spam email deletion (destroying legitimate email is intolerable).
  • High-confidence loan approvals.

---

4. Balanced Metrics for Severe Imbalance: Balanced Accuracy & MCC

While $F_1$ is far superior to overall accuracy, it has a blind spot: $F_1$ ignores True Negatives completely.

If a model accurately classifies 10,000 legitimate negative instances, the $F_1$ formula never inspects the $\text{TN}$ quadrant. For extreme class imbalance, two alternative metrics provide complete matrix coverage:

1. Balanced Accuracy

Computes the unweighted arithmetic mean of Recall (TPR) and Specificity (TNR):

$\text{Balanced Accuracy} = \frac{\text{Recall} + \text{Specificity}}{2} = \frac{1}{2} \left( \frac{\text{TP}}{\text{TP} + \text{FN}} + \frac{\text{TN}}{\text{TN} + \text{FP}} \right)$

• On balanced data, Balanced Accuracy equals standard Accuracy.
• A trivial majority classifier always scores exactly $50.0\%$, instantly revealing lack of skill.

2. Matthews Correlation Coefficient (MCC)

The Matthews Correlation Coefficient treats binary classification as a Pearson product-moment correlation between actual and predicted binary vectors:

$\text{MCC} = \frac{\text{TP} \cdot \text{TN} - \text{FP} \cdot \text{FN}}{\sqrt{(\text{TP} + \text{FP})(\text{TP} + \text{FN})(\text{TN} + \text{FP})(\text{TN} + \text{FN})}}$

• Range: $[-1.0, +1.0]$.
  • $+1.0$: Perfect classification.
  • $0.0$: Equivalent to random guessing or predicting a constant majority class.
  • $-1.0$: Total inverse disagreement.
• Completeness: MCC is the only metric that uses all four quadrants symmetrically in both the numerator and denominator.

---

5. Implementation Lab

tip

Lab Exercise: F-Scores, Harmonic Means & Balanced Metrics

An interactive, fully functional Google Colab / Jupyter Notebook is available to evaluate Harmonic vs. Arithmetic divergence, tune $F_\beta$ weights, and benchmark MCC against $F_1$ live in your browser.

How to run the lab:

1. Click the "Open In Colab" badge above to launch the interactive notebook.
2. Run cells sequentially to simulate extreme metric imbalance and compute MCC benchmarks.

Key Experiments to Run:

• Experiment 1 (The Harmonic Penalty): Verify how $P=0.90, R=0.10$ drops from an Arithmetic Mean of $50.0\%$ down to an $F_1$-Score of $18.0\%$.
• Experiment 2 (F-Beta Sensitivity): Train a disease detector ($P=62\%, R=87\%$) and observe $F_2 = 80.4\%$ reward high recall while $F_{0.5} = 76.0\%$ penalizes false alarms.
• Experiment 3 (MCC on 95:5 Imbalance): Compare a trivial dummy model (Accuracy $95\%$, MCC $0.000$) against a trained model (Accuracy $95\%$, MCC $0.592$).

---

6. Common Mistakes

1. Using the Arithmetic Mean for F1

• The Misconception: "The F1-score is simply the average of precision and recall: (P + R) / 2."
• Wrong Thinking: Computing the arithmetic mean instead of the harmonic mean.
• The Right Principle: $F_1 = 2 \frac{P \cdot R}{P + R}$. The arithmetic mean masks catastrophic failure when one metric collapses to near zero.

2. Inverting Beta Weights in F-Beta

• The Misconception: "F2 prioritizes precision because 2 is greater than 1."
• Wrong Thinking: Assuming $\beta > 1$ weights precision.
• The Right Principle: $\beta$ represents how many times more important Recall is compared to Precision. $F_2$ ($\beta=2$) weights Recall twice as much as Precision. $F_{0.5}$ ($\beta=0.5$) weights Precision twice as much as Recall.

3. Assuming F1 Cares About True Negatives

• The Misconception: "A high F1-score proves the model accurately handles both positive and negative classes."
• Wrong Thinking: Forgetting that TN is absent from the F1 formula.
• The Right Principle: $F_1$ only inspects $TP, FP, FN$. It provides zero verification of negative class accuracy. To evaluate performance across both classes symmetrically on imbalanced datasets, report Matthews Correlation Coefficient (MCC) or Balanced Accuracy.

4. Comparing F1 Across Differing Class Prevalences

• The Misconception: "An F1 score of 0.70 means the same thing on a 50:50 dataset as on a 99:1 dataset."
• Wrong Thinking: Treating $F_1$ as an absolute scale-invariant metric.
• The Right Principle: On a 99:1 dataset, baseline precision is $1\%$, making an $F_1$ of $0.70$ an extraordinary achievement. On a balanced dataset, random guessing yields $F_1 \approx 0.50$. Always anchor $F_1$ against class distribution.

5. Zero Denominator Crashes in MCC

• The Misconception: "MCC can always be calculated for any classifier output."
• Wrong Thinking: Failing to handle degenerate models that predict only one class.
• The Right Principle: If a model predicts 'Negative' for every sample ($TP = FP = 0$), the MCC denominator $\sqrt{(TP+FP)\dots}$ evaluates to zero. Scikit-learn handles this edge case by returning $0.0$.

---

7. Summary

Metric Formulations

• F1-Score: Harmonic mean $2 \frac{P \cdot R}{P + R}$. Strictly penalizes extreme divergence between $P$ and $R$.
• F-Beta Score: Weighted harmonic mean. $\beta > 1$ weights Recall; $\beta < 1$ weights Precision.
• Balanced Accuracy: $\frac{\text{TPR} + \text{TNR}}{2}$. Trivial baseline is always $50\%$.
• MCC: Pearson correlation between actual and predicted labels $[-1, +1]$. Uses all 4 quadrants.

Engineering Selection

• Balanced Data: $F_1$ is the reliable industry standard.
• Life-or-Death Recall Focus: Use $F_2$ (medicine, disaster warnings, intrusion detection).
• Costly False Alarms: Use $F_{0.5}$ (automated spam deletion, auto-banning users).
• Severe Class Imbalance: Deploy MCC alongside Precision-Recall curves.

info

Key Takeaways

• Harmonic mean enforces balance: A model cannot achieve high $F_1$ by sacrificing Recall to artificially inflate Precision.
• Choose your β intentionally: Use $F_2$ when missed cases are catastrophic; use $F_{0.5}$ when false alarms destroy customer trust.
• MCC is the ultimate test: Because it accounts for True Negatives, MCC is the most rigorous single scalar metric for imbalanced classification.

Next Section: ROC-AUC and Threshold Tuning - sweeping all decision thresholds simultaneously and quantifying discrimination power via the Area Under the Curve.

---

8. Active Recall

Test your understanding by answering first, then clicking to reveal the underlying principles.

Interactive Checkpoint Quiz

Interactive Checkpoint: Metric Selection for Medical Screening

A medical research team wants a single metric to benchmark diagnostic algorithms for a rare deadly disease. False negatives must be penalized twice as heavily as false positives. Which metric should they select?

Review Flashcards

1. [THEORY] Why is the harmonic mean of two non-zero numbers always less than or equal to their arithmetic mean?

• The AM-GM-HM Inequality: For any positive real numbers $a$ and $b$: $\text{Arithmetic Mean } \ge \text{ Geometric Mean } \ge \text{ Harmonic Mean}$ $\frac{a + b}{2} \ge \sqrt{ab} \ge \frac{2ab}{a + b}$
• Equality holds if and only if $a = b$ (Precision equals Recall).
• The greater the divergence between Precision and Recall, the larger the penalty gap between Arithmetic and Harmonic mean.

2. [MATH] A model achieves Precision = 0.80 and Recall = 0.60. Calculate F1 by hand.

• Product: $P \cdot R = 0.80 \times 0.60 = 0.48$.
• Sum: $P + R = 0.80 + 0.60 = 1.40$.
• F1 Calculation: $F_1 = 2 \cdot \frac{0.48}{1.40} = \frac{0.96}{1.40} = \frac{96}{140} = \frac{24}{35} \approx 0.6857 \quad (68.6\%)$
• Notice $F_1 \approx 68.6\%$ is lower than the Arithmetic Mean ($70.0\%$).

3. [ANALYZE] Why can Matthews Correlation Coefficient (MCC) identify a useless model on imbalanced data when F1 cannot?

• Missing True Negatives: $F_1$ only inspects $TP, FP, FN$.
• If a dataset has 9,900 negatives and 100 positives, a naive model that predicts positive for 500 samples (including 80 actual positives and 420 false alarms) gets:
  • $P = 80 / 500 = 16\%$, $R = 80 / 100 = 80\% \implies F_1 \approx 26.7\%$.
• MCC evaluates all four cells ($TP=80, FP=420, FN=20, TN=9480$):
  • Because the model generated 420 false alarms on the negative class, MCC reflects this error in its denominator and yields a near-zero correlation ($\text{MCC} \approx 0.35$).

4. [PRACTICE] How does Balanced Accuracy prevent the Accuracy Paradox on an imbalanced dataset (99% negative, 1% positive)?

• A trivial model that always predicts Negative gets:
  • $\text{Recall} = 0 / 1 = 0.0$ ($0\%$).
  • $\text{Specificity} = 99 / 99 = 1.0$ ($100\%$).
• Balanced Accuracy: $\text{Balanced Accuracy} = \frac{0.0 + 1.0}{2} = 0.50 \quad (50.0\%)$
• While standard accuracy reports an inflated $99.0\%$, Balanced Accuracy reveals that the model performs no better than a random coin toss ($50\%$).

5. [THEORY] Can F1 ever exceed both Precision and Recall?

• No, never: Because $F_1$ is a mean (specifically a harmonic mean), it is mathematically constrained to lie strictly between the two values: $\min(\text{Precision}, \text{Recall}) \le F_1 \le \max(\text{Precision}, \text{Recall})$
• $F_1$ equals Precision and Recall if and only if Precision equals Recall ($P = R \implies F_1 = P$).