Correlation and Covariance

Topic - Bivariate statistics evaluate the joint mathematical relationships between two random variables. While covariance detects whether variables move in the same or opposite directions, correlation normalizes covariance into a standardized, unitless metric bounded between $[-1, 1]$.

---

1. Covariance

Covariance quantifies the joint variability of two continuous random variables $X$ and $Y$.

Mathematical Formulations

$\text{Cov}(X, Y) = \sigma_{XY} = \frac{1}{N} \sum_{i=1}^N (x_i - \mu_X)(y_i - \mu_Y) \quad \text{(Population)}$

$s_{XY} = \frac{1}{n - 1} \sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) \quad \text{(Sample)}$

Directional Interpretation

• $\text{Cov}(X, Y) > 0$: Positive Association. As $X$ increases, $Y$ tends to increase.
• $\text{Cov}(X, Y) < 0$: Negative Association. As $X$ increases, $Y$ tends to decrease.
• $\text{Cov}(X, Y) = 0$: Zero Linear Association. Variables show no linear trend.

The Scale Dependency Flaw

Covariance retains the product of the original measurement units (e.g., kilograms $\times$ meters). Consequently, the magnitude of covariance cannot indicate the strength of the relationship; scaling one variable by 1,000 inflates the covariance by 1,000 without altering the underlying relationship.

---

2. Pearson Correlation Coefficient ($r$)

The Pearson product-moment correlation standardizes covariance by dividing it by the product of both variables' standard deviations, yielding a scale-free metric:

$r = \frac{\text{Cov}(X, Y)}{s_X s_Y} = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^n (x_i - \bar{x})^2} \sqrt{\sum_{i=1}^n (y_i - \bar{y})^2}}$

Boundary Properties (Cauchy-Schwarz Inequality)

By the Cauchy-Schwarz inequality, the Pearson correlation coefficient is strictly bounded: $-1 \le r \le 1$.

Positive Correlation

• $r = +1.0$: Perfect positive collinearity. All points lie on an upward-sloping line.
• $0.7 \le r < 1.0$: Strong positive linear association.

Negative Correlation

• $r = -1.0$: Perfect negative collinearity. All points lie on a downward-sloping line.
• $-1.0 < r \le -0.7$: Strong negative linear association.

Zero Correlation

• $r = 0.0$: Absence of any linear association.
• Crucial Warning: $r = 0$ does not imply statistical independence. Deterministic non-linear relationships (e.g., $Y = X^2$ over $[-1, 1]$) yield $r = 0$ despite complete functional dependence.

---

3. Spearman's Rank Correlation ($\rho$)

When data violates normality assumptions, contains extreme outliers, or is purely ordinal, Spearman's rank correlation coefficient evaluates the monotonic relationship by operating on the ranks of the observations:

$r_s = 1 - \frac{6 \sum d_i^2}{n(n^2 - 1)}$

Where $d_i = \text{rank}(x_i) - \text{rank}(y_i)$ represents the difference between the ranks of paired observations.

---

4. Implementation Lab

Compute and visualize bivariate covariance and correlation matrices using NumPy, Pandas, and Seaborn.

import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

# Generate synthetic bivariate data
np.random.seed(42)
x = np.random.normal(50, 10, 500)
noise = np.random.normal(0, 5, 500)
y = 2.0 * x + noise

df = pd.DataFrame({'Feature_X': x, 'Feature_Y': y})

# Covariance Matrix
cov_matrix = df.cov()
print("--- Covariance Matrix ---")
print(cov_matrix)

# Pearson Correlation Matrix
corr_matrix = df.corr(method='pearson')
print("\n--- Pearson Correlation Matrix ---")
print(corr_matrix)

# Spearman Rank Correlation
spearman_corr = df['Feature_X'].corr(df['Feature_Y'], method='spearman')
print(f"\nSpearman Rank Correlation: {spearman_corr:.4f}")

info

Key Takeaways

• Correlation does not imply causation: A high correlation indicates an observed mathematical association, but gives no proof of a causal link. Confounding variables must be ruled out through controlled experiments.
• Scale Invariance: Covariance depends on feature units; Correlation is completely scale-invariant and bounded by $[-1, 1]$.
• Linearity Constraint: Pearson correlation measures strictly linear relationships. Use scatterplots or Spearman correlation for non-linear monotonic data.

Next Topic: Probability Foundations - moving from descriptive metrics into formal probability spaces.