iwantcoding.com
🔥 Daily 👥 Rooms 🏆 Top Log in Sign up

Bias / Variance

By · Reviewed & last updated · Part of Machine Learning Tutorial

Every model balances bias (underfits because it’s too rigid) against variance (overfits because it’s too flexible). The diagnosis informs the fix — more data / regularisation / capacity / features.

Diagnose + remediate

EXAMPLE
# 1) The classic diagnostic — train vs val scores
# HIGH bias  (underfit):    train low, val low,  small gap
# HIGH var.  (overfit):     train high, val low, big gap
# Good fit:                 train high, val high, small gap

from sklearn.datasets        import make_classification
from sklearn.model_selection import train_test_split, learning_curve
from sklearn.linear_model    import LogisticRegression
from sklearn.ensemble        import RandomForestClassifier
import numpy as np

X, y = make_classification(n_samples=400, n_features=20, n_informative=5, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

for name, model in [
    ('underfit (linear, C=0.001)',   LogisticRegression(C=0.001, max_iter=200)),
    ('balanced (linear, C=1)',       LogisticRegression(C=1,     max_iter=200)),
    ('overfit (deep forest)',        RandomForestClassifier(max_depth=None, min_samples_split=2)),
]:
    model.fit(Xtr, ytr)
    print(f'{name:<35}  train {model.score(Xtr, ytr):.3f}  val {model.score(Xte, yte):.3f}')

# 2) Learning curve — does more data help?
sizes, train_s, val_s = learning_curve(
    LogisticRegression(C=1, max_iter=200), Xtr, ytr,
    cv=5, train_sizes=np.linspace(0.1, 1.0, 6),
)
# If train & val curves CONVERGE and PLATEAU low → high bias (more data won't help)
# If they DIVERGE → high variance (more data WILL help)

# 3) Fix HIGH BIAS
#    • Use a more capable model (more parameters, more layers, deeper trees)
#    • Add features / interactions
#    • Reduce regularisation (bigger C, smaller weight_decay)
#    • Train longer (more epochs / iterations)

# 4) Fix HIGH VARIANCE
#    • Add data (the highest-leverage cure)
#    • Reduce capacity (simpler model, fewer features)
#    • More regularisation (smaller C, dropout, L2)
#    • Early stopping (stop training when val plateaus)
#    • Data augmentation

Why it matters

Don’t guess. Plot a learning curve. The shape tells you in 30 seconds whether your bottleneck is data, capacity, or regularisation — saving days of blind hyperparameter tuning.

Tip: Tweak the snippet with Try it Yourself », then sit the quiz at the bottom of the page.

Example

Example
# High bias: model can't fit train (underfit).
# High variance: model fits train but not test (overfit).
# Fix bias: bigger model, more features.
# Fix variance: more data, regularisation, simpler model.
Try it Yourself »

Discussion

Loading…