Quiz

# ============================================================
# Q1) What is the bias-variance trade-off?
# ============================================================
# ANSWER: Total error = bias^2 + variance + irreducible noise.
# - High bias: model too simple, underfits, train and val both poor.
# - High variance: model too flexible, fits noise, train >> val.
# Lever-arm fixes:
#   underfit -> more features, deeper model, less regularisation
#   overfit  -> more data, more regularisation, simpler model, dropout, early stop

# ============================================================
# Q2) When do you use L1 vs L2 regularisation?
# ============================================================
# ANSWER:
#   L1 (Lasso): drives weights to exactly zero -> feature selection
#   L2 (Ridge): shrinks weights smoothly -> usually better numeric stability
#   Use ElasticNet (mix) when you want both effects and you do not know which.

# ============================================================
# Q3) Which metric for an imbalanced binary problem?
# ============================================================
# ANSWER: accuracy is useless when one class dominates.
# Prefer: PR-AUC, F1 of the minority class, recall @ a precision threshold.
# When false positives have a HUGE cost, report precision @ recall.
# When false negatives are catastrophic, optimise recall.

# ============================================================
# Q4) What is data leakage and how do you detect it?
# ============================================================
# ANSWER: information from the future or from the label sneaks into features.
# Classic shapes:
#   - target encoding fit on the full dataset, not per fold
#   - features computed AFTER the target (e.g. 'time spent on form' as a churn feature)
#   - splitting by row when data is actually per-user (use GroupKFold)
#   - duplicates that bridge train and test sets
# Detection: a baseline that 'cheats by accident' will look TOO GOOD. Inspect.

# ============================================================
# Q5) Why does my cross-validation score not match production?
# ============================================================
# ANSWER: usually one of:
#   - non-IID data (time, user, location) -> use the right split strategy
#   - feature drift between train and prod -> monitor input distributions
#   - selection bias in labels (only certain users were labelled)
#   - the CV pipeline leaks (see Q4)

# ============================================================
# Q6) What is class imbalance, and what should you do about it?
# ============================================================
# ANSWER: choose ONE of:
#   - class_weight='balanced' (in sklearn) — cheap, often enough
#   - oversample minority (SMOTE) or undersample majority
#   - cost-sensitive loss
#   - change the decision threshold using your CHOSEN metric, post-hoc
# Sampling INSIDE cross-validation, not before — or you leak.

# ============================================================
# Q7) When do you stop training a deep model?
# ============================================================
# ANSWER: early stopping on a validation set, with patience.
#   - track the metric you actually care about, not just loss
#   - save the best checkpoint, not the latest
#   - if loss keeps falling but val metric plateaus, you are overfitting

# ============================================================
# Q8) Why is my model good in offline eval but bad after deploy?
# ============================================================
# ANSWER: usually selection bias + feedback loop.
#   - offline data was sampled differently from production traffic
#   - the model's own decisions change the distribution of what it sees next
#   - features available offline may be unavailable / different at inference
# Mitigate with A/B tests, shadow deploys, and per-feature 'inference vs training' parity checks.

# Scoring
# 8 / 8 -> shipping ML
# 6 / 8 -> bookmark this lesson + Goodfellow ch. 5
# < 6   -> revisit before designing experiments that have customers attached

# 3 questions per lesson.