Quiz

# ============================================================
# Q1) When should you use @torch.no_grad() vs @torch.inference_mode()?
# ============================================================
# ANSWER: @torch.inference_mode() is the modern default for prediction.
# - no_grad: disables gradient tracking
# - inference_mode: disables gradient tracking AND view tracking, slightly faster
# Use inference_mode anywhere the tensor will not need to be backprop'd through
# OR turned into a Tensor with requires_grad later.

# ============================================================
# Q2) Why is your training loss going down but validation loss going up?
# ============================================================
# ANSWER: overfitting. Common fixes:
# - more / better data
# - more regularisation (dropout, weight decay)
# - early stopping on val metric (not loss) with patience
# - simpler model
# - check for data leakage (val and train share rows / time periods)

# ============================================================
# Q3) When should you call .item() on a tensor?
# ============================================================
# ANSWER: ONLY for logging/reporting; NEVER inside the train loop's hot path.
# .item() synchronises CPU and GPU, killing throughput on GPU training.
# Accumulate losses on the device, then .item() once per epoch.

# ============================================================
# Q4) What is the right way to seed for reproducibility?
# ============================================================
# ANSWER: seed Python, NumPy, AND PyTorch — and set the CUDA deterministic mode.
import random, numpy as np, torch
def seed_everything(seed: int):
    random.seed(seed); np.random.seed(seed)
    torch.manual_seed(seed); torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

# Note: full determinism costs ~10-30% speed (cuDNN cannot pick the fastest kernel).
# Most teams seed for runs they want repeatable + accept some non-determinism elsewhere.

# ============================================================
# Q5) When is DataParallel wrong and DistributedDataParallel right?
# ============================================================
# ANSWER: always prefer DDP.
# DataParallel: single-process, multi-GPU — bottlenecked by the Python GIL and
#   the master GPU collecting gradients. Slow above ~2 GPUs.
# DDP: one process per GPU, NCCL all-reduce, scales close to linearly.
# torchrun --standalone --nproc-per-node=4 train.py

# ============================================================
# Q6) Your model uses 90% of GPU memory. What knobs do you have?
# ============================================================
# ANSWER, in order of cheapest first:
# 1) reduce batch size; use gradient_accumulation_steps to keep effective batch
# 2) torch.amp.autocast + GradScaler -> mixed precision, ~50% memory cut
# 3) torch.utils.checkpoint -> trade compute for memory on big models
# 4) bigger model variant only with model parallelism (FSDP, DeepSpeed)
# 5) gradient checkpointing on encoder/decoder transformer blocks

# ============================================================
# Bonus — what is the WORST default to leave on in production?
# ============================================================
# ANSWER: model.train() instead of model.eval().
# Dropout is still active, BatchNorm uses batch stats. Predictions become noisy
# and depend on batch composition. Always call model.eval() before inference.

# ============================================================
# Scoring
# ============================================================
# 6 / 6 -> can review training loops at speed
# 4 / 6 -> bookmark pytorch/cheatsheet
# < 4   -> read PyTorch's 'Performance Tuning Guide' before scaling

# 3 questions per lesson.