Certificate

# ===== Award criteria =====
# Pass: >= 70% across these dimensions
#   1. Data ingestion         (10 pts)
#   2. Cleaning + validation  (15 pts)
#   3. Transformations        (20 pts)
#   4. Joins + reshaping      (15 pts)
#   5. Aggregations + groupby (15 pts)
#   6. Visualisation          (10 pts)
#   7. Communication          (15 pts)
# Distinction: >= 85%

# ===== Project brief =====
# Pick one real dataset >= 50k rows and deliver:
# - notebook.ipynb (reproducible from a clean kernel)
# - report.md (max 800 words, one-page summary)
# - figures/ (PNG exports of 3-5 charts)

# Example dataset choices:
# - NYC taxi trips (Jan 2024)
# - Open weather stations daily (2023)
# - GitHub archive events (sampled)
# - Australian Bureau of Stats: housing or labour

# ===== Submission outline =====
# 1. Question         What are you trying to answer?
# 2. Data             Source, schema, license
# 3. Ingestion        pd.read_csv with explicit dtypes + parse_dates
# 4. Cleaning         Missing handling, dtype coercions, dedup
# 5. Validation       Sanity checks + assertions
# 6. Transformations  Feature engineering (vectorised, no apply where avoidable)
# 7. Analysis         groupby + agg + pivot to answer the question
# 8. Visualisation    3-5 plots, each with a takeaway in the caption
# 9. Communication    One-page report; charts referenced by number
# 10. Reproducibility requirements.txt + 'Restart and Run All' works clean

# ===== Sample notebook scaffold =====
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv(
    'taxi_jan_2024.csv',
    parse_dates=['pickup_datetime', 'dropoff_datetime'],
    dtype={'passenger_count': 'Int8', 'fare_amount': 'float32'},
)
assert df['fare_amount'].between(0, 1000).mean() > 0.99, 'fare outliers above expected'

df = (
    df
    .assign(duration_min=lambda d: (d.dropoff_datetime - d.pickup_datetime).dt.total_seconds() / 60)
    .query('1 <= duration_min <= 120')
    .query('1 <= passenger_count <= 6')
)

hourly = (
    df
    .assign(hour=df.pickup_datetime.dt.hour)
    .groupby('hour')
    .agg(trips=('fare_amount', 'count'), median_fare=('fare_amount', 'median'))
)

fig, ax = plt.subplots(figsize=(8, 4))
hourly['trips'].plot(ax=ax, marker='o')
ax.set_title('Trips by hour of day, Jan 2024')
ax.set_ylabel('Trips'); ax.set_xlabel('Hour')
plt.tight_layout(); plt.savefig('figures/hourly.png', dpi=150)

# ===== Marking sheet (example) =====
# 1. Ingestion        10/10  dtypes pinned, dates parsed
# 2. Cleaning         13/15  outliers handled, missing strategy explained
# 3. Transformations  18/20  vectorised, no apply except for one helper
# 4. Joins/reshape    12/15  one pivot, one merge with validate=
# 5. Aggregations     15/15  groupby + named agg
# 6. Visualisation     9/10  three labelled, captioned charts
# 7. Communication    13/15  clear question, takeaways tied to charts
# Total: 90/100 -> Distinction

# ===== Don'ts =====
# - Long .apply(lambda...) on hot loops; vectorise or .map a dict
# - inplace=True everywhere; pandas is moving away from inplace mutations
# - No assertions or schema checks; broken silently if upstream changes
# - Plot without labels, titles, captions

# /certificate/numpy-pandas