Exercises

# ============================================================
# Drill 1 — Prompt design
# ============================================================
# Task: build a chain that classifies support tickets into 5 categories.
# Choose: zero-shot prompt vs few-shot vs fine-tune.
#
# ANSWER: few-shot with 2-3 examples per class works almost as well as fine-tuning
# for 5-10 classes, costs nothing to maintain, and adapts to new classes via prompt
# edits. Fine-tune only when latency matters or labelled examples > a few hundred.

# Concrete prompt
prompt = '''Classify each ticket into one of:
- billing
- account
- delivery
- product
- other

Examples:
Ticket: 'My credit card was charged twice.' -> billing
Ticket: 'My order is 3 days late.' -> delivery
Ticket: 'I forgot my password.' -> account

Ticket: '{text}' ->'''

# ============================================================
# Drill 2 — Retriever choice
# ============================================================
# Task: 10k Markdown docs with metadata (author, tag). Query: 'docs by alice
# about refunds in 2026'.
#
# ANSWER: hybrid search. Use vector store for semantic recall, filter by metadata
# at the store level (do NOT post-filter in Python), and rerank top-10.
# Pseudo:
# retriever = vs.as_retriever(search_kwargs={
#     'k': 20, 'filter': { 'author': 'alice', 'year': 2026 }
# })

# ============================================================
# Drill 3 — Structured output
# ============================================================
# Task: extract { invoice_no, total, currency, due_date } from PDFs.
#
# ANSWER: llm.with_structured_output(PydanticModel)
# Schema-enforced, with retries on validation failure. Reject malformed.
# Treat parse failures as 4xx-equivalents in the orchestrator.

# ============================================================
# Drill 4 — Evals
# ============================================================
# Task: prove the chain still works after a prompt rewrite.
#
# ANSWER: a golden dataset (50 inputs + expected outputs), cheap evaluators
# (exact match, contains keyword, JSON valid, citation present), and a CI gate
# that fails on regression > N%.
# Use LangSmith / pytest. Run on every PR that touches prompts.

# ============================================================
# Drill 5 — Safety
# ============================================================
# Task: a chain that calls a write tool ('cancel_order'). Defend against
# prompt injection.
#
# ANSWER, layered:
# - User input is untrusted; never interpolate verbatim into system instructions
# - Schema-validate tool outputs (e.g., id format)
# - Require explicit confirmation in the conversation BEFORE tool runs
# - Per-user rate limit + max-amount checks on the tool side
# - Log every tool call with the user id + the raw arguments
# - On suspected injection (filter heuristics), refuse with a logged event
# Treat the LLM as untrusted code. Defend the system around it.

# ============================================================
# Drill 6 — Cost
# ============================================================
# Task: a feature that costs $200/day in tokens. Halve it.
#
# ANSWER, in order of leverage:
# 1) cache identical requests (Redis-backed LLM cache)
# 2) route by intent: small model by default, escalate when needed
# 3) shorter prompts (system instructions in 1 paragraph, not 3)
# 4) prompt-cache prefixes on supported models (Claude prompt caching)
# 5) batch API on supported models (Anthropic / OpenAI batch -> 50% off)
# 6) re-evaluate retriever top-k; fewer docs = fewer tokens

# ============================================================
# Scoring
# ============================================================
# 6 / 6 -> ready to lead a LangChain feature
# 4 / 6 -> bookmark langchain/cheatsheet + safety lessons
# < 4   -> read the production-grade-langchain docs and revisit

# Fill in: chain = prompt | llm | ____()