Summary

# ===== Use LCEL by default =====
# LCEL = pipeline of typed runnables. Easier to reason about, debug, evaluate.
# Reach for agents (langgraph) only when the model must DECIDE which tool to call.

# ===== Retrieval =====
# Few docs (< 100):       in-memory Chroma
# 1k - 1M chunks:         Chroma persistent, FAISS, Qdrant local
# > 1M or production:     Pinecone, Weaviate, Vespa, OpenSearch
# Hybrid (text + struct): BM25 + vector with reranker
# Chunk size:             500-1000 tokens, 10-20% overlap
# Rerank:                 cross-encoder (cohere-rerank) when top-k matters

# ===== Models =====
# Default to small/cheap (gpt-4o-mini, Haiku). Route to bigger only when needed.
# Router: rule-based or a tiny classifier deciding 'big model?' per request.
# Cache identical requests with InMemoryCache or RedisCache.

# ===== Structured output =====
# llm.with_structured_output(PydanticModel) — easiest, type-safe, retries
# Reject malformed; treat parse failure as 4xx
# Avoid free-form parsing in production

# ===== Memory =====
# Stateless API:        per-request only
# Chat:                 per-session via SessionMemory hooked to DB
# Semantic memory:      vector store of past Q&A retrieved before answering

# ===== Safety =====
# - Input filter (regex + length cap) for obvious prompt injection
# - PII redaction before sending to the model
# - System prompt that hardens against ignore-instruction attacks
# - Output schema validation
# - Tool scope: least privilege; require confirmation on write tools
# - Per-user rate limit + concurrency + token budget

# ===== Cost =====
# 1. Cache identical requests
# 2. Route to cheaper model when possible
# 3. Shorter prompts (system instructions tight; no boilerplate)
# 4. Prompt caching on supported models
# 5. Batch API for offline workloads (50% discount typical)
# 6. Reduce retriever top-k where quality holds

# ===== Evaluation =====
# Golden dataset (50-200 inputs) with expected outputs or rubric
# Cheap evaluators first: exact match, contains, citation present, JSON valid
# LLM-as-judge sparingly: cost + variance
# CI regression gate: fail PR if pass rate drops > N%

# ===== Minimal RAG skeleton =====
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.prompts import ChatPromptTemplate
from langchain_chroma import Chroma
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

llm = ChatOpenAI(model='gpt-4o-mini', temperature=0)
vs  = Chroma(persist_directory='./store', embedding_function=OpenAIEmbeddings())
retriever = vs.as_retriever(search_kwargs={'k': 4})

def format_docs(docs):
    return '\n\n'.join(f'[{i+1}] {d.page_content[:1000]}'
                          for i, d in enumerate(docs))

prompt = ChatPromptTemplate.from_messages([
    ('system', 'Answer using ONLY the context. Cite [N] for each claim. If unsure, say so.'),
    ('human', 'Q: {question}\n\nContext:\n{context}'),
])

chain = (
    { 'context': retriever | format_docs, 'question': RunnablePassthrough() }
    | prompt
    | llm
    | StrOutputParser()
)

# ===== Decision matrix =====
# Cheap extraction?            structured_output + small model
# RAG over docs?               retriever + prompt + small model (+ rerank if top-k matters)
# Tool use / decisions?        langgraph agent + bounded tools
# Conversational over data?    session memory + RAG + per-user state
# Offline analysis at scale?   batch API + cache

# ===== Pitfalls =====
# - Skipping evals -> prompt edits silently regress quality
# - Mixing tenants in one global memory -> data leak
# - Trusting LLM output as code/SQL/URL -> sandbox + validate
# - Temperature > 0 in extraction -> non-deterministic JSON
# - No cost ceiling -> a viral moment becomes a bill

# Next: LangGraph deep dive, evaluation harnesses, agent tools, multi-modal LLMs.