Cheatsheet

# ===== When to use which abstraction =====

# Just call an LLM with a prompt?         -> ChatPromptTemplate | llm
# Add memory across turns?                -> ConversationBufferMemory (small)
#                                            ConversationSummaryBuffer (large)
# Need structured output?                  -> llm.with_structured_output(PydanticModel)
# Combine retrieval with generation?       -> LCEL pipe with a retriever
# Need tool use / decisions?               -> langgraph + create_react_agent
# Need multi-step workflows with state?    -> langgraph StateGraph (preferred over agents)

# ===== Retrievers cheat sheet =====

# Few static docs (< 100):                Chroma in-memory
# 100 - 10M chunks:                       Chroma persistent, FAISS, Qdrant
# > 10M chunks or production:             Pinecone, Weaviate, Vespa, OpenSearch
# Multi-tenancy:                          per-tenant collection or namespace
# Mixed structured + text:                hybrid search (BM25 + vector) via rank_bm25 + vector
# Best chunk size:                        500-1000 tokens, 10-20% overlap
# Re-rank?                                cross-encoder (e.g. cohere-rerank) when top-k matters

# ===== Structured output strategies =====

# 1) llm.with_structured_output(Schema)   -> easiest, uses function calling
# 2) PydanticOutputParser                  -> simpler models, regex-based
# 3) Guardrails / Instructor               -> retries on schema failure
# 4) JSON mode                             -> when the model supports it natively

# ===== Memory patterns =====

# Short chat:       in-memory buffer
# Long chat:        summary buffer (LLM compresses old turns)
# Persisted chat:   write each turn to a DB, reload on session start
# Semantic memory:  vector-store of past Q/A; retrieve before answering

# ===== Cost + safety controls (use ALL of these in production) =====

# Rate limiting:    per-user concurrency + per-day token budget
# Caching:          set_llm_cache(InMemoryCache()) or RedisCache for shared
# Prompt-injection: heuristic pre-filter + structured output validation
# PII:              redact before send; never log full prompts
# Hallucinations:   require citations; reject answers without sources
# Token leaks:      never echo user-supplied text inside system instructions

# ===== Minimal RAG skeleton =====
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
from langchain_chroma import Chroma

llm = ChatOpenAI(model='gpt-4o-mini', temperature=0)
vs  = Chroma(persist_directory='./store', embedding_function=OpenAIEmbeddings())
retriever = vs.as_retriever(search_kwargs={'k': 4})

def format_docs(docs):
    return '\n\n'.join(f'[{i+1}] {d.page_content[:1000]}' for i, d in enumerate(docs))

prompt = ChatPromptTemplate.from_messages([
    ('system', 'Answer using ONLY the context. Cite [N] for each claim. If unsure, say so.'),
    ('human', 'Q: {question}\n\nContext:\n{context}'),
])

rag = (
    {'context': retriever | format_docs, 'question': RunnablePassthrough()}
    | prompt | llm | StrOutputParser()
)

# ===== Evaluation =====
# Always ship an eval dataset alongside the chain.
# Cheap evaluators (exact match, contains keywords) first — they cost nothing.
# LLM-as-judge sparingly for nuanced quality.
# Gate releases on regression vs the last commit.

# ===== Common pitfalls =====
# - Chunks too big: model ignores middle of the context window
# - Chunks too small: retrieval misses
# - Memory leaking across users: build per-request, never per-process
# - Temperature > 0 in production extraction: non-deterministic JSON
# - No timeout: a slow model call hangs your endpoint

# Prompt | LLM | OutputParser  •  Retriever | Prompt | LLM  •  Tools + Agent  •  Memory