LLM Apps and RAG

Not this time. RAG fetches relevant passages from your own documents and puts them in the prompt, so the model can answer from facts it was never trained on or that change often, and cite them.

Not this time. Embeddings capture meaning, not exact strings. Keyword search (BM25) matches the exact code. Hybrid search runs both and merges them, for example with reciprocal rank fusion.

Not this time. RRF ignores raw scores and adds up 1/(k + rank) over each result list, with k = 60 by default in Elasticsearch. Documents ranked high in several lists win.

Not this time. A reranker scores each retrieved candidate together with the question (a cross-encoder or rerank API) and reorders them, so the best few go into the prompt. It is too slow to run over the whole index, so it runs on the top results.

Not this time. pgvector's documentation says HNSW has a better speed-recall trade-off but builds more slowly and uses more memory, and it needs no training step, so it can be created on an empty table. IVFFlat builds faster with less memory but should be built after the data is loaded.

Not this time. If the needed passage never reached the prompt, no prompt change can fix it. The fault is in retrieval: parsing, chunking, search method, query wording or filters.

Not this time. They found accuracy was highest when the relevant information was at the start or end of the context and dropped when it was in the middle, even for long-context models. Put the best passages first.

Not this time. It guarantees the response parses and matches the schema's shape. It does not make the values correct, and a refusal or hitting max_tokens can still produce non-matching output, so code still validates values.

Not this time. Streaming sends tokens as they are generated, so users see the answer begin almost at once. Time to first token is what they feel.

Not this time. Work that does not need an immediate answer suits a batch API. Anthropic's Message Batches API costs 50% less, and most batches finish within an hour.

Not this time. The cache only reuses an identical prefix. Anything that changes every request belongs after the cached part, near the end of the request, or the cache never hits.

Not this time. Rewrite the follow-up into a standalone query using the conversation, so the search contains the missing subject. Search with the raw follow-up finds the wrong passages.

Not this time. MTEB tested many models on 8 kinds of task and many datasets and languages, and found that no single method was best everywhere. That is why you test candidates on your own data.

Not this time. A 429 means a rate limit was exceeded. Wait at least as long as retry-after says, then retry, with backoff and a cap on attempts. A 429 caused by a spend limit has no retry-after and will not succeed on retry.

Not this time. Run model-written SQL with a read-only database role limited to approved tables. Prompts can be bypassed; database permissions cannot.