How do you build retrieval over PDFs that contain tables, charts and scanned pages?

Most bad RAG over PDFs is bad parsing. Use a parser that keeps layout, headings and tables, run OCR on scanned pages, turn tables into structured text (or keep them whole), and describe charts and images in text with a vision-capable model so they can be searched. Keep a link from every chunk back to its page, so answers can show the source and, when needed, the page itself can be sent to the model.

Why plain text extraction fails

• Two-column pages come out with lines interleaved.
• Tables become streams of numbers with no column names.
• Scanned pages produce no text at all.
• Charts and diagrams, often the most important content, disappear.

A sturdier pipeline

1. Classify pages: digital text, scanned, or mixed.
2. Parse with layout: a layout-aware parser, or a document AI service, that returns headings, paragraphs, lists and tables as separate elements.
3. OCR scanned pages, and check its confidence; flag low-quality pages for review.
4. Tables: convert to Markdown or HTML with headers kept. For large tables, one chunk per row group with the column headers repeated. For numeric questions, consider loading tables into a database and answering with a query.
5. Charts and images: ask a vision model to describe each one (title, axes, trend, key values) and index the description, linked to the page.
6. Chunk by structure, with the document title and section heading in each chunk.

Sending pages to the model directly

For a small number of documents, a model that reads PDFs can take the file itself. Anthropic's PDF support processes text and each page as an image, so it can read charts and tables, within limits: the documentation lists 600 pages per request (100 when the context window is under 1 million tokens), a request size limit, and warns that dense pages can fill the context window first. A good pattern is to retrieve the right pages with search, then send those pages as PDF or images for the final answer.

Evaluate on the hard parts

Build questions whose answers sit in tables, charts and scanned pages, and measure retrieval and answer accuracy for each kind separately. Overall scores hide the fact that every chart question fails.

Operational notes

• Keep the original files and page numbers; show the page in the answer's source link.
• Re-parse when you change parsers, and compare retrieval before switching.
• Watch costs: vision descriptions and page images use many tokens, so do them once at indexing time.

Summary: "Parse for structure, describe what's visual, keep page links, and test tables and charts on their own."