Describe the term "hallucination" in language models. How can prompt engineering mitigate this issue?

A hallucination is output that sounds confident and plausible but is false, or isn't supported by the sources the model was given: an invented citation, a wrong figure, a nonexistent API function. Prompting reduces it by grounding answers in supplied sources, allowing the model to say it doesn't know, asking for quotes and citations, and checking answers, but it can't eliminate it, so applications need verification too.

Why models hallucinate A language model generates the most plausible continuation of text. It has no built-in check against reality, and its training rewards fluent answers. Hallucinations are most likely for rare facts, recent events after its training data, exact numbers, names, citations and questions that assume something false.

Prompting techniques that help

1. Ground answers in supplied context Retrieve relevant documents and instruct: "Answer only from the documents below. If they don't contain the answer, say so." This turns a memory question into a reading task.

2. Permit uncertainty Tell the model explicitly that "I don't know" or "the documents don't say" is an acceptable answer. Without that permission, models tend to guess.

3. Quotes first, then answer For long documents: "First extract the exact quotes relevant to the question, then answer using only those quotes." Unsupported claims become easy to spot.

4. Citations Ask for a source for each claim (document title or section), and have code check that each cited quote really appears in the source.

5. Ask it to verify A second step: "For each statement in this answer, find a supporting quote in the documents; remove statements without one."

6. Avoid leading questions "Why did the 2019 update remove feature X?" invites an invented story if it didn't. Ask whether it happened, and tell the model to correct false premises.

7. Narrow the task Specific questions with clear scope hallucinate less than open-ended requests.

8. Consistency checks Generate several answers; if they disagree on a fact, treat it as unreliable.

Controls beyond the prompt

• Tools for exact data: a database query or calculator rather than the model's memory.
• Structured outputs so fields can be validated.
• Human review for high-stakes uses (legal, medical, financial).
• Evaluations measuring faithfulness to sources, run before every change.
• Showing users the sources so they can check.

Hallucination rates have fallen as models improved, but no current model is free of them, so design the application as if some answers will be wrong.