Skip to the page
Tecyfy

LLM apps and RAG interview question · medium

Users report that some answers stop in the middle of a sentence. How do you find the cause and fix it?

The short answerLog why each response ended, then handle each reason.

The full answer

Log why each response ended, then handle each reason. On the Claude API every response carries a stop_reason, and a cut-off answer almost always has one of a few causes: it hit the max_tokens limit, it filled the context window, the stream broke partway, or your own code cut it (a timeout, a length limit in the UI, a parser). The fix is never one setting; it is making the application check how each answer ended before showing it.

1. Find which cut it

Log stop_reason, output tokens, the request's max_tokens and whether the stream completed, for every call. Then look at the truncated answers:

What the log showsCause
stop_reason: max_tokens, output tokens equal to the limitThe answer needed more room than you allowed
stop_reason: model_context_window_exceededInput plus output filled the context window
No final event, or an error event after text arrivedThe stream failed partway
stop_reason: end_turn but the text is cut in your UIYour code: a timeout, a character limit, a markdown renderer dropping text

When streaming, the stop reason arrives near the end in the message_delta event, so a client that stops reading early never sees it. That alone is worth checking.

2. Fix by cause

  • max_tokens. Raise it for the features that need long answers, or continue the response: send the partial text back as the assistant turn with a short user message asking to continue, and join the parts. Also ask whether the answer should be that long; a tighter prompt is often the better fix.
  • Context window full. The input is too big: long chat history, too many retrieved chunks, large tool results. Trim or summarise history and cap what goes into each request.
  • Broken streams. An error can arrive mid-stream after an HTTP 200, so your normal error handling never runs. Handle error events in the stream, retry or show a clear failure, and never treat partial text as a finished answer. For long non-interactive work (over 10 minutes), use streaming or the Batch API, since idle connections can be dropped by networks.
  • Your own code. Check gateway and load balancer timeouts against your longest responses, and any length limits in storage or the UI.

3. Never show a silent cut

Whatever the cause, the reader should not get half an answer that looks complete. If a response ends for any reason other than a natural finish, either continue it, retry it, or show it with a clear note that it is incomplete. The same check also catches refusal, which returns HTTP 200 with no error, and pause_turn in server-tool loops.

4. Watch it

Track the share of responses per stop reason, per feature. A rise in max_tokens stops after a prompt change tells you the new prompt makes answers longer, before users notice.

What interviewers want: checking stop_reason on every response, the difference between the token limit, the context window, a broken stream and your own code, handling mid-stream errors, and never presenting a cut answer as complete.

Last updated 16 October 2026

Sources

Where each fact comes from

  1. 1Handling stop reasons
  2. 2Errors