3 / 6
LLM evaluation

Test very long inputs and truncation

Medium70 pts~25 min
  • Token limits
  • Truncation
Practice app · Acme Support Assistant

A deterministic LLM-style support assistant with retrieval (RAG), JSON mode, safety policies and tool calls, exposed via UI and API.

BASE_URL
/api/practice
Console app
/lab/ai-testing-test-very-long-inputs-and-truncation

Your starter code already declares BASE_URL — call the API relative to it.

Objective

Send a long prompt and a tight max_tokens budget, and assert the response reports truncation honestly.

Your task

  1. 1Build a ~3000-character prompt: neutral filler (e.g. "lorem ipsum " repeated) followed by "How long does standard shipping take?".
  2. 2Send it with max_tokens: 5 → assert truncated === true and finish_reason === "length".
  3. 3Assert usage.completion_tokens ≤ 5 and usage.prompt_tokens is larger than for a short prompt.
  4. 4Send it without max_tokens → assert truncated === false and finish_reason === "stop".

Acceptance criteria

  • POST /ai/chat returns 200
  • max_tokens is used
  • At least 4 assertions pass

LLM evaluation · AI Testing · Prompt & input testing