Your new model deployment produces fluent answers that score badly on evaluations. Where do you look?

TL;DR: Fluent output that scores badly is almost always a formatting problem in the serving layer rather than a problem with the weights, because a model whose weights are wrong produces visibly broken text. Five candidates. The chat template, which decides how the system prompt, the turns and the roles are rendered into tokens, and which is model-specific and easy to get subtly wrong. The stop tokens, since some models list several end-of-sequence ids and configuring one leaves the others as ordinary tokens the model can emit and continue past. The tool-call and reasoning parsers, which are model-specific and, when missing, cause tools never to be called and internal reasoning to leak into the answer. The tokenizer, if a mismatched or re-derived one is in use. And quantization, if you requantized rather than serving the released format. Isolate by reproducing the model card's own example prompts byte for byte, because that removes your prompt, your template and your parameters from the equation in one step.

How to approach it

Reproduce the card's examples first, because it isolates the serving layer from your application in one test. Then work the five candidates in order of how often they are the cause. Compare rendered token ids rather than rendered strings, since whitespace and special tokens are exactly where this goes wrong. Close with the check that belongs in onboarding so this is caught before traffic sees it.

A strong answer

A typical situation: a model is deployed, smoke tests pass, and users report that it ignores instructions in the system prompt. The output is grammatical and on topic. An evaluation suite scores it well below the published numbers for the same model.

The isolation test:

reproduce the model card's own examples
  take the exact prompts the card publishes, with the card's own parameters
  run them through your deployment
  compare against the card's published outputs

what each result means
  matches            -> the serving layer is correct and the problem is in your application's
                        prompt construction or parameters
  differs            -> the serving layer is wrong, and the five candidates below apply
  differs wildly     -> tokenizer or quantization, since those break everything rather than
                        subtly

do it at the token level, not the string level
  render your request through the chat template and print the token ids
  compare against the ids the card's example produces
  whitespace, a missing special token, or an extra newline are invisible in a string diff and
    obvious in an id diff
sanity: this one test takes twenty minutes and it partitions the problem into "yours" or
        "ours", which is worth more than any amount of prompt experimentation

Model Onboarding: From Hugging Face to Production covers where this belongs in the sequence.

The five candidates:

the stop-token case, which is the most invisible
  GLM-5.3's published config lists three eos_token_id values
  configuring one of them leaves the other two as ordinary tokens
  the model emits one, expects generation to end, and continues
  the result is an answer followed by more text, or by template markers, or by a second
    attempt at the same answer

  the symptom
    outputs longer than expected
    template or role markers appearing in user-visible text
    an evaluation scoring badly because the answer is buried in trailing content

  the check
    read every id in the config's eos_token_id list and confirm the engine has all of them
sanity: this produces fluent, plausible output that a human skims past and an evaluation
        scores badly, which is exactly the description in the question

Reading config.json to Size a Model You Have Never Run covers the fields, and the stop-token list is one worth reading explicitly rather than trusting a default.

What the stop-token failure costs, which is why it is worth a gate rather than a code review:

a deployment that emits past its end-of-sequence token
  intended answer length, mean:                       220 tokens
  observed length with two of three stop ids missing: 640 tokens
  extra tokens per request = 640 - 220 = 420

  cost, at 40 requests per second
    extra tokens/s = 40 x 420 = 16,800
    on a fleet delivering 12,800 useful tokens/s at its operating point, that is
      16,800 / 12,800 = 1.31, so the deployment is doing 2.31 times the decode work for the
      same useful output
    put another way, 57 percent of generated tokens are waste

  and the quality effect
    the useful answer is followed by trailing content, which an evaluation scores and a user
    sees
sanity: a single missing token id costs more than half the fleet's decode capacity and shows
        up as a quality problem rather than a capacity one, which is why it survives

Separating plumbing from quantization:

plumbing failures
  affect structure: where the answer starts and stops, whether instructions are followed,
    whether tools are called
  are the same on every prompt, because the template is the same
  disappear entirely when the card's examples are reproduced correctly

quantization effects
  affect content quality rather than structure
  are uneven across tasks: long-context reasoning and code generation degrade before
    short-answer tasks
  scale with how aggressive the format is, and are absent if you serve the released format
  the check: run the same prompts against a bf16 or against the released-format deployment
    and compare scores per task category

so the order
  1. reproduce the card's examples          separates plumbing from everything else
  2. if plumbing is clean, compare formats  separates quantization from the model itself
  3. only then question the model            which is rarely the answer for a released model
sanity: a released model's published evaluations were produced with its released format and
        its own template, so a large gap against them points at your configuration rather
        than at the weights

The signals to watch once it is deployed, so a regression is caught rather than reported:

per-deployment metrics
  mean and p99 output token count per request
    a step change here is the stop-token failure announcing itself
  the fraction of responses containing template or role markers
    should be zero; anything above it is a template or stop-token problem
  tool-call rate, as a fraction of requests that should have called a tool
    a drop to zero after a deploy is the parser flag
  p99 TTFT and per-user tokens/s
    unchanged by these failures, which is why capacity dashboards do not catch them

and the gate before traffic
  the model card's example prompts, compared at the token-id level, run in CI on every
  configuration change
sanity: every signal here is about the shape of the output rather than the speed of it, and
        a serving dashboard built only from latency and throughput is blind to all of them

The reversal condition: if you requantized the model yourself rather than serving the released format, the ordering inverts and quantization becomes the first suspect rather than the last. The published evaluations do not apply to a format the authors did not produce, so there is no external baseline and the comparison has to be against your own bf16 or released-format run on your own task mix. That is a real project rather than a check, which is the argument for serving the released format unless there is a measured reason not to: it comes with an evaluation you did not have to run.

What interviewers probe next

• "Why compare token ids rather than strings?" Because whitespace and special tokens are where templates go wrong, and both are invisible in a string comparison.
• "What if the card publishes no examples?" Use the tokenizer's own chat template application as the reference and compare against the engine's rendering, which tests the same thing.
• "How would you catch this in onboarding?" A correctness gate that reproduces the card's examples and exercises a tool call, run before any traffic reaches the deployment.
• "Could it be sampling parameters?" Yes, and it is worth checking: a temperature or penalty set by a gateway default can change behaviour substantially without breaking anything.

Common mistakes

• Assuming fluent output means the serving layer is correct.
• Comparing rendered strings rather than token ids, which hides the failures that matter.
• Configuring one end-of-sequence token when the config lists several.
• Deploying without the model's tool and reasoning parsers, so tools are silently never called.
• Suspecting the weights first, when a released model's published evaluations were produced with its own template and format.

Key takeaways

• Fluent and wrong is a plumbing failure; broken weights produce visibly broken text.
• Reproduce the model card's own examples at the token-id level, which partitions the problem in twenty minutes.
• Five candidates: chat template, stop tokens, tool-call parser, reasoning parser, then tokenizer or quantization.
• Some models list several eos_token_id values, and configuring one leaves the rest as ordinary tokens.
• Plumbing failures affect structure and are uniform; quantization effects affect content and concentrate on long context and code.
• One missing stop token took mean output from 220 to 640 tokens, making 57 percent of generated tokens waste while every latency metric looked normal.