ItInsightsWorld SoftwareDevelopment Guide: Fixing Tool-Call Schema Accuracy in vLLM for Kimi and DeepSeek

Agent systems used by development teams often fail not because the model cannot reason, but because the serving stack cannot emit a valid tool call. This ItInsightsWorld SoftwareDevelopment guide focuses on a concrete engineering case: Kimi K2.6 served by vLLM 0.28 produced valid schemas only 92% of the time, meaning 2 attempts in every 25 could fail. It also explains how vLLM 0.31 exposes constrained decoding as a server-side option that can bring schema accuracy to 100% for Kimi and DeepSeek when the workload is configured correctly.

How tool-call schemas are passed to an LLM

Tool calling has two practical stages. First, the application tells the model which external functions exist. Second, the application supplies the shape the model must output for structured calls. OpenCode is a client example: when it connects to Kimi, it sends a JSON array describing available tools. One listed tool is webfetch, which retrieves Web page content.

The schema for that tool can declare parameters, their types, and required fields. It may reference https://json-schema.org/draft/2020-12/schema, allow text, markdown, or html as output formats, choose markdown by default, and describe an optional timeout in seconds with a maximum of 120. Before the LLM sees the JSON description, vLLM or another inference engine wraps it in tokens expected by the model. For Kimi, the declaration can be presented as system tool-declaration tokens around the array.

At run time, a user request may cause the model to emit tokens such as functions.webfetch:0 with a JSON argument object. The inference server parser extracts that call and converts it into an assistant JSON object containing tool_calls. The harness then runs webfetch and returns the result to the model.

Why schema accuracy becomes a reliability problem

A tool call is successful only when the emitted JSON matches the schema. The simplest serving method lets the model generate calls freely and trusts prior training to produce correct syntax. That works most of the time, but the residual errors matter in multi-turn sessions because each malformed call leaves a failed request, an error message, and a retry in the context window. The next model turn then consumes space and cost on noise rather than progress.

To measure this class of failure, Moonshot AI published the K2VV ToolCall benchmark. It helps providers verify that Kimi is served with quality comparable to the original service, including valid tool-call generation. The benchmark workload contains 2000 requests designed to test when the LLM generates tool calls and whether those calls are syntactically correct. It reports trigger similarity and schema accuracy. Trigger similarity measures whether the service decides to make a call and how closely that decision matches Moonshot AI's own service. Schema accuracy measures the share of calls that follow the schema and can be invoked successfully. If tool-call accuracy is 50%, half of the generated calls are malformed.

The root cause is probabilistic sampling. A language model selects the next token by likelihood, not by running a compiler or type checker. Malformed calls can still appear, especially when the model learned from many tool datasets whose schemas and conventions do not match the current inference harness. A production system may therefore see a model emit search_web instead of web_search, or city_name instead of location, because those names existed in training data but not in the live tool set.

Constrained decoding turns schema validity into a runtime rule

In standard generation, an LLM computes probabilities for possible next tokens and samples by weighted choice. Constrained decoding changes that flow: the engine blocks tokens that would make the tool call syntactically invalid and allows only tokens that preserve validity at each step. Instead of sampling first and validating later, generation is guided by the schema. vLLM implemented this capability as a server-side flag in vLLM 0.31 for deployments that need tool-call strictness.

The allowed token set comes from the supplied tool schema. vLLM uses structural tags from XGrammar to define a valid path for a call: begin tool call, tool name, tool call index, begin arguments, schema-constrained JSON, end tool call. During the tool-call index phase, for example, the engine can restrict sampling to integer tokens and mask other candidates. Invalid branches are blocked during generation rather than repaired after output.

Image source: Dong, Yixin, et al. XGrammar: Flexible and efficient structured generation engine for large language models. Proceedings of Machine Learning and Systems 7 (2025).

This behavior moves schema accuracy from best-effort expectation to a guaranteed runtime contract. The model still selects a token, but the candidate list has already been filtered to syntactically valid choices. The overhead can remain low because token masks for each phase are precomputed and applied efficiently on the GPU during generation.

Step-by-step tutorial: validating and enabling strict tool calls

1. Inventory the served stack. Record model versions such as Kimi K2.6 and DeepSeek V4 Flash, plus the inference engine version, such as vLLM 0.28 or vLLM 0.31.
2. Capture a baseline with a tool-call evaluation similar to the K2VV ToolCall benchmark. In the cited case, Kimi K2.6 on vLLM 0.28 reached 92% schema accuracy, which translates into 2 failures in every 25 attempts.
3. Check the client harness and schema source. For example, OpenCode may send webfetch parameters, output format, and a 120 second timeout cap to Kimi.
4. Compare free generation with schema-guided generation. For Kimi and DeepSeek, enabling constrained decoding in vLLM 0.31 can reach 100% schema accuracy in the tested benchmark path.
5. Re-run the benchmark after any upgrade or flag change, including moves related to vLLM v0.25, before production deployment. A score of 91% after a default change was the signal in the documented case.
6. Roll out only after confirming valid tool names, argument objects, parsing behavior, and acceptable latency under the chosen constrained-decoding mode.

Comparison table: free generation versus constrained decoding

Engineering dimension	Free generation	Constrained decoding
Schema handling	The model relies on training patterns and may still produce invalid syntax.	The engine masks invalid tokens and follows grammar rules from XGrammar.
Documented Kimi case	Kimi K2.6 with vLLM 0.28 showed 92% schema accuracy; 2 of 25 calls could fail.	vLLM 0.31 exposes the feature as a server-side flag and can reach 100% for Kimi and DeepSeek.
Operational default	Safer for model-agnostic serving when grammar or interaction bugs exist.	Stronger for client-quality expectations, but teams must test per model.
Typical symptom when misused	Context pollution, retry cost, and malformed assistant tool_calls.	Requests may diverge when the grammar does not match the model format.
Model examples	gpt-oss-20b, Qwen 3.6, and GLM 5.1 reported problems when strictness became default.	Kimi K2.6 and DeepSeek V4 Flash were the focus of the verified improvement.

Where constrained-decoding behavior diverged

Moonshot AI recommends turning constrained decoding on, and its own API behaves that way. The operational record from Ubicloud shows a more complicated path: constrained decoding was first added in a private vLLM fork. After vLLM introduced the feature in v0.25, the upstream implementation was used for Kimi and DeepSeek. Soon after, the tool-call accuracy score dropped back to 91%.

Analysis found that vLLM's constrained-decoding implementation remained correct. The default behavior for requests that did not explicitly enable constrained decoding changed. Supporting constrained decoding is a per-model task, and issues fall into three camps:

• General model issues: unconstrained output can diverge from a model's tool schema or calling format, forcing token choices that derail generation. gpt-oss-20b, Qwen 3.6, and GLM 5.1 users reported problems after the feature was enabled by default.
• Grammar translation bugs: translating a model's tool schema or calling format into that model's grammar may contain defects. This can produce invalid calls, request failures, silently ignored constrained decoding, or lower accuracy.
• Feature interaction bugs: constrained decoding may interact incorrectly with other serving-engine features. A speculative decoding interaction can make grammar rules wrongly reject speculative tokens and fail requests.

Because these risks affect a broad model range, vLLM maintainers treated disabled-by-default as the safer choice for vLLM as a model-agnostic serving engine. The later --tool-strict-level flag gives operators a way to enable tool-call strictness intentionally. In the Kimi and DeepSeek serving story, Ubicloud enabled constrained decoding, re-ran the K2VV ToolCall benchmark, verified 100% tool-call schema accuracy across served versions, and deployed the change to production.

Common pitfalls for agent serving teams

• Treating benchmark quality as if it automatically equals served quality.
• Assuming 92% schema accuracy is acceptable when it still means 2 failures in every 25 attempts.
• Upgrading from vLLM 0.28 to vLLM 0.31, or adopting a feature introduced in vLLM v0.25, without re-running a tool-call benchmark.
• Leaving constrained decoding enabled or disabled without checking whether the model's grammar and client schema match.
• Ignoring speculative decoding interactions, which can turn a valid grammar into a failed request path.
• Failing to inspect the context window after tool-call failures, which hides why later turns become noisier.

FAQ

Is 92% schema accuracy enough for production?

It can be misleading. The cited Kimi K2.6 result on vLLM 0.28 means 2 failures in 25 attempts, and each failure may create retry cost and context pollution in multi-turn sessions.

Why does vLLM not always enable constrained decoding by default?

vLLM serves many models, and problems can fall into three camps: general model divergence, grammar translation bugs, and feature interaction bugs. That is why vLLM maintainers treated a stricter default as risky for a model-agnostic engine.

Does constrained decoding fix every tool-calling mistake?

No. It addresses syntactic validity by allowing only schema-valid tokens during generation. The model can still choose a wrong tool or wrong arguments from the valid set.

What should a team measure before enabling strictness?

Run a benchmark comparable to the K2VV ToolCall benchmark, check trigger similarity and schema accuracy, and verify results for each served model version such as Kimi K2.6 and DeepSeek V4 Flash.

Action recommendations

• Make tool-call schema accuracy a release metric, not a post-incident clue.
• If you serve Kimi K2.6 or DeepSeek V4 Flash with vLLM, test with a 2000-request benchmark such as the K2VV ToolCall benchmark or a comparable workload.
• Document the schema contract for each harness: OpenCode-style tool arrays, the https://json-schema.org/draft/2020-12/schema reference, allowed output formats, and a 120 second timeout cap when present.
• Use constrained decoding when first-party client quality matters, but treat it as model-dependent; test after every vLLM version change, including vLLM v0.25 and vLLM 0.31.
• Expect 2 malformed calls in 25 attempts if accuracy is 92%; design retries, logs, and context handling accordingly.
• Review the three issue categories before production rollout: general model divergence, grammar translation bugs, and feature interaction bugs involving speculative decoding.

From an ItInsightsWorld delivery perspective, the important shift is operational: schema correctness is a property of the whole serving path, not only the model weight. Kimi and DeepSeek agent quality can change when vLLM behavior or defaults change. A documented 92% result for Kimi K2.6 became a 100% result in the constrained-decoding benchmark path, giving teams a concrete method for reducing retry noise and context pollution while keeping the workflow predictable.