[integrations][java][python] Apply Ollama native structured output

[weiqingy]

Linked issue: #280

Purpose of change

Ollama has constrained decoding to a JSON schema since server v0.5.0, through the format request field. This connection refused it: Java inherited the base 4-arg chat, which throws UnsupportedOperationException for a non-null schema, and Python called _reject_unsupported_output_schema, which raises NotImplementedError. Callers were held on the prompt-engineering fallback, where the model is asked in prose for conforming JSON and the reply is scraped for code fences.

This adds the native path in both languages, following the OpenAI, Azure and Anthropic connections.

Runtime flow

Java: chat delegates to doChat, which calls the new package-private buildRequest inside its existing try block and then issues the call. buildRequest assembles the request as before, and if the schema is a Class and the capability predicate agrees, derives a JSON schema with victools and sets it as the request's format. The seam is new, so it is extracted in a separate commit with no behavior change.

Python: chat converts messages and tools, pops model, then consults the predicate and _native_format. A translated schema becomes a format= argument spliced into the SDK call. With none, no format key is passed.

Key decisions

The capability predicate reports capable for every model, in both languages. Ollama applies the grammar in the server's sampler rather than the model, and exposes no model-level signal to key on: /api/show reports a capability set carrying nothing schema-related, and /api/version reports only a version string. A server also runs arbitrary local models, so an allowlist of the kind the hosted providers use would be invented, and would report not-capable for models that work. Probing at open() was rejected because it detects only one of the three deployments that break the guarantee.

Java derives the schema itself with victools, the first connection here not to hand a class to a provider SDK. ObjectMapper.generateJsonSchema needs no dependency and already serves the prompt fallback, but it is deprecated and draft-03 flavored, which would put the two languages on different dialects.

victools moves from transitive runtime to direct compile scope. It already ships in the distribution through two provider SDKs, so no artifact is added and no notice changes. Its version comes from importing the victools BOM: a version on the module declaration would win nearest-wins mediation and become the global victools version while pinning only one of its three artifacts, letting the siblings drift on a later SDK upgrade with no build warning.

Implementation Description

Behavioral contracts

1. A POJO Class in Java, or an OutputSchema wrapping a BaseModel in Python, is sent as the request's native format.
2. Any other schema form, notably a RowTypeInfo, leaves format unset and keeps the prompt-engineering fallback.
3. Without a schema the request is what it was before this change. format is omitted from the body rather than sent as null.
4. The Java schema declares draft 2020-12, keeps properties in declaration order, gives a Map its value schema, excludes accessor-derived properties, and marks every field required except an Optional one.
5. The Python payload is model_json_schema() verbatim, $defs included.
6. The schema travels as a request field, never as a sampling option.
7. Capability is reported for every model name, including an absent or empty one.
8. A schema this connection cannot translate natively is answered rather than refused.

Failure behavior

A server below v0.5.0 rejects format with HTTP 400. Java rewraps it as RuntimeException through the existing catch, Python propagates the SDK error. This is the intended loud failure.

Ollama Cloud accepts the request without enforcing the schema, and the MLX runner accepts the field and discards it. Both return an unconstrained response and no error, and neither is distinguishable from a model name, so neither is detected. The escape hatch for all three is structured_output_strategy: PROMPT, which is not yet dispatched.

A recursive Java class generates a self reference the server rejects with HTTP 400. The victools option that rewrites it is deliberately unset: it makes the server drop the grammar silently for any class carrying a nested type twice, trading a loud failure on a rare shape for a silent one on a common shape.

A BaseModel pydantic cannot render raises instead of falling back, matching the merged OpenAI and Azure connections.

An explicit NATIVE strategy is not visible at this layer, so a caller requesting it for a schema form this branch skips receives an unconstrained response rather than an error. A TODO(#912) marks the site in both languages.

Tests

Contract	Tests
1	buildRequestSetsFormatForPojoSchema, test_native_applied_for_base_model
2	buildRequestLeavesFormatUnsetForRowTypeInfo, test_native_not_applied_for_row_type_info
3	buildRequestOmitsFormatWithoutSchema, test_format_absent_without_schema
4	generatedSchemaShapeIsConstraining
5	test_schema_is_model_json_schema
6	test_schema_not_passed_as_sampling_option. No Java test: setFormat is a typed request field, so there is no equivalent way to misroute it
7	supportsNativeStructuredOutputIsServerNotModelGated, test_supports_native_structured_output, both parameterized over several names plus null and empty
8	test_schema_accepted_not_rejected. Java is covered by contract 2's test, which returns a request rather than throwing

The Java tests are this connection's first, asserting the built request through buildRequest with no network and no mocking framework. The Python tests assert the SDK call kwargs against a mocked client. Both run in the unit arm. The Java RowTypeInfo test substitutes a non-Class object exercising the same gate, because Flink is a provided dependency of the api module and so is off this module's test classpath. The Python test uses a real RowTypeInfo.

API

supportsNativeStructuredOutput and the 4-arg chat are existing extension points, overridden here. buildRequest is package-private, and toNativeFormat and _native_format are private.

For a caller that does nothing differently, nothing changes. No framework path passes a schema to a connection today, so the native branch is reachable only from a direct call, and a request without a schema is byte-identical to before. For a caller that does pass one, a previously raised error becomes a constrained response for a POJO or BaseModel, and a silent prompt fallback for any other form.

victools changes scope from runtime to compile in dependent modules. Resolved versions are unchanged and no packaged bytes change.

Documentation

• doc-needed
• doc-not-needed
• doc-included

Was this patch authored or co-authored using generative AI tooling?

• Yes
• No

Generated-by: Claude Code 2.1.226

[weiqingy]

Hi @wenjin272 , could you take a look at this PR when you get a chance? A fourth sample for the Implementation Description experiment on #894, native structured output for Ollama.

Two things worth reporting from writing it, both about size.

It came out around 7k characters against the 6k target, and #965 was 7.5k. Two in a row over, for the same reason: this one has eight behavioral contracts and six distinct failure paths, and the format asks for every one of them. I could only reach 6k by dropping a contract or a failure path, which seemed worse than being a screen longer. Is the target something that should flex with the size of the change, or does two consecutive overruns say the field list is too much for a provider PR?

The description also caught something the review had missed: one contract has no Java test. The contracts-to-tests table made it visible because an empty cell sits next to seven filled ones. Worth noting #965's description surfaced nothing new about its code, so this is not consistent, and that difference might itself be the useful signal about when the format pays off.

If it is proving useful on your side, would it make sense to start on the "Author-side Implementation Descriptions + two-stage review for AI-assisted PRs" guide? Happy to keep producing samples first if you would rather see more before committing to a format.