fix(cli): handle sync/async serve functions in m serve

[markstur]

Misc PR

Type of PR

• Bug Fix
• New Feature
• Documentation
• Other

Description

• Link to Issue: Part of: Create OpenAI API-compatible HTTP interface for mellea #521

fix(cli): handle sync/async serve functions in m serve

Fixes sync/async mismatch in `m serve` by detecting function type and
handling appropriately:
- Async serve functions are awaited directly
- Sync serve functions are wrapped in asyncio.to_thread() to prevent
  blocking FastAPI's event loop

This ensures the server can handle concurrent requests efficiently
regardless of whether user-defined serve functions are sync or async.

Changes:
- cli/serve/app.py: Add asyncio/inspect imports, update make_chat_endpoint()
  to detect coroutine functions and wrap sync functions in to_thread()
- test/cli/test_serve_sync_async.py: Add comprehensive test suite (9 tests)
  including empirical timing test that proves non-blocking behavior

feat: Map OpenAI parameters to ModelOption sentinels:

   - temperature → ModelOption.TEMPERATURE
   - max_tokens → ModelOption.MAX_NEW_TOKENS
   - seed → ModelOption.SEED


Signed-off-by: Mark Sturdevant <mark.sturdevant@ibm.com>

Testing

• Tests added to the respective file if code was changed
• New code has 100% coverage if code as added
• Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

[github-actions]

The PR description has been updated. Please fill out the template for your PR to be reviewed.

[planetf1]

see comments

[markstur]

Thanks @planetf1 for the review! I fixed each. Please see the comment/question about whether we really want an allowlist. I interpreted that as not the main objective there (but I have a feeling I read it with bias). Happy to address it if you think we should pursue that right now.

[ajbozarth]

On a first read code looks good, but Claude did find a couple items to look at:

Issue 1: None values are mapped to sentinel keys

_build_model_options uses model_dump() which includes all fields, including those set to None. Fields like seed: None and max_tokens: None (unset by the caller) become @@@seed@@@: None and @@@max_new_tokens@@@: None in the output.

Backends that do if ModelOption.SEED in model_options: will see them. Depending on backend implementation, passing None at a sentinel key could cause unexpected behavior or override a backend default.

Suggestion: Filter out None values:

filtered_options = {
    key: value
    for key, value in request.model_dump().items()
    if key not in excluded_fields and value is not None
}

---

Issue 2: functions/tools/tool_choice etc. pass through unfiltered

ChatCompletionRequest has several OpenAI-specific fields (functions, tools, function_call, tool_choice, response_format, logit_bias, etc.) that aren't in excluded_fields. They'll land in model_options as None for most requests (or with real values for tool-using callers).

Pre-existing behavior, yes — but _build_model_options is a good opportunity to clean this up. At minimum these should probably be added to excluded_fields, since Mellea backends don't consume raw OpenAI tool schemas via model_options.

---

Minor: nested helper could be module-level

_build_model_options has no closure over anything in make_chat_endpoint. Moving it to module level would make it easier to test directly (there are currently no direct unit tests for it).

[markstur]

Suggestion: Filter out None values:
Added exclude_none=True parameter to model_dump

Issue 2: functions/tools/tool_choice etc. pass through unfiltered

• I added special handling for n > 1 because I noticed @planetf1 called that out in the issue.
• I also added a handler to turn that and pydantic validation errors into OpenAI API errors.
• But with I looked at doing similar handling for all the other params is seemed like a lot so I just added them to excluded_fields like you suggested.

It turns out, me adding this bit of model options mapping in the sync/async PR was bad creep. Who knew?
So excluded is probably a good start and I will start implementing and/or handling them better in next PR(s).

Minor: nested helper could be module-level

_build_model_options has no closure over anything in make_chat_endpoint. Moving it to module level would make it easier to test directly (there are currently no direct unit tests for it).
Done and unit tests added.

Thanks!

[ajbozarth]

One more nit from Claude otherwise LGTM