Workflow Name textboxes take time to load

[kousikmahadeva]

In a heavier usage, /metdata/workflow takes minutes to fetch the data.
This is reproducable in latest version.

[nthmost-orkes]

Will try to reproduce now!

[nthmost-orkes]

Reproduction confirmed

Reproduced on a clean Redis-backed Conductor instance (conductor-oss build, conductor:server, built from this repo's main branch).

Test setup

• 2,000 distinct workflow definitions × 5 versions each = 10,000 total defs
• ~18KB per definition (30 tasks with input parameters)
• Redis co-located with server (localhost Docker network — best-case latency)

Results

Serial requests (3 runs):

Metric	Value
Response time (mean)	~2.6s
Response payload	~258MB

Concurrent load (10 concurrent clients, 30 total requests):

Metric	Value
First batch (reqs 0–9)	134–140s (1 HTTP 500)
Batches 2–3 (reqs 10–29)	All timed out at 300s
Effective 500/timeout rate	~67% (20/30 never returned)
min	134.4s
max	300.25s (timeout)
mean	246.3s

Server is essentially saturated after the first wave of concurrent requests — subsequent requests never complete within a 5-minute timeout.

---

Root cause

RedisMetadataDAO.getAllWorkflowDefs() performs N+1 serial Redis round-trips:

1. One SMEMBERS call to fetch all workflow names
2. One HGETALL per workflow name to fetch all versions

With 2,000 workflow names that's 2,001 Redis calls per request, all serialized. Each call then deserializes and returns the full definition body for every version of every workflow.

Three compounding factors:

• N+1 Redis round-trips — latency scales linearly with number of distinct workflow names
• No pagination — full scan on every request regardless of what the caller needs
• Full definition bodies in response — the UI only needs names/versions for the dropdown, but gets complete task graphs

Note: GET /api/metadata/workflow/names-and-versions already exists and returns only names/versions — worth checking whether the UI can be pointed at that endpoint as a near-term mitigation.

Longer-term fixes would involve pipelining/batching the Redis calls or adding a metadata cache.

[kousikmahadeva]

Thank you so much for looking into this.

you are right on API change.
GET /api/metadata/workflow/names-and-versions cuts down file size and time massively.

If it helps,
I use the below patch of mine and it does seem to help cut numbers by 20x...
#845

[nthmost-orkes]

@kousikmahadeva ah thanks! I didn't see you'd posted this, had already started working on a fix. lemme compare these -- if yours is good we can pull yours in, maybe with a few changes. cheers.