Concurrent databricks auth token --force-refresh invocations race on the shared OAuth token cache, causing hard authentication failures

[hopper-signifyd]

Describe the issue

Hello! We have a monorepo that does Databricks things. Oftentimes we run processes in parallel that need to interact with Databricks. After updating to newer versions of the DBX SDK and CLI, we started getting all sorts of weird auth / token related failures. We're hacking around these failures by creating our own shim and pointing the DBX SDK at it. But this is annoying and we'd really rather not do that.

I imagine we are not the only ones facing this issue. Others have probably run into it and just decided to roll back and wait for a fix.

Related: #4767 #4564

I had Claude write up a summary of what we're facing. Let me know if you need more information from me!

AI stuff:

Concurrent databricks auth token --force-refresh invocations for the same profile (same OS user, running as separate processes) race on the shared OAuth token cache and can fail with a hard error instead of returning a token — e.g.:

Error: forced token refresh: cache update: exit status 45

Root cause: databricks-sdk-go's credentials/u2m/persistent_auth.go already documents this as a known gap in PersistentAuth.refresh():

// TODO: This read-refresh-write sequence is not coordinated across processes.
// Because the CLI is stateless, two separate CLI invocations can load the same
// cached refresh token, both attempt a refresh, and race to update the cache.
// This should be fixed in a follow-up by adding cross-process coordination
// around refresh and cache writes.

Two concurrent processes both Lookup() the same cached refresh token, both exchange it with the IdP, and both attempt to Store() the result. On the default secure (macOS Keychain) storage mode, Store() shells out to /usr/bin/security -i via zalando/go-keyring's Set() (keyring_darwin.go); two concurrent security invocations against the same service/account can collide, and the loser's cmd.Wait() returns a raw exec.ExitError ("exit status 45" in our case) with no retry. The plaintext file store has an in-process sync.Mutex and atomic rename-on-write, but neither protects against a second process doing its own concurrent read-refresh-write cycle — there is no cross-process lock anywhere in this path (not in PersistentAuth, not in the file store, not in the keyring backend).

This got noticeably worse once the SDKs started appending --force-refresh (sdk-go#1628, sdk-py#1378, following cli#4767 / cli#4564): before that, Token()'s proactive-refresh path silently fell back to the still-valid cached token on a refresh failure, masking this race most of the time. ForceRefreshToken() intentionally has no such fallback ("the caller explicitly asked for a fresh token, so silently falling back to a stale one would be incorrect" — its own doc comment), so the race now surfaces as a hard, user-visible failure under any concurrent workload.

Steps to reproduce the behavior

--force-refresh forces a refresh unconditionally on every call (it doesn't check whether the cached token is still valid), so the race reproduces without needing to wait for a token near expiry:

1. databricks auth login --profile myprofile
2. Fire several concurrent auth token --force-refresh calls for the same profile from the same OS user:

   for i in $(seq 1 8); do
     databricks auth token --profile myprofile --force-refresh >/tmp/out_$i.json 2>/tmp/err_$i.txt &
   done
   wait
   grep -l "cache update\|exit status" /tmp/err_*.txt
   

3. One or more of the concurrent invocations fails.

Expected Behavior

All concurrent auth token --force-refresh calls for the same profile succeed (or, if the CLI chooses to serialize/coordinate them, block briefly and then succeed), since they're all just asking for a valid current token for the same authenticated session.

Actual Behavior

Some subset of the concurrent calls fail outright with an error such as forced token refresh: cache update: exit status 45, with no retry or fallback, breaking any automation that shells out to the CLI from multiple processes at once.

OS and CLI version

v1.7.0, macOS 26.5.1 arm64

Is this a regression?

Not exactly a regression in the traditional sense — the underlying race has always existed in PersistentAuth.refresh() (per its own upstream TODO), but it was previously masked: Token()'s proactive-refresh-with-buffer falls back to the existing valid token on a failed refresh, so concurrent processes rarely produced a user-visible error. Since the SDKs began appending --force-refresh (databricks-sdk-py >= 0.120 with CLI >= v0.296.0), refresh calls route through ForceRefreshToken(), which has no such fallback, so the same underlying race now fails hard instead of silently.

Debug Logs

Things like this: "Parent revision: failed after 212.31s during five concurrent Unity Catalog syncs with forced token refresh: cache update: exit status 45"