HDDS-14793. Fix race condition in XceiverClientGrpc#connectToDatanode causing intermittent NPEs.#9997
Draft
ptlrs wants to merge 6 commits intoapache:masterfrom
Draft
Conversation
…rpc#sendCommandAsync`.
…e` causing intermittent NPEs.
adoroszlai
reviewed
Mar 29, 2026
Contributor
adoroszlai
left a comment
adoroszlai
left a comment
There was a problem hiding this comment.
Thanks @ptlrs for working on this.
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java
Outdated
Show resolved
Hide resolved
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java
Outdated
Show resolved
Hide resolved
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java
Outdated
Show resolved
Hide resolved
Contributor
Author
|
Thanks for the review @adoroszlai, I have pushed the fixes. |
yandrey321
reviewed
Mar 30, 2026
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java
Outdated
Show resolved
Hide resolved
adoroszlai
reviewed
Apr 1, 2026
Contributor
adoroszlai
left a comment
adoroszlai
left a comment
There was a problem hiding this comment.
Thanks @ptlrs for updating the patch. compute logic looks good now.
Comment on lines
+239
to
+243
| if (details == null || !dnChannelInfoMap.containsKey(details.getID())) { | ||
| return false; | ||
| } | ||
|
|
||
| private boolean isConnected(ManagedChannel channel) { | ||
| return channel != null && !channel.isTerminated() && !channel.isShutdown(); | ||
| return !dnChannelInfoMap.get(details.getID()).isChannelInactive(); |
Contributor
There was a problem hiding this comment.
Avoid separate containsKey and get.
if (details == null) {
return false;
}
ChannelInfo info = dnChannelInfoMap.get(details.getID());
return info != null && !info.isChannelInactive();
Comment on lines
+266
to
+267
| .map(ChannelInfo::getChannel) | ||
| .collect(Collectors.toList()); |
Contributor
There was a problem hiding this comment.
ChannelInfo.channel may be null (or at least there are checks for that elsewhere).
Suggested change
| .map(ChannelInfo::getChannel) | |
| .collect(Collectors.toList()); | |
| .map(ChannelInfo::getChannel) | |
| .filter(Objects::nonNull) | |
| .collect(Collectors.toList()); |
Comment on lines
735
to
754
| throw new IOException("This channel is not connected."); | ||
| } | ||
|
|
||
| ManagedChannel channel = channels.get(dn.getID()); | ||
| // If the channel doesn't exist for this specific datanode or the channel | ||
| // is closed, just reconnect | ||
| if (!isConnected(channel)) { | ||
| // If the channel doesn't exist for this specific datanode or the channel is closed, just reconnect | ||
| if (!isConnected(dn)) { | ||
| reconnect(dn); | ||
| } | ||
|
|
||
| } | ||
|
|
||
| private void reconnect(DatanodeDetails dn) | ||
| throws IOException { | ||
| ManagedChannel channel; | ||
| try { | ||
| connectToDatanode(dn); | ||
| channel = channels.get(dn.getID()); | ||
| } catch (Exception e) { | ||
| throw new IOException("Error while connecting", e); | ||
| } | ||
|
|
||
| if (!isConnected(channel)) { | ||
| if (!isConnected(dn)) { | ||
| throw new IOException("This channel is not connected."); | ||
| } |
Contributor
There was a problem hiding this comment.
Given that connectToDatanode handles all cases ("closed", "already connected", "needs new connection"), checkOpen can be simplified:
connectToDatanode(dn);
if (!isConnected(dn)) {
throw new IOException("This channel is not connected.");
}and reconnect can be removed.
Comment on lines
-201
to
-203
| } catch (RuntimeException e) { | ||
| LOG.error("Failed to create channel to datanode {}", dn, e); | ||
| throw new IOException(e.getCause()); |
Contributor
There was a problem hiding this comment.
We need to keep the translation back from unchecked exception to IOException, because callers may not handle the former.
try {
dnChannelInfoMap.compute(...);
} catch (UncheckedIOException e) {
LOG.error("Failed to create channel to datanode {}", dn, e);
throw e.getCause();
}…pc to prevent runtime exceptions.
Contributor
Author
|
@adoroszlai thanks for the instructions on how to run the Flaky test CI . After running it on master and a test branch which has the latest changes of this PR we see the following results:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The
XceiverClientGrpc#connectToDatanodeintermittently fails with an NPE.The problem is that for a given datanode, there is a race condition between creating a channel and creating a stub.
When a new channel is created for a DN, it is put into the
channelsmap. However, presence of a channel in the map does not imply that the corresponding stub for the same DN also exists in theasyncStubsmap.If the stub is accessed after creating a channel but before the creation of stub, we can get an NPE.
This PR fixes the problem by:
dnChannelInfoMapfor both the channels and stubs instead of two independent mapsChannelInfoclass to group the channel and stubWhat is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-14793
How was this patch tested?
CI: https://github.com/ptlrs/ozone/actions/runs/23703558972
Flaky test runner: https://github.com/ptlrs/ozone/actions/runs/23823575690