RATIS-2514. Fix flaky TestReadOnlyRequestWithGrpc.testReadAfterWrite.

[slfan1989]

What changes were proposed in this pull request?

This pull request fixes a flaky assertion in TestReadOnlyRequestWithGrpc.testReadAfterWrite.

The previous test sent multiple async writes without waiting for them to complete, then issued a linearizable read and a read-after-write request concurrently. It asserted that the read-after-write result should be greater than or equal to the concurrent linearizable read result.

That assumption is unreliable because the two reads may have different linearization points and may observe different subsets of concurrent writes.

This patch keeps the original async write workload and both read paths, but makes the test deterministic by:

• waiting for the async writes to complete before issuing the reads;
• verifying that both linearizable read and read-after-write observe the completed prior writes;
• waiting for followers to catch up before cluster shutdown to avoid shutdown-time leak detection races.

What is the link to the Apache JIRA

RATIS-2514. Fix flaky TestReadOnlyRequestWithGrpc.testReadAfterWrite.

How was this patch tested?

Tested with:

./mvnw -pl ratis-test -am -Pgrpc-tests -Dtest=TestReadOnlyRequestWithGrpc#testReadAfterWrite test

Result:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.ratis.grpc.TestReadOnlyRequestWithGrpc
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.746 s - in org.apache.ratis.grpc.TestReadOnlyRequestWithGrpc
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------

[szetszwo]

@slfan1989 , this problem probably has already be fixed in the master branch. Since the difference between two branches are huge. I suggest doing a merge/rebase first.

• RATIS-1931_grpc-zero-copy 238 commits ahead of and 350 commits behind master.

[szetszwo]

The sendReadAfterWrite feature is for two async calls, (1) writeAsync and then (2) readAfterWriteAsync such that (2) must be able see the change by (1). So, we cannot call get() here. Otherwise, the test becomes useless. For exampe,

0. initial: X = 1
1. writeAsync: set X = 3
2. readAfterWriteAsync: must see X = 3 (it is a bug if it can see X = 1)

This may be a bug already fixed (and moved to LinearizableReadTests) in the master branch.

[slfan1989]

@szetszwo Thanks for pointing this out! I agree that this test should preserve the async write-then-read-after-write ordering; calling get() between the two async calls would make the test ineffective for the intended read-after-write semantics.

Given that RATIS-1931_grpc-zero-copy is currently far behind master, and this area may already have been fixed or moved to LinearizableReadTests on master, I agree the better next step is to merge/rebase from master first. After syncing the branch, I will re-check whether the issue still exists before making any local test changes.