fix(grpc): cancel active jobs on delete#1604
Open
tmckayus wants to merge 1 commit into
Open
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (8)
🚧 Files skipped from review as they are similar to previous changes (6)
📝 WalkthroughWalkthroughDeleteResult now cancels queued or running jobs before removing tracker, pending request, and log state. The gRPC service uses centralized deletion logic and returns helper-provided status messages. Integration tests cover queued-job deletion, running-job cancellation, and worker recovery. ChangesDeleteResult lifecycle
Estimated code review effort: 3 (Moderate) | ~20 minutes Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cpp/src/grpc/server/grpc_job_management.cpp`:
- Around line 374-376: Update the cleanup flow around delete_log_file to inspect
its unlink result, treat a missing log file as an acceptable already-clean
state, and handle other deletion errors as failure. Only set the “Result
deleted” message and return success when the log is absent or successfully
removed; otherwise report failure without claiming complete cleanup.
In `@cpp/src/grpc/server/grpc_service_impl.cpp`:
- Around line 530-544: The DeleteResult flow must invalidate all active
chunked-download sessions for the deleted job, including sessions created
concurrently with deletion. Update StartChunkedDownload and delete_job to
associate each session with job_id, erase matching entries while holding
chunked_downloads_mutex during deletion, and coordinate the
deletion/session-creation ordering so no session can be created after deletion
completes.
In `@cpp/tests/linear_programming/grpc/grpc_integration_test.cpp`:
- Around line 1154-1161: The queued-job deletion test at
cpp/tests/linear_programming/grpc/grpc_integration_test.cpp:1154-1161 must prove
the cancelled job never starts: after the existing terminal-state assertion,
submit a probe job and assert it starts and finishes within bounded waits,
demonstrating the worker was released. Update the running-job deletion test at
cpp/tests/linear_programming/grpc/grpc_integration_test.cpp:1195-1206 to measure
only delete_job() latency, then submit a probe job and require bounded start and
completion; retain deterministic assertions that cancellation completes and the
replacement worker processes the probe.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: d5b1f734-c5de-4d43-8e64-65d64492f9ca
📒 Files selected for processing (8)
cpp/docs/grpc-server-architecture.mdcpp/src/grpc/client/grpc_client.hppcpp/src/grpc/server/grpc_job_management.cppcpp/src/grpc/server/grpc_server_types.hppcpp/src/grpc/server/grpc_service_impl.cppcpp/tests/linear_programming/grpc/grpc_integration_test.cppdocs/cuopt/source/cuopt-grpc/api.rstdocs/cuopt/source/cuopt-grpc/grpc-server-architecture.md
Comment on lines
+530
to
+544
| (void)context; | ||
| std::string job_id = request->job_id(); | ||
|
|
||
| size_t erased = 0; | ||
| { | ||
| std::lock_guard<std::mutex> lock(tracker_mutex); | ||
| erased = job_tracker.erase(job_id); | ||
| } | ||
|
|
||
| if (erased == 0) { | ||
| std::string message; | ||
| if (!delete_job(job_id, message)) { | ||
| response->set_status(cuopt::remote::ERROR_NOT_FOUND); | ||
| response->set_message("Job not found: " + job_id); | ||
| response->set_message(message); | ||
| if (config.verbose) { | ||
| SERVER_LOG_DEBUG("[gRPC] DeleteResult job not found: %s", job_id.c_str()); | ||
| } | ||
| return Status::OK; | ||
| } | ||
|
|
||
| delete_log_file(job_id); | ||
|
|
||
| response->set_status(cuopt::remote::SUCCESS); | ||
| response->set_message("Result deleted"); | ||
| response->set_message(message); |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
Invalidate active chunked-download sessions during deletion.
StartChunkedDownload snapshots completed results into chunked_downloads, but delete_job only erases tracker/pending maps and the log file. A client holding a download_id can therefore continue calling GetResultChunk after DeleteResult, while the raw result remains resident until the session is finished or expires. Associate sessions with job_id and erase matching sessions under chunked_downloads_mutex as part of deletion, including the race where a session is created concurrently with deletion.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@cpp/src/grpc/server/grpc_service_impl.cpp` around lines 530 - 544, The
DeleteResult flow must invalidate all active chunked-download sessions for the
deleted job, including sessions created concurrently with deletion. Update
StartChunkedDownload and delete_job to associate each session with job_id, erase
matching entries while holding chunked_downloads_mutex during deletion, and
coordinate the deletion/session-creation ordering so no session can be created
after deletion completes.
Source: Path instructions
Contributor
Author
There was a problem hiding this comment.
separate scope, not for this PR
There was a problem hiding this comment.
@tmckayus, understood—this is out of scope for this PR. Would you like me to open a follow-up GitHub issue for the chunked-download session invalidation work?
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
repo="NVIDIA/cuopt"
title="Invalidate chunked-download sessions when deleting a job"
body=$(cat <<'EOF'
## Summary
`DeleteResult` should invalidate all active chunked-result download sessions associated with the deleted job.
## Rationale
`StartChunkedDownload` snapshots completed result arrays into `chunked_downloads`. Deleting the job currently removes the tracker entry, pending data, and log file, but a client that already holds a `download_id` can continue retrieving the snapshotted result. This leaves deleted job data accessible and resident until the download session is finished or expires.
## Affected areas
- `cpp/src/grpc/server/grpc_service_impl.cpp`
- `StartChunkedDownload`
- `GetResultChunk`
- download-session lifecycle coordination
- `cpp/src/grpc/server/grpc_job_management.cpp`
- `delete_job` cleanup path
- `cpp/src/grpc/server/grpc_server_types.hpp`
- any required download-session metadata or lifecycle declarations
- gRPC integration tests covering `DeleteResult` and chunked downloads
## Required changes
1. Associate each `ChunkedDownloadState` entry with its originating `job_id`.
2. During `delete_job`, remove every matching download session while holding `chunked_downloads_mutex`.
3. Coordinate `StartChunkedDownload` and deletion so that no session can be created after deletion completes, including concurrent requests.
4. Ensure subsequent `GetResultChunk` calls for an invalidated `download_id` return the existing unknown-download/not-found behavior.
5. Add integration coverage for deleting a job with an active chunked download and for the creation/deletion race.
## Acceptance criteria
- After a successful `DeleteResult`, no active chunked-download session remains for that job.
- A previously issued `download_id` cannot return any further result data after deletion.
- Concurrent `StartChunkedDownload` and `DeleteResult` cannot leave a usable session for a deleted job.
- Existing chunked-download behavior for non-deleted jobs remains unchanged.
## Backlinks
- Pull request: https://github.com/NVIDIA/cuopt/pull/1604
- Review discussion: https://github.com/NVIDIA/cuopt/pull/1604#discussion_r3634569391
- Requested by: `@tmckayus`
EOF
)
gh issue create \
--repo "$repo" \
--title "$title" \
--body "$body" \
--assignee "tmckayus"Length of output: 5284
@tmckayus, created and assigned the follow-up issue: #1612.
🧬 Code Graph Analysis Results
Relevant snippets
cpp/src/grpc/server/grpc_server_types.hpp
-
Lines 313-316
inline std::string get_log_file_path(const std::string& job_id) { return LOG_DIR + "/job_" + job_id + ".log"; }
Used by:
StreamLogs()to locate the per-job log file. -
Lines 318-318
inline int64_t server_max_message_bytes() { return config.max_message_bytes; }
Used by: result/chunk size checks and setting
max_message_bytesin responses. -
Lines 345-345
generate_job_id()Used by:
StartChunkedUpload()andStartChunkedDownload()to createupload_id/download_id. -
Lines 370-371
submit_job_async(std::vector<uint8_t>&& request_data, uint32_t problem_category)
Used by: unary
SubmitJob(). -
Lines 372-373
submit_chunked_job_async(PendingChunkedUpload&& chunked_data, uint32_t problem_category)
Used by:
FinishChunkedUpload(). -
Lines 374-374
check_job_status(const std::string& job_id, std::string& message)
Used by:
CheckStatus(),WaitForCompletion(),StreamLogs(),GetIncumbents(), and multiple terminal-state decisions. -
Lines 375-375
cancel_job(const std::string& job_id, JobStatus& job_status_out, std::string& message)
Used by:
CancelJob(). -
Lines 377-377
delete_job(const std::string& job_id, std::string& message)
Used by:
DeleteResult().
cpp/src/grpc/server/grpc_pipe_serialization.hpp
- Lines 459-467 (includes full function)
Used by: unary
inline std::vector<uint8_t> serialize_submit_request_to_pipe( const cuopt::remote::SubmitJobRequest& request) { size_t byte_size = request.ByteSizeLong(); if (byte_size == 0 || byte_size > static_cast<size_t>(std::numeric_limits<int>::max())) return {}; std::vector<uint8_t> blob(byte_size); request.SerializeToArray(blob.data(), static_cast<int>(byte_size)); return blob; }
SubmitJob()to serialize the request into the pipe payload. Returns an empty vector if the serialized size is 0 or exceedsintmax.
cpp/src/grpc/server/grpc_field_element_size.hpp
- Lines 25-28 (includes full function body)
Used by:
inline int64_t array_field_element_size(int32_t container_field_num, int32_t field_id) { `#include` "generated_array_field_element_size.inc" }
SendArrayChunk()to validate known(container_field_num, field_id)pairs and determine per-element byte size (the implementation is generated via the included.incfile).
CI Test Summary✅ All 31 test job(s) passed. |
Signed-off-by: Trevor McKay <tmgithub1@gmail.com>
tmckayus
force-pushed
the
fix/grpc-delete-cancels-job
branch
from
July 23, 2026 14:40
6d0d1ef to
583d3de
Compare
ramakrishnap-nv
approved these changes
Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix an oversight in the grpc server. If a job is simply deleted while it is queued or running, the result tracking is removed but the job will continue to run (and might run when it reaches the top of the queue). The correct semantics are to call "cancel" first on non-terminal jobs before running "delete".