Skip to content

CLDSRV-964: remove MPU checksums for external backends (GCP, AWS S3, Azure) #6242

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bert-e merged 2 commits into from
Aug 4, 2026
+485 −232
Merged

CLDSRV-964: remove MPU checksums for external backends (GCP, AWS S3, Azure) #6242

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2ebc57c
CLDSRV-964: remove MPU checksums for external backends (GCP, AWS S3, …
leif-scality Jul 30, 2026
123fa0c
CLDSRV-964: prettier lint
leif-scality Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 26 additions & 32 deletions lib/api/completeMultipartUpload.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,30 +52,24 @@ const allChecksumXmlTags = Object.values(checksumAlgorithms).map(algo => algo.xm
* does not match the stored part's ChecksumValue, return InvalidPart.
* - If checksumType === 'COMPOSITE' and checksumIsDefault is false, every part
* in the request body MUST include the matching Checksum<Algo> field;
* missing → InvalidRequest. (Relaxed for external backends, which store no
* per-part checksum - but a checksum the client does submit is still checked,
* and rejected, since there is no stored value to match.)
* missing → InvalidRequest.
*
* @param {object} jsonList - parsed CompleteMultipartUpload XML
* @param {array} storedParts - parts as returned by services.getMPUparts
* @param {string} mpuSplitter - splitter used in part keys
* @param {object} mpuChecksum - { algorithm, type, isDefault }
* @param {boolean} isExternal - external-backend MPU; relax the COMPOSITE
* per-part requirement (external parts carry no stored checksum)
* @returns {Error|null}
*/
function validatePerPartChecksums(jsonList, storedParts, mpuSplitter, mpuChecksum, isExternal) {
function validatePerPartChecksums(jsonList, storedParts, mpuSplitter, mpuChecksum) {
const mpuAlgo = mpuChecksum.algorithm;
if (!mpuAlgo) {
// Legacy / pre-checksums MPU, no algorithm tracked, nothing to validate.
// Legacy / pre-checksums MPU, or external-backend MPU,
// no algorithm tracked, nothing to validate.
return null;
}
const expectedTag = checksumAlgorithms[mpuAlgo] ? checksumAlgorithms[mpuAlgo].xmlTag : null;
// Skip enforcement if the MPU's algorithm is unknown (shouldn't happen).
// External backends store no per-part checksum, so don't require one; a
// checksum the client does submit is still rejected below (no stored value).
const requireForEachPart =
mpuChecksum.type === 'COMPOSITE' && !mpuChecksum.isDefault && expectedTag !== null && !isExternal;
const requireForEachPart = mpuChecksum.type === 'COMPOSITE' && !mpuChecksum.isDefault && expectedTag !== null;

const storedByPartNumber = new Map();
storedParts.forEach(item => {
Expand Down Expand Up @@ -357,14 +351,21 @@ function completeMultipartUpload(authInfo, request, log, callback) {
}
const mpuType = storedMetadata.checksumType;
if (!mpuType) {
// Legacy MPU created before checksumType was tracked.
const typeErr = errorInstances.InvalidRequest.customizeDescription(
'The upload was not created with a checksum mode. ' +
'The complete request must not include a x-amz-checksum-type header.',
);
return next(typeErr, destBucket);
}
if (headerTypeUpper !== mpuType.toUpperCase()) {
// External-backend MPUs record no checksum config
// (CLDSRV-964): ignore the header, like every other
// checksum input on CompleteMPU for external backends.
const mpuLocation = storedMetadata.controllingLocationConstraint;
const isExternalMpu =
!!constants.externalBackends[config.getLocationConstraintType(mpuLocation)];
if (!isExternalMpu) {
// Legacy MPU created before checksumType was tracked.
const typeErr = errorInstances.InvalidRequest.customizeDescription(
'The upload was not created with a checksum mode. ' +
'The complete request must not include a x-amz-checksum-type header.',
);
return next(typeErr, destBucket);
}
} else if (headerTypeUpper !== mpuType.toUpperCase()) {
const typeErr = errorInstances.InvalidRequest.customizeDescription(
`The upload was created using the ${mpuType} checksum mode. ` +
'The complete request must use the same checksum mode.',
Expand Down Expand Up @@ -469,14 +470,7 @@ function completeMultipartUpload(authInfo, request, log, callback) {
type: storedMetadata.checksumType,
isDefault: storedMetadata.checksumIsDefault,
};
const isExternalMpu = !!constants.externalBackends[config.getLocationConstraintType(location)];
const checksumErr = validatePerPartChecksums(
jsonList,
storedParts,
splitter,
mpuChecksum,
isExternalMpu,
);
const checksumErr = validatePerPartChecksums(jsonList, storedParts, splitter, mpuChecksum);
if (checksumErr) {
log.debug('per-part checksum validation failed', {
error: checksumErr,
Expand Down Expand Up @@ -611,11 +605,11 @@ function completeMultipartUpload(authInfo, request, log, callback) {
totalMPUSize,
next,
) {
// External-handled MPUs (ingestion / external backends) come in
// with completeObjData set and no filteredPartsObj — the data
// store already aggregated the parts, and we have no per-part
// info to feed the compute step. Skip in that case.
if (!filteredPartsObj) {
// Skip the final-checksum compute and its header validation:
// - if no filteredPartsObj then there is no per-part info to compute from (aws_s3/gcp/ingestion
// return no filteredPartsObj; azure returns filteredPartsObj, but its parts store no checksum)
// - if completeObjData is present it means the MPU was completed by an external backend
if (!filteredPartsObj || completeObjData) {
return continueProcessParts(null);
}
computeFinalChecksum(
Expand Down
Loading
Loading