Skip to content

CLDSRV-1056: Fail a self-copy when no known master key decrypts its data - #6351

Open
dvasilas wants to merge 7 commits into
development/9.3from
bugfix/CLDSRV-1056-keep-data-of-damaged-object-on-self-copy
Open

dvasilas wants to merge 7 commits into
development/9.3from
bugfix/CLDSRV-1056-keep-data-of-damaged-object-on-self-copy

Conversation

@dvasilas

@dvasilas dvasilas commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

TLDR

#6346 fixed CLDSRV-910.

But after the fix, a self-copy of an object damaged by CLDSRV-910 to a master key other than the one in its metadata replaces its content with the wrong content and deletes the original.

With this PR, before such a self-copy rewrites the data, it checks which master key decrypts it and uses that key. If none of the keys it tries does, the copy fails and nothing is written.


The original bug (CLDSRV-910)

Using a self-copy to re-encrypt an object under another master key changed only the master key ID in the object metadata. The data and its data key stayed encrypted under the old master key.
Reading such an object unwraps the data key with the wrong master key: either GET returns incorrect content, or it fails if KMS detects the wrong key.
Before the CLDSRV-910 fix, a self-copy naming the old master key repaired the object: it changed the master key ID in the metadata back to the old key.

Problem

Since the CLDSRV-910 fix, a self-copy to a master key other than the one in the object metadata rewrites the data.
Here is what happens if such a self-copy is done to an object damaged by CLDSRV-910, even when it names the master key the data is encrypted under.
The self-copy:

  • Decrypts the data with the master key in the object metadata (the wrong one)
  • Stores the wrong content, encrypted under the destination master key
  • Deletes the original data

The original content can no longer be recovered.

Since the CLDSRV-910 fix, a self-copy rewrites the data.
Here is what happens if a self-copy with another master key is done to an object damaged by CLDSRV-910.
The self-copy:

  • Decrypts the data with the master key in the object metadata (the wrong one)
  • Stores the wrong content
  • Deletes the original data.

The object can no longer be repaired.

This affects only objects damaged by CLDSRV-910.
Self-copy for objects not affected by CLDSRV-910 works as expected.

Fix

With this PR, before rewriting the data, a self-copy decrypts the smallest encrypted part with up to 3 master keys, in this order: the master key in the object metadata, the master key recorded on the locations, and the destination master key.
It compares the MD5 of the decrypted part with the MD5 recorded for that part.
The copy reads the data with the first key that matches. If no key matches, or the part has no recorded MD5, the copy fails with InternalError, and nothing is written.

For example, the data is encrypted under K1 and the metadata names K2.
K2 doesn't match; K1 does:

  • A copy to K1 leaves the data as it is and records K1 in the metadata.
  • A copy to another key, K3, decrypts the data with K1 and encrypts it under K3.

When the new check runs

When the check runs, it costs one read of the smallest part and one KMS decrypt per key tried.

This check runs only when

  • The self-copy rewrites the data and replaces it (versioning not enabled), and
  • The locations record a master key.

This covers any object updated by a self-copy that didn't rewrite the data (this includes objects damaged by CLDSRV-910), even if its metadata no longer says it is encrypted. A self-copy without SSE, in a bucket without default encryption, removes the encryption from the metadata but leaves the data encrypted.
Other objects are rewritten as before, without the extra read.


Notes

  • An encrypted object written before S3C 7.1 has no recorded MD5. If its locations record a master key, a self-copy that rewrites it now fails with InternalError.
  • The check also covers a self-copy to another location constraint (x-amz-metadata-directive: REPLACE with x-amz-meta-scal-location-constraint). That copy always rewrites the data, so it replaced damaged objects with the wrong content before CLDSRV-910 too.
  • Not covered:
    • A self-copy with UploadPartCopy and CompleteMultipartUpload (CLDSRV-1071).
    • A self-copy without SSE headers, in a bucket without default encryption, removes the encryption from the object metadata but leaves the data encrypted (CLDSRV-1072).

A self-copy uses it to read the source data and find the master key that decrypts it; the code is the same as on 9.4.
@bert-e

bert-e commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Hello dvasilas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval ⭐
/bypass_build_status Bypass the build and test status ⭐
/bypass_commit_size Bypass the check on the size of the changeset TBA ⭐
/bypass_incompatible_branch Bypass the check on the source branch prefix ⭐
/bypass_jira_check Bypass the Jira issue check ⭐
/bypass_peer_approval Bypass the pull request peers' approval ⭐
/bypass_leader_approval Bypass the pull request leaders' approval ⭐
/bypass_source_branch_lineage Bypass the cross-branch contamination check ⭐
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@bert-e

bert-e commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Incorrect fix version

The Fix Version/s in issue CLDSRV-1056 contains:

  • None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 9.3.26

  • 9.4.11

  • 9.5.3

  • 9.6.0

Please check the Fix Version/s of CLDSRV-1056, or the target
branch of this pull request.

Comment thread lib/api/objectCopy.js Dismissed
@leif-scality
leif-scality self-requested a review November 5, 2026 22:53
@dvasilas
dvasilas force-pushed the bugfix/CLDSRV-1056-keep-data-of-damaged-object-on-self-copy branch from 3c6de9a to 479bc28 Compare November 11, 2026 09:23
@bert-e

bert-e commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Incorrect fix version

The Fix Version/s in issue CLDSRV-1056 contains:

  • None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 9.3.30

  • 9.4.15

  • 9.5.5

  • 9.6.0

Please check the Fix Version/s of CLDSRV-1056, or the target
branch of this pull request.

Comment thread lib/api/objectCopy.js Resolved
@codecov

codecov Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
8623 2 8621 0
View the top 2 failed test(s) by shortest run time
"after each" hook for "should return an error if attempt to copy nonexistent object"::Object Part Copy With default signature "after each" hook for "should return an error if attempt to copy nonexistent object"
Stack Traces | 4s run time
The specified bucket does not exist.
"before each" hook for "should return an error if attempt to copy nonexistent object"::Object Part Copy With default signature "before each" hook for "should return an error if attempt to copy nonexistent object"
Stack Traces | 6.15s run time
We encountered an internal error. Please try again.
View the full list of 14 ❄️ flaky test(s)
downloaded file should equal file that was put::aws-node-sdk v4auth query tests downloaded file should equal file that was put

Flake rate in main: 13.81% (Passed 668 times, Failed 107 times)

Stack Traces | 0.015s run time
Expected values to be strictly equal:

1 !== 0
should create a bucket::aws-node-sdk v4auth query tests should create a bucket

Flake rate in main: 13.81% (Passed 668 times, Failed 107 times)

Stack Traces | 5.97s run time
Expected values to be strictly equal:
+ actual - expected

+ '500 INTERNAL SERVER ERROR'
- '200 OK'
should delete a bucket::aws-node-sdk v2auth query tests should delete a bucket

Flake rate in main: 13.81% (Passed 668 times, Failed 107 times)

Stack Traces | 5.33s run time
Expected values to be strictly equal:
+ actual - expected

+ '500 INTERNAL SERVER ERROR'
- '204 NO CONTENT'
should delete a bucket::aws-node-sdk v4auth query tests should delete a bucket

Flake rate in main: 13.81% (Passed 668 times, Failed 107 times)

Stack Traces | 0.026s run time
Expected values to be strictly equal:
+ actual - expected

+ '404 NOT FOUND'
- '204 NO CONTENT'
should delete an object with native characters::aws-node-sdk v4auth query tests should delete an object with native characters

Flake rate in main: 13.81% (Passed 668 times, Failed 107 times)

Stack Traces | 0.025s run time
Expected values to be strictly equal:
+ actual - expected

+ '404 NOT FOUND'
- '204 NO CONTENT'
should delete an object::aws-node-sdk v4auth query tests should delete an object

Flake rate in main: 13.81% (Passed 668 times, Failed 107 times)

Stack Traces | 0.024s run time
Expected values to be strictly equal:
+ actual - expected

+ '404 NOT FOUND'
- '204 NO CONTENT'
should do a bucket listing with result::aws-node-sdk v4auth query tests should do a bucket listing with result

Flake rate in main: 13.81% (Passed 668 times, Failed 107 times)

Stack Traces | 0.029s run time
Cannot read properties of undefined (reading 'map')
should get an object::aws-node-sdk v4auth query tests should get an object

Flake rate in main: 13.81% (Passed 668 times, Failed 107 times)

Stack Traces | 0.026s run time
Expected values to be strictly equal:
+ actual - expected

+ '404 NOT FOUND'
- '200 OK'
should list objects in bucket::aws-node-sdk v4auth query tests should list objects in bucket

Flake rate in main: 13.81% (Passed 668 times, Failed 107 times)

Stack Traces | 0.026s run time
Expected values to be strictly equal:
+ actual - expected

+ '404 NOT FOUND'
- '200 OK'
should put an object with an acl setting and a storage class setting::aws-node-sdk v4auth query tests should put an object with an acl setting and a storage class setting

Flake rate in main: 13.81% (Passed 668 times, Failed 107 times)

Stack Traces | 0.032s run time
Expected values to be strictly equal:
+ actual - expected

+ '404 NOT FOUND'
- '200 OK'
should put an object with native characters::aws-node-sdk v4auth query tests should put an object with native characters

Flake rate in main: 13.81% (Passed 668 times, Failed 107 times)

Stack Traces | 0.028s run time
Expected values to be strictly equal:
+ actual - expected

+ '404 NOT FOUND'
- '200 OK'
should put an object::aws-node-sdk v4auth query tests should put an object

Flake rate in main: 13.81% (Passed 668 times, Failed 107 times)

Stack Traces | 0.026s run time
Expected values to be strictly equal:
+ actual - expected

+ '404 NOT FOUND'
- '200 OK'
should return 204 on delete of non-existing object::aws-node-sdk v4auth query tests should return 204 on delete of non-existing object

Flake rate in main: 13.81% (Passed 668 times, Failed 107 times)

Stack Traces | 0.026s run time
Expected values to be strictly equal:
+ actual - expected

+ '404 NOT FOUND'
- '204 NO CONTENT'
should return a 204 on delete of an already deleted object::aws-node-sdk v4auth query tests should return a 204 on delete of an already deleted object

Flake rate in main: 13.81% (Passed 668 times, Failed 107 times)

Stack Traces | 0.026s run time
Expected values to be strictly equal:
+ actual - expected

+ '404 NOT FOUND'
- '204 NO CONTENT'

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@bert-e

bert-e commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

…rites it

Until CLDSRV-910, a self-copy to another master key changed only the metadata and left the data encrypted under the old one; a self-copy that rewrites such data decrypts it with the wrong key and replaces the content with wrong bytes.
@dvasilas
dvasilas force-pushed the bugfix/CLDSRV-1056-keep-data-of-damaged-object-on-self-copy branch from 479bc28 to 98543fe Compare November 11, 2026 12:55
@dvasilas
dvasilas marked this pull request as ready for review November 11, 2026 12:58
Comment on lines +41 to +51
const rank = part => (part.size === 0 ? Infinity : part.size);
return parts.reduce((smallest, part) => {
const match = partETagPattern.exec(part.dataStoreETag);
if (!match || !part.locations.every(location => location.cipheredDataKey)) {
return smallest;
}
if (smallest && rank(smallest) <= rank(part)) {
return smallest;
}
return { locations: part.locations, md5: match[1], size: part.size };
}, null);

@leif-scality leif-scality Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor (rare, but the loss is permanent): a wrong key can pass the check on a tiny last part

  • Cause: the check decrypts only the smallest part that has an MD5 (sourceMasterKey.js:41-51). With a KMS that never rejects a wrong master key (file and memory), a wrong key turns a one-byte part into the right byte once in 256.
  • Effect: the check accepts that key, the copy decrypts every other part with it, stores the result and deletes the original. The client gets 200.
  • Odds: 1 in 256 for a one-byte part, 1 in 65,536 for two bytes, 1 in 16.7 million for three, per wrong key tried before the right one. The metadata key is tried first, so every damaged object is exposed to it. Retrying does not change the outcome for a given key and object.
  • Not affected: objects that are not multipart (the whole object is checked, so a match means the content is right), and a KMS that rejects a wrong key.
  • Test: should not accept a key that decrypts only a one-byte last part (code below). It builds a 5 MiB + 1 byte multipart object under K1, creates keys until one decrypts the last byte correctly, sets it as the metadata key, and self-copies to K3:
    AssertionError [ERR_ASSERTION]: the copy stored wrong content
    
  • Fix: do not trust a part of a few bytes alone. Add the next smallest parts until at least 16 bytes are checked.
Test

Goes in describe('which master key decrypts the data') in tests/unit/api/objectCopy.js.

it('should not accept a key that decrypts only a one-byte last part', async function test() {
    this.timeout(60000);
    const { promisify } = require('util');
    const parts = [Buffer.alloc(constants.minimumAllowedPartSize, 'a'), Buffer.from('z')];
    const [k1, k3] = await promisify(createKeys)(2);
    // A two-part object under k1, whose last part is one byte.
    const uploadId = await promisify(initiateMpu)(objectKey, kmsHeaders(k1));
    for (let i = 0; i < parts.length; i++) {
        // eslint-disable-next-line no-await-in-loop
        await new Promise((resolve, reject) =>
            objectPutPart(authInfo, partRequest(objectKey, uploadId, i + 1, parts[i]), undefined, log, err =>
                (err ? reject(err) : resolve()),
            ),
        );
    }
    await promisify(completeMpu)(objectKey, uploadId, parts.map(md5));
    const { location } = await promisify(getObjectMD)();
    const last = location[location.length - 1];
    // Decrypts the stored byte of the last part with a master key, as data.get does.
    const lastByteWith = masterKeyId =>
        new Promise((resolve, reject) =>
            kms.createDecipherBundle(
                {
                    cryptoScheme: last.cryptoScheme,
                    masterKeyId,
                    cipheredDataKey: Buffer.from(last.cipheredDataKey, 'base64'),
                },
                0,
                log,
                (err, bundle) => (err ? reject(err) : resolve(bundle.decipher.update(ds[last.key].value))),
            ),
        );
    // Create keys until one decrypts that byte to 'z': 1 key in 256.
    let k2;
    for (let i = 0; i < 20000 && !k2; i++) {
        // eslint-disable-next-line no-await-in-loop
        const candidate = await promisify(createKey)();
        // eslint-disable-next-line no-await-in-loop
        if ((await lastByteWith(candidate)).equals(parts[1])) {
            k2 = candidate;
        }
    }
    assert(k2, 'no such key found');
    // The damaged state: data under k1, which the locations record, and k2 in the metadata.
    await promisify(updateObjectMD)(md => {
        // eslint-disable-next-line no-param-reassign
        md['x-amz-server-side-encryption-aws-kms-key-id'] = k2;
        recordKeyOnLocations(md, k1);
    });
    await promisify(selfCopy)(kmsHeaders(k3));
    const body = await promisify(getBody)();
    assert(body.equals(Buffer.concat(parts)), 'the copy stored wrong content');
});

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I changed the algorithm that selects the smallest part to select a part of at least 16 bytes if one exists, and fall back to a smaller part if none exists.

…aster key

A wrong key decrypts a part of a few bytes to the right bytes too often: one key in 256 for one byte.
*/
function _partToCheck(dataLocator) {
const parts = [];
dataLocator.forEach(location => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I tend to prefer the ES6 construct for (const ... of ...), more flexible (e.g. you can break out of it).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed.

Comment on lines +48 to +65
const tier = part => {
if (part.size === 0) {
return 2;
}
return part.size < minCheckedPartSize ? 1 : 0;
};
const isPreferred = (part, other) =>
tier(part) < tier(other) || (tier(part) === tier(other) && part.size < other.size);
return parts.reduce((chosen, part) => {
const match = partETagPattern.exec(part.dataStoreETag);
if (!match || !part.locations.every(location => location.cipheredDataKey)) {
return chosen;
}
if (chosen && !isPreferred(part, chosen)) {
return chosen;
}
return { locations: part.locations, md5: match[1], size: part.size };
}, null);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Proposing a simpler and maybe slightly better way to reason about the choice, I'd use the following comparison logic and remove the tiering logic and the special case for empty parts:

If both parts being compared are larger than minCheckedPartSize, pick the smaller one, otherwise pick the larger one.

The difference I see with the current tiered algorithm is that it takes the larger of two parts smaller than minCheckedPartSize (instead of the smaller), which I think is desirable to minimize the risk of false positive. It would also naturally pick an empty part only if all parts are empty (assuming this only happens in the single-part non-MPU case).

Also you could include the ETag match and the fact that locations have a ciphered key within isPreferred, making the full comparison logic self-contained.

By the way, I was under the impression that prior to CLDSRV-910, it was possible to have an unencrypted object with an encryption key attached after a buggy ObjectCopy, would this mean that we could also attempt to compute the MD5 of the part as it is stored in its plain-text form, to detect this case and potentially fix it too? I may be wrong, or maybe it's already the case, please ignore if I am missing something.

@dvasilas dvasilas Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that's much more elegant and easier to understand!
I put the ETag and cipheredDataKey checks in a filter before the reduce.

On the unencrypted object case, see #6351 (comment)

Comment thread lib/api/objectCopy.js
// key on the locations, so objects without one are skipped.
if (
!recordedMasterKeyId ||
!_overwritesSource(sourceIsDestination, destBucketMD) ||

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering why we only process the overwrites, not server-side copies to a new object, is it not prone to the same issue where the destination would be unreadable? Is it to limit the performance impact?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right that we have the same issue when copying to a new object (the destination's data will be incorrect).
I didn't address it here (1) because, in that case, the source is kept and can be repaired (with a self-copy, the source is lost) and (2) to keep the fix focused on the self-copy case (self-copy will be the recommended way to re-encrypt under a new key).

Comment thread lib/api/objectCopy.js
if (
!recordedMasterKeyId ||
!_overwritesSource(sourceIsDestination, destBucketMD) ||
!dataLocator.some(location => location.cipheredDataKey) ||

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cf the end of my comment https://github.com/scality/cloudserver/pull/6351/changes#r4621291776 wondering why we only check the MD5 of ciphered parts.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prior to CLDSRV-910, it was possible to have an unencrypted object with an encryption key attached after a buggy ObjectCopy

That is true. But even if metadata have a master key ID, we can tell that the object data is unencrypted because locations have no cipheredDataKey.
A self-copy of an object with unencrypted data doesn't need this check, because it reads the data as stored, without decryption.

Comment thread lib/api/objectCopy.js Outdated
Comment on lines +632 to +633
// eslint-disable-next-line no-param-reassign
request.sourceServerAccessLog && (request.sourceServerAccessLog.error = error);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not fond of this construct, I'd use a regular if

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 changed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants