HDDS-15606. Support S3 object metadata headers (Cache-Control, Expires, Content-Encoding)

[Gargi-jais11]

What changes were proposed in this pull request?

Ozone does not persist standard S3 object headers on PUT or return them on HEAD/GET. s3-tests fail for:

• Cache-Control (no-cache from NoCacheFilter instead of stored value) - test_object_write_cache_control,
• Expires (off by 6000s because filter sets Expires=now) - test_object_write_expires, and
• Content-Encoding (KeyError on HEAD) -test_object_content_encoding_aws_chunked.

Implement persist-on-PUT and return-on-HEAD/GET for these headers, strip aws-chunked from Content-Encoding per AWS semantics, and ensure object headers override HDDS NoCacheFilter defaults on S3 object responses.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15606

How was this patch tested?

Added IT and UT.

Before Fix:

bash-5.1$ aws --endpoint-url $ENDPOINT s3api put-object \
  --bucket testbucket \
  --key foo \
  --body /tmp/foo-body \
  --cache-control "public, max-age=14400"
{
    "ETag": "\"c157a79031e1c40f85931829bc5fc552\""
}
bash-5.1$ aws --endpoint-url $ENDPOINT s3api head-object \
  --bucket testbucket \
  --key foo \
  --query 'CacheControl'
"no-cache"                    <---------------- wrong output


bash-5.1$ EXPIRES=$(date -u -v+6000S '+%a, %d %b %Y %H:%M:%S GMT' 2>/dev/null \
  || date -u -d '+6000 seconds' '+%a, %d %b %Y %H:%M:%S GMT')
aws --endpoint-url $ENDPOINT s3api put-object \
  --bucket testbucket \
  --key foo-expires \
  --body /tmp/foo-body \
  --expires "$EXPIRES"
{
    "ETag": "\"c157a79031e1c40f85931829bc5fc552\""
}
bash-5.1$ aws --endpoint-url $ENDPOINT s3api head-object \
  --bucket testbucket \
  --key foo-expires \
  --query 'Expires'
"Thu, 16 Jul 2026 11:58:46 GMT"                     <---------------- wrong output


bash-5.1$ KEY=encoding
aws --endpoint-url $ENDPOINT s3api put-object --bucket testbucket --key $KEY --body /tmp/foo-body --content-encoding gzip
aws --endpoint-url $ENDPOINT s3api head-object --bucket testbucket --key $KEY --query 'ContentEncoding'
{
    "ETag": "\"c157a79031e1c40f85931829bc5fc552\""
}
null                     <---------------- wrong output

bash-5.1$ aws --endpoint-url $ENDPOINT s3api put-object --bucket testbucket --key $KEY --body /tmp/foo-body --content-encoding "deflate, gzip"
aws --endpoint-url $ENDPOINT s3api head-object --bucket testbucket --key $KEY --query 'ContentEncoding'
{
    "ETag": "\"c157a79031e1c40f85931829bc5fc552\""
}
null                    <---------------- wrong output

bash-5.1$ aws --endpoint-url $ENDPOINT s3api put-object --bucket testbucket --key $KEY --body /tmp/foo-body --content-encoding "gzip, aws-chunked"
aws --endpoint-url $ENDPOINT s3api head-object --bucket testbucket --key $KEY --query 'ContentEncoding'
{
    "ETag": "\"c157a79031e1c40f85931829bc5fc552\""
}
null                    <---------------- wrong output

bash-5.1$ aws --endpoint-url $ENDPOINT s3api put-object --bucket testbucket --key $KEY --body /tmp/foo-body --content-encoding "aws-chunked"
aws --endpoint-url $ENDPOINT s3api head-object --bucket testbucket --key $KEY --query 'ContentEncoding'
{
    "ETag": "\"c157a79031e1c40f85931829bc5fc552\""
}
null

After Fix:

bash-5.1$ ozone sh bucket create /s3v/testbucket
bash-5.1$ aws --endpoint-url $ENDPOINT s3api put-object \
  --bucket testbucket \
  --key foo \
  --body /tmp/foo-body \
  --cache-control "public, max-age=14400"
{
    "ETag": "\"c157a79031e1c40f85931829bc5fc552\""
}
bash-5.1$ aws --endpoint-url $ENDPOINT s3api head-object \
  --bucket testbucket \
  --key foo \
  --query 'CacheControl'
"public, max-age=14400"                <-------------Fixed

------------
------------

bash-5.1$ aws --endpoint-url http://s3g:9878/ s3api put-object   --bucket testbucket --key foo --body /tmp/foo-body   --expires "Wed, 21 Oct 2026 07:50:00 GMT"
{
    "ETag": "\"c157a79031e1c40f85931829bc5fc552\""
}
bash-5.1$ aws --endpoint-url http://s3g:9878/ s3api head-object \
  --bucket testbucket --key foo --query Expires
"Wed, 21 Oct 2026 07:50:00 GMT"                <-------------Fixed

----------
----------

bash-5.1$ KEY=encoding
bash-5.1$ aws --endpoint-url $ENDPOINT s3api put-object --bucket testbucket --key $KEY --body /tmp/foo-body --content-encoding gzip
{
    "ETag": "\"c157a79031e1c40f85931829bc5fc552\""
}

bash-5.1$ aws --endpoint-url $ENDPOINT s3api head-object --bucket testbucket --key $KEY --query 'ContentEncoding'
"gzip"                <-------------Fixed

bash-5.1$ aws --endpoint-url $ENDPOINT s3api put-object --bucket testbucket --key $KEY --body /tmp/foo-body --content-encoding "deflate, gzip"
aws --endpoint-url $ENDPOINT s3api head-object --bucket testbucket --key $KEY --query 'ContentEncoding'
{
    "ETag": "\"c157a79031e1c40f85931829bc5fc552\""
}
"deflate, gzip"                <-------------Fixed

bash-5.1$ aws --endpoint-url $ENDPOINT s3api put-object --bucket testbucket --key $KEY --body /tmp/foo-body --content-encoding "gzip, aws-chunked"
aws --endpoint-url $ENDPOINT s3api head-object --bucket testbucket --key $KEY --query 'ContentEncoding'
{
    "ETag": "\"c157a79031e1c40f85931829bc5fc552\""
}
"gzip"                <-------------Fixed

bash-5.1$ aws --endpoint-url $ENDPOINT s3api put-object --bucket testbucket --key $KEY --body /tmp/foo-body --content-encoding "aws-chunked"
aws --endpoint-url $ENDPOINT s3api head-object --bucket testbucket --key $KEY --query 'ContentEncoding'
{
    "ETag": "\"c157a79031e1c40f85931829bc5fc552\""
}
null                <-------------Fixed

[ArafatKhan2198]

The PR internally renames:

x-amz-meta-cache-control

to:

cache-control-custom

But a user can also provide:

x-amz-meta-cache-control-custom

Both values then use the same internal key, so one value can overwrite the other.
The internal storage key should use a reserved namespace that users cannot create.

[ArafatKhan2198]

Handle aws-chunked without case sensitivity

Currently, only this exact value is removed:

aws-chunked

These would not be removed:

AWS-CHUNKED
Aws-Chunked

HTTP encoding names are case-insensitive, so the code should use:

equalsIgnoreCase()

[ArafatKhan2198]

Add tests for copy and multipart upload

The code changes normal upload, multipart upload, and object-copy behaviour.

However, the integration tests mainly verify normal PutObject.

Tests should also confirm that headers work correctly when:

• uploading through multipart upload;
• copying metadata with COPY;
• replacing metadata with REPLACE.

Overall, the main feature is implemented correctly, but these edge cases could cause incorrect headers or lost metadata.