cloud: add changefeed OSS configuration for premium. #23585
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,28 +1,54 @@ | ||
| --- | ||
| title: Sink to Cloud Storage | ||
| summary: This document explains how to create a changefeed to stream data from TiDB Cloud to cloud storage. It includes restrictions, configuration steps for the destination, replication, and specification, as well as starting the replication process. | ||
| --- | ||
|
|
||
| # Sink to Cloud Storage | ||
|
|
||
|
|
||
| This document describes how to create a changefeed to stream data from <CustomContent plan="dedicated">TiDB Cloud Dedicated</CustomContent><CustomContent plan="premium">TiDB Cloud Premium</CustomContent> to cloud storage. | ||
|
|
||
| <CustomContent plan="dedicated"> | ||
|
|
||
| > **Note:** | ||
| > | ||
| > - To stream data from {{{ .dedicated }}} to cloud storage, make sure that your TiDB cluster version is v7.1.1 or later. To upgrade your TiDB Cloud Dedicated cluster to v7.1.1 or later, [contact TiDB Cloud Support](/tidb-cloud/tidb-cloud-support.md). | ||
| > - For [{{{ .starter }}}](/tidb-cloud/select-cluster-tier.md#starter) instances, the changefeed feature is unavailable. | ||
| > - For [{{{ .essential }}}](/tidb-cloud/select-cluster-tier.md#essential) instances, the changefeed feature is only available upon request. For more information, see [Changefeed](/tidb-cloud/essential-changefeed-overview.md). | ||
| > - For [{{{ .premium }}}](/tidb-cloud/select-cluster-tier.md#premium) instances, see [Sink to Cloud Storage](https://docs.pingcap.com/tidbcloud/changefeed-sink-to-cloud-storage/?plan=premium). | ||
|
|
||
| </CustomContent> | ||
|
|
||
| <CustomContent plan="premium"> | ||
|
|
||
| > **Note:** | ||
| > | ||
| > - For [{{{ .starter }}}](/tidb-cloud/select-cluster-tier.md#starter) instances, the changefeed feature is unavailable. | ||
| > - For [{{{ .essential }}}](/tidb-cloud/select-cluster-tier.md#essential) instances, the changefeed feature is only available upon request. For more information, see [Changefeed](/tidb-cloud/essential-changefeed-overview.md). | ||
| > - For [{{{ .dedicated }}}](/tidb-cloud/select-cluster-tier.md#tidb-cloud-dedicated) clusters, see [Sink to Cloud Storage](https://docs.pingcap.com/tidbcloud/changefeed-sink-to-cloud-storage/). | ||
|
|
||
| </CustomContent> | ||
|
|
||
| ## Restrictions | ||
|
|
||
| - For each <CustomContent plan="dedicated">TiDB Cloud Dedicated cluster</CustomContent><CustomContent plan="premium">TiDB Cloud Premium instance</CustomContent>, you can create up to 100 changefeeds. | ||
| - Because TiDB Cloud uses TiCDC to establish changefeeds, it has the same [restrictions as TiCDC](https://docs.pingcap.com/tidb/stable/ticdc-overview#unsupported-scenarios). | ||
| - If the table to be replicated does not have a primary key or a non-null unique index, the absence of a unique constraint during replication could result in duplicated data being inserted downstream in some retry scenarios. | ||
|
|
||
| ## Step 1. Configure destination | ||
|
|
||
| <CustomContent plan="dedicated"> | ||
|
|
||
| Navigate to the overview page of the target TiDB Cloud Dedicated cluster. Click **Data** > **Changefeed** in the left navigation pane, click **Create Changefeed** to go to the **Destination** page, and then select **Amazon S3**, **GCS**, or **Azure Blob Storage** as the destination, depending on the cloud provider on which your TiDB Cloud Dedicated cluster is hosted. The configuration process varies depending on the destination you choose. | ||
|
|
||
| </CustomContent> | ||
|
|
||
| <CustomContent plan="premium"> | ||
|
|
||
| Navigate to the overview page of the target TiDB Cloud Premium instance. Click **Data** > **Changefeed** in the left navigation pane, click **Create Changefeed** to go to the **Destination** page, and then select **Amazon S3** or **Alibaba Cloud OSS** as the destination, depending on the cloud provider on which your TiDB Cloud Premium instance is hosted. The configuration process varies depending on the destination you choose. | ||
|
|
||
| </CustomContent> | ||
|
|
||
| <SimpleTab> | ||
| <div label="Amazon S3"> | ||
|
|
||
|
|
@@ -63,6 +89,9 @@ To use an access key for authentication, follow these steps: | |
| - **Secret Access Key** | ||
|
|
||
| </div> | ||
|
|
||
| <CustomContent plan="dedicated"> | ||
|
qiancai marked this conversation as resolved.
|
||
|
|
||
| <div label="GCS"> | ||
|
|
||
| For **GCS**, before filling **GCS Endpoint**, you need to first grant the GCS bucket access. Take the following steps: | ||
|
|
@@ -118,6 +147,11 @@ For **GCS**, before filling **GCS Endpoint**, you need to first grant the GCS bu | |
| 7. In the TiDB Cloud console, go to the Changefeed's **Destination** page, and fill in the **bucket gsutil URI** field. | ||
|
|
||
| </div> | ||
|
|
||
| </CustomContent> | ||
|
|
||
| <CustomContent plan="dedicated"> | ||
|
|
||
| <div label="Azure Blob Storage"> | ||
|
|
||
| For **Azure Blob Storage**, you must configure the container and get a SAS token in the Azure portal first. Take the following steps: | ||
|
|
@@ -158,6 +192,59 @@ For **Azure Blob Storage**, you must configure the container and get a SAS token | |
| - **SAS Token**: enter the generated SAS token obtained in step 3. | ||
|
|
||
| </div> | ||
|
|
||
| </CustomContent> | ||
|
|
||
| <CustomContent plan="premium"> | ||
|
|
||
| <div label="Alibaba Cloud OSS"> | ||
|
|
||
| For **Alibaba Cloud OSS**, follow these steps to configure the changefeed: | ||
|
|
||
| 1. In the [Alibaba Cloud console](https://www.alibabacloud.com/), perform the following prerequisite steps: | ||
|
|
||
| 1. Create an OSS bucket in the same region as your TiDB cluster. For detailed instructions, see [Create a bucket](https://www.alibabacloud.com/help/en/oss/user-guide/create-a-bucket-4). | ||
| 2. Create a RAM user for the changefeed and generate an AccessKey pair. For detailed instructions, see [Create an AccessKey pair](https://www.alibabacloud.com/help/en/ram/user-guide/create-an-accesskey-pair). | ||
| 3. Grant the RAM user OSS permissions. For the required permissions, see [OSS permissions and access control](https://www.alibabacloud.com/help/en/oss/user-guide/permissions-and-access-control-overview). The RAM user needs at least the following permissions: | ||
|
|
||
| - `oss:ListObjects` | ||
| - `oss:GetObject` | ||
| - `oss:PutObject` | ||
| - `oss:DeleteObject` | ||
|
Comment on lines
+208
to
+213
There was a problem hiding this comment. 🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -e
printf '%s\n' '--- target file size ---'
wc -l tidb-cloud/changefeed-sink-to-cloud-storage.md
printf '%s\n' '--- target context ---'
sed -n '1,230p' tidb-cloud/changefeed-sink-to-cloud-storage.md
printf '%s\n' '--- related terminology ---'
rg -n -i --glob '*.md' --glob '*.mdx' 'RAM user|AccessKey|OSS|ListObjects|PutObject|DeleteObject|least.?privilege|rotate|rotation|custom policy|bucket.*prefix|prefix.*bucket' .
printf '%s\n' '--- changed files/stat ---'
git status --short
git diff --statRepository: pingcap/docs Length of output: 50369 🏁 Script executed: #!/bin/bash
set -e
printf '%s\n' '--- target document ---'
sed -n '1,225p' tidb-cloud/changefeed-sink-to-cloud-storage.md
printf '%s\n' '--- existing OSS policy guidance ---'
sed -n '438,495p' tidb-cloud/premium/backup-and-restore-premium.md
printf '%s\n' '--- external storage access guidance ---'
rg -n -A35 -B8 'Configure Alibaba Cloud Object Storage Service|Alibaba Cloud OSS access' tidb-cloud/configure-external-storage-access.md
printf '%s\n' '--- target diff/status ---'
git status --short
git diff --stat -- tidb-cloud/changefeed-sink-to-cloud-storage.mdRepository: pingcap/docs Length of output: 18313 🌐 Web query:
💡 Result: To configure TiCDC with an Alibaba Cloud OSS sink, you need to grant the RAM user or role specific permissions to manage the objects in your OSS bucket [1][2]. While TiCDC documentation focuses on Amazon S3, the operations required for cloud storage sinks are functionally equivalent across providers [3][4][5]. The minimum permissions required for TiCDC to write data to an OSS bucket are: - oss:ListObjects - oss:GetObject - oss:PutObject If you intend to use advanced features like parallel multipart uploads (where the sink configuration's flush-concurrency is greater than 1), you should also include the following permissions [3][4][5]: - oss:AbortMultipartUpload - oss:ListMultipartUploadParts (and typically oss:ListParts) Additionally, if your configuration requires object deletion or cleanup, ensure you have: - oss:DeleteObject Regarding the use of prefixes, you can restrict these permissions to a specific directory within your bucket by setting the Resource to acs:oss::://* [1][6]. For the oss:ListObjects action specifically, the Resource must point to the bucket itself (acs:oss:::), and you can use the Condition element to restrict listing to a specific path using the oss:Prefix key [1][7]. Example RAM policy snippet for a specific prefix: { "Version": "1", "Statement": [ { "Effect": "Allow", "Action": "oss:ListObjects", "Resource": "acs:oss:::", "Condition": { "StringLike": { "oss:Prefix": ["/", "/"] } } }, { "Effect": "Allow", "Action": ["oss:GetObject", "oss:PutObject", "oss:DeleteObject"], "Resource": "acs:oss::://" } ] } Citations:
🌐 Web query:
💡 Result: In Alibaba Cloud OSS, access control via RAM policies differentiates between bucket-level operations (like ListObjects) and object-level operations (like PutObject, GetObject, and DeleteObject) [1][2]. When restricting access to a specific directory or prefix, the policy must be structured to handle these distinct resource types [1][3]. 1. Object-Level Operations (PutObject, GetObject, DeleteObject) For these operations, you define the Resource using the specific path pattern [1][4]. Resource format: acs:oss:::{bucket_name}/{prefix}/* [1][3]. Example: To grant access to "folder1/", use acs:oss:::mybucket/folder1/ [1][5]. 2. Bucket-Level Operations (ListObjects) The ListObjects operation applies to the bucket itself, not individual objects [1][3]. To restrict listing to a specific prefix, you must use the bucket resource ARN and apply an oss:Prefix condition [1][6]. Resource: acs:oss:::{bucket_name} [1][3]. Condition: Use oss:Prefix within a StringLike condition to specify the allowed prefix [1][3]. Example RAM Policy The following policy grants ListObjects for a specific prefix, along with full access (Put, Get, Delete) to objects within that same path [1][3]. { "Version": "1", "Statement": [ { "Effect": "Allow", "Action": "oss:ListObjects", "Resource": "acs:oss:::examplebucket", "Condition": { "StringLike": { "oss:Prefix": "Development/" } } }, { "Effect": "Allow", "Action": [ "oss:GetObject", "oss:PutObject", "oss:DeleteObject" ], "Resource": "acs:oss:::examplebucket/Development/" } ] } Key Considerations: - Directory Navigation: If using the OSS console, you may also need to include oss:Delimiter "/" in the ListObjects condition to allow the console to traverse directory structures correctly [6][7]. - Wildcards: You can use * in resource paths and prefix conditions to cover multiple objects or sub-directories [1][2][4]. - Multiple Prefixes: The oss:Prefix condition supports lists, allowing you to specify multiple allowed paths in a single statement [1][5]. Citations:
🌐 Web query:
💡 Result: The terms oss:AbortMultipartUpload and oss:ListMultipartUploadParts refer to operations within the Alibaba Cloud Object Storage Service (OSS) API, specifically related to the management of multipart uploads. These API operations are used to handle the lifecycle of large object uploads that are broken into smaller parts: 1. ListMultipartUploadParts: This operation allows you to list the parts that have been uploaded for a specific multipart upload task. It is useful for verifying which parts have been successfully received by OSS, often as part of a reconciliation or error-handling process during data transfer. 2. AbortMultipartUpload: This operation cancels a multipart upload task. When an upload is aborted, OSS deletes the parts that were already uploaded for that specific upload ID, freeing up storage and preventing unnecessary charges for incomplete uploads. In the context of the TiFlow project (which includes TiCDC and Data Migration components), developers have worked on improving storage sink implementations by utilizing multipart uploaders to handle large data files more efficiently, such as in AWS S3 or compatible storage systems. While the provided project documentation references issues and PRs related to optimizing multipart uploads in cloud storage sinks, these specific OSS API methods are standard components of Alibaba Cloud's SDK and API ecosystem for managing storage operations reliably. Citations:
🌐 Web query:
💡 Result: Alibaba Cloud Object Storage Service (OSS) provides two primary mechanisms for managing permissions: Resource Access Management (RAM) policies and Bucket Policies [1][2]. Both use JSON-formatted authorization policies to define access control [3]. RAM Policies (Identity-based) RAM policies are attached to RAM users, groups, or roles within your Alibaba Cloud account [4][1]. They are the primary method for controlling what actions these internal entities can perform on OSS resources [5]. - Usage: Ideal for managing permissions for employees, systems, or applications within your own account [5]. - Configuration: You attach custom or system policies (e.g., AliyunOSSFullAccess) to a RAM user or role [4][6]. - Granularity: Supports fine-grained control, including directory-level permissions, by using prefix conditions in the policy [1]. Bucket Policies (Resource-based) Bucket policies are attached directly to a specific OSS bucket [2]. - Usage: Primarily used for cross-account access, granting anonymous access, or restricting access based on network conditions like IP addresses or Virtual Private Clouds (VPCs) [2]. - Configuration: The policy includes a Principal element that identifies the user, role, or account being authorized [7][3]. - Management: Can be managed via the OSS console, OSS SDKs, or tools like ossutil [8][7][9]. Policy Evaluation When a request is made, OSS evaluates all applicable policies [2]. The system follows a deny-first principle, meaning an explicit Deny rule takes precedence over any Allow rule [5][2]. If no explicit rule matches, the request is denied by default [2]. Key Policy Elements Authorization policies consist of the following core JSON elements [3]: - Effect: Either Allow or Deny [3]. - Action: The specific OSS operation (e.g., oss:GetObject, oss:PutObject) [1][3]. - Resource: The target OSS resource (e.g., acs:oss:::bucket-name/*) [3]. - Principal: The entity being granted or denied access (required for bucket policies) [7][3]. - Condition: Optional criteria that must be met for the policy to take effect, such as SourceIp or SourceVpc [2][3]. Citations:
🌐 Web query:
💡 Result: AccessKey secrets are designed to be displayed only once, at the time of creation, for security reasons [1][2][3]. They cannot be retrieved later [1][4][5]. If you lose or misplace your secret access key, you must create a new AccessKey pair [1]. To rotate an AccessKey pair properly without causing service disruption: 1. Create a new AccessKey pair for the user [6]. 2. Update your applications or services to use the newly created AccessKey pair [6]. 3. Verify that your applications are functioning correctly using the new credentials [6]. You can check the last used time/service in the management console to confirm the old key is no longer active before disabling it [6]. 4. Disable the old AccessKey pair [6][5]. 5. After confirming that everything is working as expected, delete the old AccessKey pair [6][5]. As a security best practice, it is recommended to rotate AccessKey pairs regularly (e.g., every 90 days) [6][7] and, whenever possible, use temporary credentials (such as Security Token Service/STS tokens) instead of long-term AccessKey pairs to minimize risk [3][8][7]. Citations:
Scope OSS permissions and manage the AccessKey Secret. Require a custom RAM policy. Scope object actions to the changefeed bucket prefix and restrict Sources: Path instructions, MCP tools |
||
|
|
||
| The following JSON example shows a policy with the required permissions. Replace `<Your bucket name>` with the name of your OSS bucket. | ||
|
|
||
| ```json | ||
| { | ||
| "Version": "1", | ||
| "Statement": [ | ||
| { | ||
| "Effect": "Allow", | ||
| "Action": [ | ||
| "oss:ListObjects", | ||
| "oss:GetObject", | ||
| "oss:PutObject", | ||
| "oss:DeleteObject" | ||
| ], | ||
| "Resource": [ | ||
| "acs:oss:*:*:<Your bucket name>", | ||
| "acs:oss:*:*:<Your bucket name>/*" | ||
| ] | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| 2. On the **Destination** page for Alibaba Cloud OSS, fill in the following fields: | ||
|
|
||
| - **Bucket URI**: enter the OSS URI in the format `oss://<BucketName>/<prefix>/`. | ||
| - **Access Key ID**: enter the AccessKey ID from the RAM user. | ||
| - **Access Key Secret**: enter the AccessKey Secret from the RAM user. | ||
|
|
||
| </div> | ||
|
qiancai marked this conversation as resolved.
|
||
|
|
||
| </CustomContent> | ||
|
|
||
| </SimpleTab> | ||
|
|
||
| Click **Next** to establish the connection from the TiDB Cloud Dedicated cluster to Amazon S3, GCS, or Azure Blob Storage. TiDB Cloud will automatically test and verify if the connection is successful. | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.