Skip to content

Commit cad9a1e

Browse files
Hana-Hasaclaude
Hana-Hasa
and
claude
committed
update kafka 1
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 751a524 commit cad9a1e

7 files changed

Lines changed: 322 additions & 0 deletions

File tree

‎English/SUMMARY.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -742,6 +742,12 @@
742742
* [Manage MDS Instance](vdb/memorystore-database-service-mds/quan-ly-cau-hinh-mds-instance.md)
743743
* [Manage MDS Config Group](vdb/memorystore-database-service-mds/quan-ly-thong-tin-mds-instance.md)
744744
* [Backup MDS Instance](vdb/memorystore-database-service-mds/sao-luu-mds-instance.md)
745+
* [Kafka Cluster Database (KDS)](vdb/kafka-cluster-kds/README.md)
746+
* [How It Works](vdb/kafka-cluster-kds/cach-thuc-hoat-dong.md)
747+
* [Getting Started with Kafka Cluster](vdb/kafka-cluster-kds/bat-dau-voi-kafka-cluster.md)
748+
* [Manage Kafka Cluster](vdb/kafka-cluster-kds/quan-ly-kafka-cluster/README.md)
749+
* [Service Monitoring](vdb/kafka-cluster-kds/giam-sat-dich-vu.md)
750+
* [Pricing](vdb/kafka-cluster-kds/cach-tinh-phi.md)
745751
* [Billing Information](vdb/billing-information.md)
746752
* [Security (Bảo mật)](vdb/security-bao-mat.md)
747753
* [vCDN](vcdn/README.md)

‎English/vdb/kafka-cluster-kds/README.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Kafka Cluster Database (KDS)
2+
3+
Kafka Cluster DB is a new service on the vDB platform, providing a powerful and flexible Kafka server cluster to manage real-time event streaming. With Kafka Cluster DB, you can easily build large-scale data processing applications, messaging systems, and centralized logging with high scalability, data durability, and outstanding performance.
4+
5+
<figure><img src="../../.gitbook/assets/image (2) (1) (1) (1) (1) (1) (1) (1).png" alt=""><figcaption></figcaption></figure>
6+
7+
## Features <a href="#tinh-nang-moi" id="tinh-nang-moi"></a>
8+
9+
**Comprehensive Kafka Cluster Management**
10+
11+
* **Create cluster:** Easily initialize Kafka clusters with flexible configuration options.
12+
* **Adjust broker count:** Increase or decrease the number of brokers to meet data processing demands.
13+
* **Expand storage:** Increase storage capacity for brokers when needed.
14+
* **Edit configuration:** Customize Kafka parameters to optimize performance and reliability.
15+
* **Version**: Supported versions 3.6, 3.6.1, 3.7
16+
17+
**Security and Access Control**
18+
19+
* **Diverse access methods:** Supports mTLS, SASL and Public Accessibility.
20+
* **User management:** Create, authorize and delete Kafka users.
21+
* **Topic management:** Create, delete, modify topic configuration.
22+
* **Certificate update:** Generate new certificates when old certificates expire or are inaccessible.
23+
* **Data encryption features**: Encryption at rest (volume), Encryption within cluster (brokers to brokers), Encryption in transit (clients to brokers)
24+
25+
## Benefits <a href="#loi-ich-chinh" id="loi-ich-chinh"></a>
26+
27+
* **High performance:** Process large event streams with low latency.
28+
* **Scalability:** Easily scale clusters to meet growth demands.
29+
* **Data durability:** Ensure data integrity and recoverability.
30+
* **Security:** Flexible authentication mechanisms and detailed access control.
31+
* **Easy to use:** Intuitive and user-friendly management interface.
32+
33+
Comparison between Kafka Cluster DB Managed Service and Traditional Kafka Cluster (self-managed)
34+
35+
| **Criteria** | **Kafka Cluster DB Managed Service** | **Traditional Kafka Cluster** |
36+
| --------------------------- | -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
37+
| **Cluster management** | The service provider (GreenNode vDB) is responsible for managing, maintaining, upgrading and monitoring the Kafka cluster. | Users self-manage the entire Kafka cluster, including installation, configuration, maintenance, upgrades and monitoring. |
38+
| **Configuration and deployment** | Easy to configure and deploy through web interface or API. | Requires in-depth Kafka knowledge and system administration skills for installation and configuration. |
39+
| **Scaling** | Easy to scale by adding resources through the management interface. | Requires manual scaling process, which can be complex and time-consuming. |
40+
| **Monitoring and troubleshooting** | The service provider (GreenNode vDB) provides monitoring tools and troubleshooting support. | You self-monitor and troubleshoot, requiring expertise and experience. |
41+
| **Cost** | Typically lower long-term costs due to savings on infrastructure investment, operational costs and personnel costs. | Higher long-term costs due to the need to invest in infrastructure, operations and personnel. |
42+
| **Flexibility** | May be limited in customization and control compared to self-managed clusters. | Allows full customization and control of the Kafka cluster. |
43+
| **Suitable for** | Businesses that want to focus on application development and don't want to invest heavily in infrastructure management. | Businesses with strong technical teams that want full system control and can handle incidents themselves. |
44+
45+
&#x20;

‎English/vdb/kafka-cluster-kds/bat-dau-voi-kafka-cluster.md‎

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
# Getting Started with Kafka Cluster
2+
3+
With vDB Kafka Cluster, you can quickly deploy and manage Kafka clusters efficiently, focusing on application development and leveraging the power of real-time event streaming. Follow the guide below to get started with the Kafka Cluster service.
4+
5+
## Step 1: Access and Login
6+
7+
1. Open a web browser and access the vDB Kafka Cluster interface at: [https://vdb.console.vngcloud.vn/kafka/cluster](https://vdb.console.vngcloud.vn/kafka/cluster)
8+
2. If you already have an account, enter your login credentials.
9+
10+
Refer to the GreenNode login guide [here](../../identity-and-access-management-iam/cac-loai-dinh-danh-iam/tai-khoan-user-accounts/cach-dang-nhap-vao-vng-cloud.md).
11+
12+
## Step 2: Create a Kafka Cluster
13+
14+
1. Click the "Create Kafka Cluster" button.&#x20;
15+
16+
<figure><img src="../../.gitbook/assets/image (744).png" alt="" width="228"><figcaption></figcaption></figure>
17+
2. Fill in the following information:
18+
* **Name:** Name your Kafka cluster for easy identification and management.
19+
* **Kafka Version:** Choose the Kafka version that suits your needs. Different versions may have different features and performance.
20+
* **Flavor (CPU, RAM):** Select the hardware configuration (CPU and RAM) for brokers in the Kafka cluster. This configuration affects the processing capability and performance of the cluster.
21+
* **Number of Brokers:** Determine the number of brokers in the Kafka cluster. The number of brokers affects the fault tolerance and scalability of the cluster.
22+
* **Storage per Broker (IOPS and Capacity):** Choose the storage capacity and IOPS (Input/Output Operations Per Second) for each broker. This affects the data storage capacity and read/write performance of the cluster.
23+
* **Network (VPC, Subnet):** Select the virtual network (VPC) and subnet where the Kafka cluster will be deployed. When initialized, the Kafka cluster will be in private mode (accessible only from within the VPC). After successful initialization, you can enable public access if needed.
24+
* **Access Method Control (mTLS, SASL):** Choose the authentication and authorization method for clients connecting to the Kafka cluster. mTLS uses client and server certificates, SASL uses username and password.
25+
* **Encryption Mode:** Choose the data encryption mode. By default, encryption in transit (data encrypted when transmitted between client and broker) and within cluster (data encrypted when stored on brokers) are enabled.
26+
* **Config Group:** Select the configuration group to apply to the Kafka cluster. The configuration group contains detailed settings for Kafka operations.
27+
3. Click the "Create" button to start the Kafka cluster creation process.&#x20;
28+
29+
<figure><img src="../../.gitbook/assets/image (745).png" alt=""><figcaption></figcaption></figure>
30+
31+
## Step 3: Create a Topic
32+
33+
1. After the Kafka cluster is successfully created, access the management page of that Kafka cluster.
34+
2. Find and click the "Create Topic" section.
35+
36+
<figure><img src="../../.gitbook/assets/image (748).png" alt=""><figcaption></figcaption></figure>
37+
38+
3. Fill in the following information:
39+
40+
* **Name:** Name your topic. Note that this name cannot be changed after initialization.
41+
* **Partition Count:** Determine the number of partitions for the topic. The number of partitions affects the scalability and parallelism of the topic.
42+
* **Replication Factor:** Determine the number of copies of each partition. This affects the fault tolerance of the topic.
43+
* **Retention Hour/Byte:** Determine the maximum storage time or capacity for data in the topic.
44+
45+
4. Click the "Create" button to create the topic.
46+
47+
## Step 4: Create Kafka User and Assign Permissions
48+
49+
1. In the Kafka cluster management page, find and click the "Create Kafka User" section.
50+
51+
<figure><img src="../../.gitbook/assets/image (749).png" alt=""><figcaption></figcaption></figure>
52+
53+
2. Fill in the following information:
54+
55+
* **Name:** Name the Kafka user.
56+
* **Permissions:** In the permission section, click "Add Permission" to select Produce (write data to topic) and Consume (read data from topic) permissions for each topic this user needs to access.
57+
58+
<figure><img src="../../.gitbook/assets/image (750).png" alt=""><figcaption></figcaption></figure>
59+
60+
* **Access Method:** Choose the access method for the Kafka user (mTLS or SASL) depending on the method enabled for the Kafka cluster.
61+
62+
3. Click the "Create" button to create the Kafka user.
63+
64+
## Step 5: Connect to the Kafka Cluster
65+
66+
There are multiple ways to connect to the Kafka cluster. The following guide will introduce you to connecting to the Kafka cluster through a private client.
67+
68+
Note: The installation guide below is performed on an Ubuntu 22.04 client server.
69+
70+
1. Initialize a Server. See detailed server initialization guide [here](../../vserver/compute-hcm03-1a/server/tao-may-chu-bang-bang-dieu-khien.md).
71+
2. Connect to the newly initialized server. See detailed guide [here.](../../vserver/compute-hcm03-1a/server/ket-noi-vao-may-chu-ao/)
72+
3. Install Java and necessary packages on the server with the following command:
73+
74+
```bash
75+
sudo apt-get update && sudo apt-get install default-jre tar unzip -y
76+
```
77+
78+
4. Next, download Apache Kafka with the following command:
79+
80+
```bash
81+
wget https://archive.apache.org/dist/kafka/{Your Kafka Cluster Version}/kafka_2.13-{Your Kafka Cluster Version}.tgz
82+
tar -xzf kafka_2.13-{YOUR MSK VERSION}.tgz
83+
```
84+
85+
Example
86+
87+
```bash
88+
wget https://archive.apache.org/dist/kafka/3.7.0/kafka_2.13-3.7.0.tgz
89+
tar -xzf kafka_2.13-3.7.0.tgz
90+
```
91+
92+
5. Next, download the TLS certificates to access the Kafka cluster.&#x20;
93+
94+
<figure><img src="../../.gitbook/assets/image (746).png" alt=""><figcaption></figcaption></figure>
95+
6. On the newly initialized server, upload and extract the TLS certificates and unzip using the command below:
96+
97+
Note: The User ID will be the directory name after extracting the downloaded certificate
98+
99+
```bash
100+
unzip vng-manage-key.zip
101+
cd {User ID}/mtls
102+
```
103+
104+
Example
105+
106+
```bash
107+
unzip vng-manage-key.zip
108+
cd user-4436a54a-feaa-4afd-bfe7-4bd3d2cae33a/mtls/
109+
```
110+
111+
7. Allow access to the Kafka cluster
112+
113+
You need to allow this server to access the Kafka cluster as a private client by allowing the IP. Follow the guide below:
114+
115+
* 7.1 View the Private IP of the newly initialized server, for example the server has private IP 10.255.0.5
116+
* 7.2 Allow this IP to access the Kafka cluster:
117+
* 7.2.1 Access the Kafka cluster management interface
118+
* 7.2.2 Navigate to **Connectivity & Security**, add a rule as shown below
119+
120+
Note: Port 9094 for mTLS and 9096 for SASL
121+
122+
<figure><img src="../../.gitbook/assets/image (751).png" alt=""><figcaption></figcaption></figure>
123+
124+
8. Produce message to Kafka topic
125+
126+
After allowing access from the client to the Kafka cluster, connect to the previously initialized client and execute the following command:
127+
128+
```bash
129+
{Path To Your Kafka Installation}/bin/kafka-console-producer.sh --bootstrap-server {Your Kafka Cluster Private endpoint} --producer.config config.properties --topic {Your Kafka Topic}
130+
```
131+
132+
Example
133+
134+
```bash
135+
/home/stackops/kafka_2.13-3.7.0/bin/kafka-console-producer.sh --bootstrap-server 10.5.0.6:9094,10.5.0.3:9094,10.5.0.5:9094 --producer.config config.properties --topic kafka-cluster-tutorial
136+
```
137+
138+
9. Consume message from Kafka topic
139+
140+
After allowing access from the client to the Kafka cluster, connect to the previously initialized client and execute the following command:
141+
142+
```bash
143+
{Path To Your Kafka Installation}/bin/kafka-console-consumer.sh --bootstrap-server {Your Kafka Cluster Private endpoint} --consumer.config config.properties --topic {Your Kafka Topic} --from-beginning
144+
```
145+
146+
Example
147+
148+
```bash
149+
/home/stackops/kafka_2.13-3.7.0/bin/kafka-console-consumer.sh --bootstrap-server 10.5.0.6:9094,10.5.0.3:9094,10.5.0.5:9094 --consumer.config config.properties --topic kafka-cluster-tutorial --from-beginning
150+
```
151+
152+
## Step 6: Delete Kafka Cluster
153+
154+
1. When you no longer need the Kafka cluster, access the Kafka cluster management page.
155+
2. Find and click the "Delete Kafka Cluster" button.
156+
3. Confirm the deletion of the Kafka cluster to avoid data loss and unnecessary costs.

‎English/vdb/kafka-cluster-kds/cach-thuc-hoat-dong.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# How It Works
2+
3+
A Kafka Cluster is a distributed system consisting of multiple Kafka servers (Kafka Brokers) working together to provide event streaming services. Data is organized into Topics, each topic is divided into multiple Partitions to increase scalability and fault tolerance.
4+
5+
## **Key Components and Terminology**
6+
7+
1. **Kafka Broker:**
8+
* A server in the Kafka cluster, responsible for storing and managing a portion of the topic data.
9+
* Receives data from Producers and provides data to Consumers.
10+
* Maintains cluster state and ensures data consistency.
11+
2. **Kafka Storage:**
12+
* The physical storage location for Kafka data, typically on hard drives or other storage devices.
13+
* Kafka uses an efficient storage structure for sequential reading and writing, achieving high performance.
14+
3. **Topic:**
15+
* A category or stream of related events.
16+
* Example: an e-commerce application might have topics like "new orders", "successful payments", "delivery completed".
17+
4. **Partition count:**
18+
* The number of partitions a topic is divided into.
19+
* Increasing partition count improves scalability and parallel processing, but also increases system complexity.
20+
5. **Replica Factor:**
21+
* The number of copies of each partition stored on different brokers.
22+
* Increasing the replication factor improves fault tolerance, but also increases required storage capacity.
23+
6. **Produce & Consume:**
24+
* **Produce:** The process of sending data (events) to a topic by Producers.
25+
* **Consume:** The process of reading data from a topic by Consumers.
26+
7. **Access method:**
27+
* **mTLS (Mutual TLS):** A two-way authentication mechanism requiring both client and server to authenticate each other using digital certificates. Ensures high security.
28+
* **SASL (Simple Authentication and Security Layer):** A flexible authentication mechanism supporting SCRAM authentication methods.
29+
8. **Encryption:**
30+
* **At Rest:** Encrypts data when stored on disk, protecting data from unauthorized access when the server is inactive.
31+
* **In Transit:** Encrypts data when transmitted between different components (client-broker), ensuring security during data transmission.
32+
* **Within Cluster:** Encrypts data when transmitted between brokers within the same Kafka cluster.
33+
34+
## **How It Works**
35+
36+
1. **Producers** send data to specific **topics**.
37+
2. Kafka distributes data into the **partitions** of the topic according to a partitioning strategy.
38+
3. Each partition is **replicated** across multiple brokers to ensure fault tolerance.
39+
4. **Consumers** subscribe to topics and read data from partitions.
40+
5. Kafka ensures that each consumer reads only one copy of each event, avoiding duplicate processing.
41+
42+
**In summary:**
43+
44+
Kafka Cluster provides a powerful and flexible platform for real-time event streaming. With high scalability, data durability and security, Kafka is an essential tool for building modern data applications.

‎English/vdb/kafka-cluster-kds/cach-tinh-phi.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Pricing
2+
3+
The pricing for vDB Kafka Cluster is based on the number of brokers in a Kafka cluster, where each broker includes the following components:
4+
5+
1. **Compute:** Computing cost based on the CPU and RAM configuration of each broker.
6+
2. **Storage:** Storage cost based on the disk capacity you allocate for each broker.
7+
3. **Encryption Volume (if applicable):** If you use the data encryption feature, there may be additional costs for encrypted storage capacity.
8+
9+
**Total pricing formula:**
10+
11+
```
12+
Total cost = (Cost per broker) x (Number of brokers)
13+
```
14+
15+
**Cost per broker:**
16+
17+
```
18+
Cost per broker = (Compute cost) + (Storage cost) + (Encryption volume cost, if applicable)
19+
```
20+
21+
**Example:**
22+
23+
Suppose you have a Kafka cluster with 3 brokers, each broker with the following configuration:
24+
25+
* Compute: 4 CPU, 16GB RAM
26+
* Storage: 100GB
27+
* Encryption Volume: Not used
28+
29+
And GreenNode has the following pricing:
30+
31+
* Compute (4 CPU, 16GB RAM): 1,500,000 VND/month
32+
* Storage: 2,000 VND/GB/month
33+
34+
**Cost calculation:**
35+
36+
* Compute cost per broker: 1,500,000 VND
37+
* Storage cost per broker: 100GB x 2,000 VND/GB/month = 200,000 VND
38+
* Cost per broker: 1,500,000 VND + 200,000 VND = 1,700,000 VND
39+
* Total cost: 1,700,000 VND/broker x 3 brokers = 5,100,000 VND/month
40+
41+
**Conclusion:**
42+
43+
This pricing model helps you easily estimate the cost of using vDB Kafka Cluster based on your needs. Choose the appropriate configuration and number of brokers to optimize cost and performance for your system.

‎English/vdb/kafka-cluster-kds/giam-sat-dich-vu.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Service Monitoring
2+

‎English/vdb/kafka-cluster-kds/quan-ly-kafka-cluster/README.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Manage Kafka Cluster
2+
3+
Kafka Cluster DB is not a traditional database. It is a cluster of Kafka servers working together to store and process real-time event streams. Kafka Cluster DB improves scalability, data durability and processing performance compared to using a single Kafka server.
4+
5+
**It is commonly used for:**
6+
7+
* Real-time data processing
8+
* Building messaging systems
9+
* Centralized logging
10+
11+
**Follow the articles below to learn more about using Kafka Cluster:**
12+
13+
* **Create Kafka Cluster:** To start using Kafka on the vDB platform, you need to create a Kafka cluster. This cluster will store and process your application's events.
14+
* **Scale Up/Down Broker Count:** Scale up when data traffic increases - adding more brokers helps distribute the load, ensuring system performance and fault tolerance. Conversely, scale down when data traffic decreases to save resources.
15+
* **Scale Up Broker Storage:** As data accumulates, increasing storage capacity ensures the Kafka cluster doesn't run out of space and continues to operate stably.
16+
* **Edit Config Group:** Kafka has many configuration parameters that affect performance, data durability, and other aspects. Editing the config group allows you to customize Kafka according to your application's specific requirements.
17+
* **Edit Access Method**: mTLS ensures higher security through two-way authentication between client and server; SASL provides flexible user authentication mechanisms.
18+
* **Public Accessibility:** Allows access from outside.
19+
* **Topic Management**: vDB Kafka Cluster provides topic management features such as create, delete, edit,...
20+
* **User Management:** vDB Kafka Cluster provides Kafka user management features such as create, delete, edit permissions, access method, re-Generate Certificate,...
21+
22+
#### Note:
23+
24+
* Always carefully check configurations before applying changes.
25+
* Back up important data before making major changes.
26+
* Refer to the detailed vDB Kafka Cluster documentation for more information.

0 commit comments

Comments
 (0)