|
| 1 | +# Push data or event logs from Logstash into an initialized OpenSearch Cluster |
| 2 | + |
| 3 | +## Prerequisites |
| 4 | + |
| 5 | +Suppose you have successfully initialized an OpenSearch Cluster with the following parameters:  |
| 6 | + |
| 7 | +<figure><img src="../../../.gitbook/assets/opensearch5.png" alt=""><figcaption></figcaption></figure> |
| 8 | + |
| 9 | +Next, proceed to push sample data into OpenSearch Dashboards or push event logs from Logstash into OpenSearch. |
| 10 | + |
| 11 | +## **Push sample data into OpenSearch Dashboards** |
| 12 | + |
| 13 | +Sample data will help you get familiar with OpenSearch Dashboards and test data visualization capabilities. |
| 14 | + |
| 15 | +#### **Step 1: Download sample data** |
| 16 | + |
| 17 | +Run the following command to download sample data: |
| 18 | + |
| 19 | +```bash |
| 20 | +curl -O https://raw.githubusercontent.com/opensearch-project/documentation-website/2.19/assets/examples/ecommerce-field_mappings.json |
| 21 | +curl -O https://raw.githubusercontent.com/opensearch-project/documentation-website/2.19/assets/examples/ecommerce.ndjson |
| 22 | +``` |
| 23 | + |
| 24 | +#### **Step 2: Create index and push data into OpenSearch** |
| 25 | + |
| 26 | +Run the following command to create an index and push data to OpenSearch: |
| 27 | + |
| 28 | +```bash |
| 29 | +# 2. Create index and data. |
| 30 | +curl -H "Content-Type: application/json" -X PUT "https://<<OpenSearch_ReceiveLogs_Endpoint>>/ecommerce" -k -H "Authorization: Basic $(echo -n 'master-user:<<Master_User_Password>>' | base64)" --data-binary "@ecommerce-field_mappings.json" |
| 31 | +curl -H "Content-Type: application/json" -X PUT "https://<<OpenSearch_ReceiveLogs_Endpoint>>/ecommerce/_bulk" -k -H "Authorization: Basic $(echo -n 'master-user:<<Master_User_Password>>' | base64)" --data-binary "@ecommerce.ndjson" |
| 32 | +``` |
| 33 | + |
| 34 | +You can get the `OpenSearch_ReceiveLogs_Endpoint` information from the vDB Portal and replace `<<Master_User_Password>>` with the master account password you previously created. |
| 35 | + |
| 36 | +Example:  |
| 37 | + |
| 38 | +```bash |
| 39 | +# 2. Create index and data. |
| 40 | +curl -H "Content-Type: application/json" -X PUT "https://open-search-dem-53461-5cfxl-hcm03.vdb-opensearch.vngcloud.vn:9200/ecommerce" -k -H "Authorization: Basic $(echo -n 'master-user:123456789aA@' | base64)" --data-binary "@ecommerce-field_mappings.json" |
| 41 | +curl -H "Content-Type: application/json" -X PUT "https://open-search-dem-53461-5cfxl-hcm03.vdb-opensearch.vngcloud.vn:9200/ecommerce/_bulk" -k -H "Authorization: Basic $(echo -n 'master-user:123456789aA@' | base64)" --data-binary "@ecommerce.ndjson" |
| 42 | +``` |
| 43 | + |
| 44 | +[\ |
| 45 | +](https://liemnt5-cidr-11430-2ue3z-hcm03.vdb-opensearch.vngcloud.tech)The result will display as follows:  |
| 46 | + |
| 47 | +```bash |
| 48 | +curl -H "Content-Type: application/json" -X PUT "https://open-search-dem-53461-5cfxl-hcm03.vdb-opensearch.vngcloud.vn:9200/ecommerce" -k -H "Authorization: Basic $(echo -n 'master-user:123456789aA@' | base64)" --data-binary "@ecommerce-field_mappings.json" |
| 49 | +{"acknowledged":true,"shards_acknowledged":true,"index":"ecommerce"} |
| 50 | + |
| 51 | +curl -H "Content-Type: application/json" -X PUT "https://open-search-dem-53461-5cfxl-hcm03.vdb-opensearch.vngcloud.vn:9200/ecommerce/_bulk" -k -H "Authorization: Basic $(echo -n 'master-user:123456789aA@' | base64)" --data-binary "@ecommerce.ndjson" |
| 52 | +{"took":4579,"errors":false,"items":[{"index":{"_index":"ecommerce","_id":"0","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":0,"_primary_term":1,"status":201}},{"index":{"_index":"ecommerce","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":1,"_primary_term":1,"status":201}},{"index":{"_index":"ecommerce","_id":"2","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":2,"_primary_term":1,"status":201}},{"index":{"_index":"ecommerce","_id":"3","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":3,"_primary_term":1,"status":201}},{"index":{"_index":"ecommerce","_id":"4","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":4,"_primary_term":1,"status":201}},{"index":{"_index":"ecommerce","_id":"5","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":5,"_primary_term":1,"status":201}},{"index":{"_index":"ecommerce","_id":"6","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":6,"_primary_term":1,"status":201}},{"index":{"_index":"ecommerce","_id":"7","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":7,"_primary_term":1,"status":201}},{"index":{"_index":"ecommerce","_id":"8","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":8,"_primary_term":1,"status":2... |
| 53 | +.... |
| 54 | +``` |
| 55 | + |
| 56 | +**Step 3: Check data on OpenSearch Dashboards** |
| 57 | + |
| 58 | +1. Access and log in to **OpenSearch Dashboards** |
| 59 | +2. Go to **Management**, select **Dashboard Management**  |
| 60 | + |
| 61 | +<figure><img src="../../../.gitbook/assets/image (7) (1) (1) (1) (1) (1) (1) (1) (1) (1) (1).png" alt=""><figcaption></figcaption></figure> |
| 62 | + |
| 63 | +3. Select **Index patterns**, then select **Create index pattern** |
| 64 | + |
| 65 | +<figure><img src="../../../.gitbook/assets/image (8) (1) (1) (1) (1) (1) (1) (1) (1).png" alt=""><figcaption></figcaption></figure> |
| 66 | + |
| 67 | +4. Enter **Index pattern name**, for example `ecommerce*` then select **Next step** |
| 68 | + |
| 69 | +<figure><img src="../../../.gitbook/assets/image (5) (1) (1) (1) (1) (1) (1) (1) (1) (1) (1) (1).png" alt=""><figcaption></figcaption></figure> |
| 70 | + |
| 71 | +5. Access **Discover** to view the sample data. |
| 72 | + |
| 73 | +<figure><img src="../../../.gitbook/assets/image (6) (1) (1) (1) (1) (1) (1) (1) (1) (1) (1).png" alt=""><figcaption></figcaption></figure> |
| 74 | + |
| 75 | +## **Push event logs from Logstash into OpenSearch** |
| 76 | + |
| 77 | +If you want to collect and push event logs from Logstash into OpenSearch, follow these steps: |
| 78 | + |
| 79 | +#### **Step 1: Install Logstash (if not already installed)** |
| 80 | + |
| 81 | +On Ubuntu/Debian: |
| 82 | + |
| 83 | +```bash |
| 84 | +sudo apt update && sudo apt install logstash |
| 85 | +``` |
| 86 | + |
| 87 | +On CentOS/RHEL: |
| 88 | + |
| 89 | +```bash |
| 90 | +sudo yum install logstash |
| 91 | +``` |
| 92 | + |
| 93 | +#### **Step 2: Configure Logstash to send logs to OpenSearch** |
| 94 | + |
| 95 | +Create a configuration file for Logstash, for example: |
| 96 | + |
| 97 | +```bash |
| 98 | +sudo nano /etc/logstash/conf.d/logstash.conf |
| 99 | +``` |
| 100 | + |
| 101 | +Add the following content: |
| 102 | + |
| 103 | +```editorconfig |
| 104 | +input { |
| 105 | + file { |
| 106 | + path => "/var/log/syslog" |
| 107 | + start_position => "beginning" |
| 108 | + sincedb_path => "/dev/null" |
| 109 | + } |
| 110 | +} |
| 111 | +
|
| 112 | +filter { |
| 113 | + mutate { |
| 114 | + add_field => { "host" => "%{host}" } |
| 115 | + } |
| 116 | +} |
| 117 | +
|
| 118 | +output { |
| 119 | + opensearch { |
| 120 | + hosts => ["OpenSearch_ReceiveLogs_Endpoint"] |
| 121 | + index => "logstash-logs" |
| 122 | + user => "master-user" |
| 123 | + password => "Your_MasterUser_Password" |
| 124 | + ssl => false |
| 125 | + } |
| 126 | +} |
| 127 | +``` |
| 128 | + |
| 129 | +Replace: |
| 130 | + |
| 131 | +* `OpenSearch_ReceiveLogs_Endpoint` with your OpenSearch Receive Logs Endpoint from the vDB Portal. |
| 132 | +* `Your_MasterUser_Password` with the master account password you previously created. |
| 133 | + |
| 134 | +#### **Step 3: Start Logstash** |
| 135 | + |
| 136 | +After configuration is complete, start Logstash to begin sending logs: |
| 137 | + |
| 138 | +```bash |
| 139 | +sudo systemctl start logstash |
| 140 | +``` |
| 141 | + |
| 142 | +#### **Step 4: Check data in OpenSearch** |
| 143 | + |
| 144 | +You can check logs using the OpenSearch API: |
| 145 | + |
| 146 | +```bash |
| 147 | +curl -X GET "https://OpenSearch_ReceiveLogs_Endpoint/logstash-logs/_search?pretty" -k -H "Authorization: Basic $(echo -n 'master-user:Your_MasterUser_Password' | base64)" |
| 148 | +``` |
| 149 | + |
| 150 | +Example: |
| 151 | + |
| 152 | +```bash |
| 153 | +curl -X GET "https://open-search-dem-53461-5cfxl-hcm03.vdb-opensearch.vngcloud.vn:9200/logstash-logs/_search?pretty" -k -H "Authorization: Basic $(echo -n 'master-user:123456789aA@' | base64)" |
| 154 | +``` |
| 155 | + |
| 156 | +If logs appear, it means Logstash has successfully sent data to OpenSearch. |
| 157 | + |
| 158 | +#### **Step 5: View logs on OpenSearch Dashboards** |
| 159 | + |
| 160 | +1. Access and log in to **OpenSearch Dashboards** |
| 161 | +2. Go to **Management**, select **Dashboard Management**  |
| 162 | + |
| 163 | +<figure><img src="../../../.gitbook/assets/image (7) (1) (1) (1) (1) (1) (1) (1) (1) (1) (1).png" alt=""><figcaption></figcaption></figure> |
| 164 | + |
| 165 | +3. Select **Index patterns**, then select **Create index pattern** |
| 166 | + |
| 167 | +<figure><img src="../../../.gitbook/assets/image (8) (1) (1) (1) (1) (1) (1) (1) (1).png" alt=""><figcaption></figcaption></figure> |
| 168 | + |
| 169 | +4. Enter **Index pattern name**, for example `logstash-logs*` then select **Next step** |
| 170 | + |
| 171 | +<figure><img src="../../../.gitbook/assets/image (9) (1) (1) (1) (1) (1) (1).png" alt=""><figcaption></figcaption></figure> |
| 172 | + |
| 173 | +5. Finally, access **Discover** to view the logs. |
0 commit comments