[New Integration] Kafka OpenTelemetry Input Package

[zmoog]

Summary

• New OTel input package (kafka_input_otel) that uses the kafkareceiver to consume OTLP-encoded logs, metrics, and traces from Kafka-compatible brokers, including Azure Event Hub
• Supports optional per-signal topics (logs, metrics, traces) with dynamic_signal_types: true
• Authentication via SASL (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512) — covers Event Hub connection string and plain Kafka clusters
• Encoding limited to otlp_json / otlp_proto (EDOT kafkareceiver constraint)

This is a POC to explore using the OTel Kafka receiver as a transport layer for ingesting telemetry from Azure Event Hub.

Key design decisions

• Single package, multiple signals: Each signal type is optional — enabled by providing a topic name. One Event Hub per signal type.
• Consumer groups: Shared default ($Default) with optional per-signal overrides.
• Not Azure-specific: Works with any Kafka-compatible broker, though Event Hub is the primary use case.

Known limitations

• Only otlp_json and otlp_proto encodings (EDOT restriction)
• No OAUTHBEARER auth (Azure encoding extension not in EDOT yet)
• Policy tests skipped pending elastic-package multi-signal support (elastic/elastic-package#3298)

Test plan

• elastic-package lint passes
• elastic-package build succeeds
• Deploy with Elastic Agent + EDOT against an Azure Event Hub namespace
• Verify OTLP JSON logs flow through kafkareceiver → Elasticsearch
• Verify OTLP JSON metrics flow through kafkareceiver → Elasticsearch
• Verify OTLP JSON traces flow through kafkareceiver → Elasticsearch
• Test with only one signal type enabled (e.g., logs only)

🤖 Generated with Claude Code

[github-actions]

Vale Linting Results

Summary: 1 warning, 1 suggestion found

⚠️ Warnings (1)

File	Line	Rule	Message
packages/kafka_input_otel/docs/README.md	36	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.

💡 Suggestions (1)

File	Line	Rule	Message
packages/kafka_input_otel/docs/README.md	21	Elastic.Wordiness	Consider using 'because' instead of 'since'.

---

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

[macroscopeapp]

🟡 Medium input/input.yml.hbs:1

When none of logs_topic, metrics_topic, or traces_topic are configured, the template produces an invalid OpenTelemetry Collector configuration with empty receivers: and pipelines: sections. The Collector fails at startup because both sections require at least one entry. Consider adding a top-level guard (e.g., wrap everything in {{#if}} checking at least one topic) or making one topic required in the manifest.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file packages/kafka_input_otel/agent/input/input.yml.hbs around line 1:

When none of `logs_topic`, `metrics_topic`, or `traces_topic` are configured, the template produces an invalid OpenTelemetry Collector configuration with empty `receivers:` and `pipelines:` sections. The Collector fails at startup because both sections require at least one entry. Consider adding a top-level guard (e.g., wrap everything in `{{#if}}` checking at least one topic) or making one topic required in the manifest.

[macroscopeapp]

🟢 Low policy/test-default.expected:48

The transform/componentid-1 processor at lines 36-48 configures log_statements and metric_statements to set data_stream attributes, but is missing trace_statements. The traces pipeline at lines 129-136 uses this same transform processor, so traces pass through without having their data_stream.type, data_stream.dataset, and data_stream.namespace attributes set. This causes traces to be indexed without the required routing attributes, potentially resulting in documents not reaching the traces-*-* data stream.

-                - set(attributes["data_stream.namespace"], "ep")
+                - set(attributes["data_stream.namespace"], "ep")
+        trace_statements:
+            - context: span
+              statements:
+                - set(attributes["data_stream.type"], "traces")
+                - set(attributes["data_stream.dataset"], "kafkareceiver")
+                - set(attributes["data_stream.namespace"], "ep")

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file packages/kafka_input_otel/_dev/test/policy/test-default.expected around line 48:

The `transform/componentid-1` processor at lines 36-48 configures `log_statements` and `metric_statements` to set `data_stream` attributes, but is missing `trace_statements`. The traces pipeline at lines 129-136 uses this same transform processor, so traces pass through without having their `data_stream.type`, `data_stream.dataset`, and `data_stream.namespace` attributes set. This causes traces to be indexed without the required routing attributes, potentially resulting in documents not reaching the `traces-*-*` data stream.

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 908a051

Failed CI Steps

• ✅ Check go sources

History

• 💔 Build #41234 failed 73491dc

cc @zmoog