enhancement: add option to enrich user info for Kolide.device integration

[hnguyen-coreweave]

Proposed commit message

Enhancement: add optional user enrichment for the Kolide device data stream.

This change introduces an opt-in setting, enable_user_enrichment, for the kolide.device CEL stream. When enabled, the CEL program uses registered_owner_info.link from each /devices record to request additional owner details from Kolide's /people/<id> endpoint and merges that response into registered_owner_info before publishing the event.

The ingest pipeline is updated to map enriched owner fields into ECS where present:

• json.registered_owner_info.email -> user.email
• json.registered_owner_info.name -> user.name

If enrichment is disabled (default) or the owner lookup fails, processing falls back to the original device payload. Package version is bumped to 0.1.3, with a matching changelog entry.

With the additional user information, elastic users now can use this kolide event stream to generate a lookup table/index that can help lookup the username to devices and vice versa

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• Added enable_user_enrichment stream variable in packages/kolide/data_stream/device/manifest.yml.
• Updated CEL program in packages/kolide/data_stream/device/agent/stream/cel.yml.hbs to perform conditional owner enrichment.
• Updated ingest mapping in packages/kolide/data_stream/device/elasticsearch/ingest_pipeline/default.yml to set user.email and user.name.
• Bumped integration version to 0.1.3 in packages/kolide/manifest.yml.
• Added changelog entry in packages/kolide/changelog.yml.
• Updated pipeline test fixtures and expected output for enriched owner payload handling.

How to test this PR locally

1. From the repository root, run:
   • elastic-package format
   • elastic-package test pipeline -v --data-streams device --package kolide
2. Validate that existing device sample events still parse correctly when enrichment fields are absent.
3. Validate that enriched sample events populate:
   • kolide.device.registered_owner_info.*
   • user.email
   • user.name
4. In package policy, toggle enable_user_enrichment:
   • false: no extra owner lookup requests are made.
   • true: owner data is merged when registered_owner_info.link is present.

Related issues

• Relates chore(deps): bump github.com/elastic/elastic-package from 0.125.0 to 0.125.1 #19960

Screenshots

N/A

[cla-checker-service]

💚 CLA has been signed

[elastic-vault-github-plugin-prod]

Reviewers

Buildkite won't run for external contributors automatically; you need to add a comment:

• /test : will kick off a build in Buildkite.

NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details.

[hnguyen-coreweave]

/test

[jamiehynds]

@vinit-chauhan would you mind reviewing this one please? Thanks for the contribution @hnguyen-coreweave!

[infra-vault-gh-plugin-prod]

Pinging @elastic/integration-experience (Team:Integration-Experience)

[vinit-chauhan]

Thanks @hnguyen-coreweave for the PR!

A couple of things came up on my mind looking at the PR:

• The dedup fingerprint only strips last_seen_at from event.original, but the enrichment merge now embeds volatile person fields (like last_authenticated_at) into it too. Since those can change independently of the device, this could produce duplicate docs on every poll even when the device itself didn't change.
• It's an extra API call per device with no caching by owner — could add up on larger fleets, and if the enrichment call fails (non-200), it silently falls back with no way to tell it happened.
• The new pipeline test fixture already has pre-merged owner data, so the actual do_request()/merge logic in cel.yml.hbs isn't really exercised by a test. In order to test it, we would need to add/update the system tests for device data-stream.

There's also an open PR (#20108) that adds a dedicated people data stream, which collects the same user detail independently without the per-device call pattern or the fingerprint issue above — worth checking if that covers what you need here.

If you still want owner details attached directly to device events, that's doable separately — Elasticsearch's enrich processor can join people data onto device docs at ingest time via a custom pipeline (logs-kolide.device@custom), keyed on the owner id.

[vera-review-bot]

No issues across the latest commits dbb156c.

Review summary

Issues found across earlier commits 8b5206c — 3 medium

• 🟡 The 0.1.3 changelog entry links to the wrong PR (#​19960, the 0.1.2 bugfix) instead of this PR (link) (Resolved)
• 🟡 Enrichment merges the whole /people response into the event, so volatile person fields enter the dedup fingerprint and cause duplicate device documents (link) (Resolved)
• 🟡 Changelog links a different PR number (link) (Resolved)

Since this is a community PR, a new commit triggers another review — at most once every 30 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[hnguyen-coreweave]

li

Thanks Vinit for reviewing the PR.

It's an extra API call per device with no caching by owner — could add up on larger fleets, and if the enrichment call fails (non-200), it silently falls back with no way to tell it happened.

this point is valid and was the biggest item that made me hesitant to submit the PR. But this is what got me thinking of making this an option that user can chose to opt-in/out

Good call on the slient fallback tho 100%

The new pipeline test fixture already has pre-merged owner data, so the actual do_request()/merge logic in cel.yml.hbs isn't really exercised by a test. In order to test it, we would need to add/update the system tests for device data-stream.

I might need help on this if we decide to pursue this route

The _enrich processor is certainly an option but the enrich policy needs to be manually called to get refreshed, unless things have changed since the last time I used it a few years ago. I wonder if there is something that can help ease the pain of having to manually refresh the enrich policy?

[vera-review-bot]

⚠️ @hnguyen-coreweave I couldn't complete the review for commits dacce0f…2216378 (5 commits) after several attempts. A maintainer can re-run it with @vera-review-bot review.

[vinit-chauhan]

Hey @hnguyen-coreweave, thanks for your thoughtful response.

Regarding the enrichment policy refresh, you could create a transform on the people index (introduced in v0.2.0) that maintains the latest state and updates on a set interval, such as every 5 minutes. Additionally, you could implement a watcher that triggers a POST to /_enrich/policy//_execute on a schedule to automatically rebuild the enrichment snapshot.

[vera-review-bot]

@hnguyen-coreweave - This PR has merge conflicts with the base branch. Please resolve them.

[hnguyen-coreweave]

Vinit, That is fair. I will wait for you PR to be merged then and follow the recomended pattern above. Thanks for looking at the PR.

Also I am talking to kolide directly and see if they can bake in the user information in the /devices API so nobody has to come up with some hacky solutions