[kolide] Add GCS input and new data streams

[vinit-chauhan]

Proposed commit message

kolide: add GCS input and five data streams

Kolide's Log Pipeline can now be collected from Google Cloud Storage
in addition to AWS S3, so customers who route their exported logs
through GCS were previously unable to use this integration.

Add a `gcs` input across the audit, auth, device_check, osquery_result,
and osquery_status data streams, mirroring the existing `aws-s3` input
templates. Add `osquery_result` and `osquery_status` data streams to
cover the raw osquery logs in the Log Pipeline, which had no ingest
path at all before this change. Add `deprovisioned_person`, `people`,
and `request` data streams to fill remaining gaps in the REST API and
webhook event coverage, and route the two new `requests.*` webhook
event types to the new `request` data stream.

Also fix `@timestamp` derivation in the issues pipeline: it previously
used `detected_at` only, so a webhook-triggered re-fetch of an issue
that had since been resolved or blocked kept the document's timestamp
pinned to the original detection time. It now takes the max of
detected_at/resolved_at/blocks_device_at (capped at ingest time, since
blocks_device_at is a future enforcement deadline) and folds those
fields into the document `_id` fingerprint so lifecycle updates don't
collide on the same document.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• Confirm GCS system tests pass against the gcs-mock/gcs-emulator for audit, auth, device_check, osquery_result, and osquery_status.
• Confirm the new osquery_result/osquery_status AWS S3 and GCS ingest pipelines handle both the file formats present in real Kolide Log Pipeline exports.
• Verify the updated Kolide dashboard renders correctly with data from the new request and deprovisioned_person data streams.

How to test this PR locally

1. elastic-package check from packages/kolide.
2. For GCS-backed data streams (audit, auth, device_check, osquery_result, osquery_status):
   elastic-package test system --config-file _dev/test/system/test-gcs-config.yml -d <data_stream>
3. For AWS S3-backed data streams (osquery_result, osquery_status):
   elastic-package test system --config-file _dev/test/system/test-default-config.yml -d <data_stream>
4. Run pipeline tests for the new/changed data streams (deprovisioned_person, people, request, issues, osquery_result, osquery_status):
   elastic-package test pipeline -d <data_stream>
5. elastic-package build && elastic-package install and manually check the Kolide dashboard against sample data.

Related issues

• https://github.com/elastic/integration-experience/issues/1070

Screenshots

[github-actions]

Elastic Docs Style Checker (Vale)

Summary: 2 warnings, 8 suggestions found

⚠️ Warnings (2): Fix when the suggestion improves clarity or correctness.

File	Line	Rule	Message
packages/kolide/data_stream/deprovisioned_person/manifest.yml	13	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/kolide/data_stream/osquery_result/fields/fields.yml	8	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.

💡 Suggestions (8): Optional style improvements. Apply when helpful.

File	Line	Rule	Message
packages/kolide/_dev/build/docs/README.md	79	Elastic.WordChoice	Consider using 'efficient, basic' instead of 'Simple', unless the term is in the UI.
packages/kolide/_dev/build/docs/README.md	235	Elastic.WordChoice	Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI.
packages/kolide/_dev/build/docs/README.md	351	Elastic.Semicolons	Use semicolons judiciously.
packages/kolide/data_stream/deprovisioned_person/fields/fields.yml	13	Elastic.Wordiness	Consider using 'before' instead of 'prior to'.
packages/kolide/data_stream/osquery_result/manifest.yml	50	Elastic.Ellipses	In general, don't use an ellipsis.
packages/kolide/data_stream/osquery_result/manifest.yml	50	Elastic.Wordiness	Consider using 'all' instead of 'all of '.
packages/kolide/data_stream/osquery_status/manifest.yml	50	Elastic.Ellipses	In general, don't use an ellipsis.
packages/kolide/data_stream/osquery_status/manifest.yml	50	Elastic.Wordiness	Consider using 'all' instead of 'all of '.

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[github-actions]

TL;DR

The Check integrations kolide job fails because the checked-in generated sample for osquery_status does not match the system-test configuration used by CI. The committed packages/kolide/data_stream/osquery_status/sample_event.json is GCS-shaped, but the default system test is configured with input: aws-s3; CI regenerates an AWS sample and then reports the README as out of date.

Remediation

• Regenerate osquery_status/sample_event.json using the default AWS system test (elastic-package test system --config-file _dev/test/system/test-default-config.yml --generate), and do the same for the other data streams with multiple system-test configurations.
• Run elastic-package build from packages/kolide so docs/README.md reflects the regenerated samples.
• Re-run elastic-package check or the Buildkite package check.

Investigation details

Root Cause

At the failed commit 9da008d2dd77c25ecef513991cf24779dec5524e, packages/kolide/data_stream/osquery_status/_dev/test/system/test-default-config.yml declares input: aws-s3, while test-gcs-config.yml declares input: gcs. The committed packages/kolide/data_stream/osquery_status/sample_event.json contains GCS-specific output (gcs.storage, input.type: gcs, and a `(redacted) log path). The system-test-generated README diff instead contains different transport/output data and regenerated volatile values.

Evidence

• Build: https://buildkite.com/elastic/integrations/builds/45962
• Job/step: Check integrations kolide
• Build log: Error: checking package failed: checking readme files are up-to-date failed: files do not match
• The same diff shows changed data_stream.namespace, elastic_agent.id, elastic_agent.version, event.ingested, and event.original, confirming that CI regenerated the sample rather than finding a source-template syntax error.

Verification

Not run; this workflow is read-only and cannot modify the PR or rerun Buildkite.

Follow-up

The PR head advanced to 433254a7172f8dc34f5f80748a370743625d1fa5 after build 45962; rerun CI after regenerating the samples to confirm the current head.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[infra-vault-gh-plugin-prod]

Pinging @elastic/integration-experience (Team:Integration-Experience)

[ilyannn]

🤖 Comment produced by Cursor/GPT-5.6 Sol under my supervision

Requesting changes for the ten actionable findings identified in the integration review. Local validation passed lint, check, and all 38 pipeline checks; the comments focus on runtime correctness, upgrade behavior, test coverage, documentation, mappings, and fixture sanitization.

[vera-review-bot]

No issues across the latest commits 0e7f104.

Review summary

Issues found across earlier commits eaa7d2e — 1 high, 1 medium, 1 low

• 🟠 docs/README.md drops the auto-generated NOTICE header, so the committed file is out of sync with elastic-package build and the docs-consistency CI check will fail. Restore the two NOTICE comment lines at the top. (link) (Unresolved)
• 🟡 The people data stream description claims a person-id-based _id that overwrites unchanged records, but the pipeline actually uses a content fingerprint (event.original minus last_authenticated_at). Reword the description to match the implementation. (link) (Unresolved)
• 🔵 README intro references a non-existent field kolide.osquery_result.columns (link) (Unresolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[ilyannn]

Looks good, though i still had some questions about the _id fingerprinting and whether it's a good idea to change it – can you talk to someone else who knows Kolide?

Also please re-check the proposed commit message, it seems outdated.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[ilyannn]

After a call, duplication should not be an issue because the queries and dashboards should group by issue id.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 7612d70

History

• 💚 Build #46080 succeeded 53d8385
• 💚 Build #46032 succeeded 0e7f104
• 💚 Build #46031 succeeded eaa7d2e
• 💚 Build #45969 succeeded 50fccb7
• 💔 Build #45967 failed a487941
• 💔 Build #45964 failed 448bfe6

cc @vinit-chauhan

[elastic-vault-github-plugin-prod]

Package kolide - 0.2.0 containing this change is available at https://epr.elastic.co/package/kolide/0.2.0/