LOG-8697: Use event.lastTimestamp instead of metadata.creationTimestamp for Kubernetes event logs by vparfonov · Pull Request #3226 · openshift/cluster-logging-operator

vparfonov · 2026-03-19T09:24:11Z

Description

The currently uses metadata.creationTimestamp as the log event timestamp for Kubernetes Events, but Kubernetes updates existing Event objects when the same event repeats, instead of creating new ones.
In this case:

metadata.creationTimestamp remains the original time the Event object was created
lastTimestamp is updated to reflect the most recent occurrence

In long-running clusters, this means an Event object may have a creationTimestamp that is weeks/months old, even though the event is still actively occurring. When Vector sends such records to Loki using creationTimestamp as the log timestamp, Loki rejects them with: greater_than_max_sample_age.

This PR address to use event.lastTimestamp as the log timestamp for EventRouter events instead of metadata.creationTimestamp.
The logic to include a fallback chain:
lastTimestamp -> firstTimestamp -> eventTime -> creationTimestamp

/cc @Clee2691 @cahartma
/assign @jcantrill

Links

Depending on PR(s):
GitHub issue:
JIRA: https://redhat.atlassian.net/browse/LOG-8697
Enhancement proposal:

openshift-ci-robot · 2026-03-21T05:02:40Z

@vparfonov: This pull request references LOG-8697 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.8.0" version, but no target version was set.

Details

In response to this:

Description

The currently uses metadata.creationTimestamp as the log event timestamp for Kubernetes Events, but Kubernetes updates existing Event objects when the same event repeats, instead of creating new ones.
In this case:

metadata.creationTimestamp remains the original time the Event object was created

lastTimestamp is updated to reflect the most recent occurrence

In long-running clusters, this means an Event object may have a creationTimestamp that is weeks/months old, even though the event is still actively occurring. When Vector sends such records to Loki using creationTimestamp as the log timestamp, Loki rejects them with: greater_than_max_sample_age.

This PR address to use event.lastTimestamp as the log timestamp for EventRouter events instead of metadata.creationTimestamp.

/cc @Clee2691 @cahartma
/assign @jcantrill

Links

Depending on PR(s):

GitHub issue:

JIRA: https://redhat.atlassian.net/browse/LOG-8697

Enhancement proposal:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

vparfonov · 2026-03-21T05:02:59Z

/retest

jcantrill · 2026-03-23T18:32:33Z

 		}

-		ExpectedLogTemplateBuilder = func(event, oldEvent *corev1.Event) types.EventRouterLog {
+		ExpectedLogTemplateBuilder = func(event, oldEvent *corev1.Event, outputType obs.OutputType) types.EventRouterLog {


Do we know of any case where first/last timestamp would be nil for which we should test if we get the creation timestamp? I see creation time default to epoc so I'm going to assume it will always be something, and hopefully not epoc. This otherwise LGTM

Great question! Investigating this actually led me to the better way to handle these checks.
Kubernetes has two Event APIs. The old style (v1 core) uses firstTimestamp/lastTimestamp, while the new style (events.k8s.io/v1) uses eventTime for better precision and scaling.

eventrouter watches the v1.Event objects, it usually sees lastTimestamp. However, if a component has migrated to the new API internally, the API server has to 'downgrade' the event for us. During that conversion, lastTimestamp can end up being zero if the event is part of a series.

I’ve updated the logic to include a fallback chain:
lastTimestamp -> firstTimestamp -> eventTime -> creationTimestamp

I also added a check for the zero timestamp (0001-01-01...).

jcantrill · 2026-03-23T18:33:00Z

/hold

jcantrill · 2026-03-23T18:33:06Z

/approve
/lgtm

jcantrill · 2026-03-23T18:33:36Z

@vparfonov I defer to you to release the hold. I believe the coverage is OK, but wanted to check with you first

openshift-ci · 2026-03-23T18:37:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jcantrill, vparfonov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [jcantrill]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@timestamp

…ernetes event logs EventRouter updates existing Event objects instead of creating new ones. The original metadata.creationTimestamp may therefore be days/weeks old while lastTimestamp reflects the most recent occurrence. Using creationTimestamp caused Loki (others sinks possibly) to reject records with "greater_than_max_sample_age". Use lastTimestamp for @timestamp and timestamp fields so event updates are ingested with the correct time. The fallback chain is: lastTimestamp -> firstTimestamp -> eventTime -> creationTimestamp

openshift-ci-robot · 2026-03-23T19:45:03Z

@vparfonov: This pull request references LOG-8697 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.8.0" version, but no target version was set.

Details

In response to this:

Description

The currently uses metadata.creationTimestamp as the log event timestamp for Kubernetes Events, but Kubernetes updates existing Event objects when the same event repeats, instead of creating new ones.
In this case:

metadata.creationTimestamp remains the original time the Event object was created

lastTimestamp is updated to reflect the most recent occurrence

In long-running clusters, this means an Event object may have a creationTimestamp that is weeks/months old, even though the event is still actively occurring. When Vector sends such records to Loki using creationTimestamp as the log timestamp, Loki rejects them with: greater_than_max_sample_age.

This PR address to use event.lastTimestamp as the log timestamp for EventRouter events instead of metadata.creationTimestamp.
The logic to include a fallback chain:
lastTimestamp -> firstTimestamp -> eventTime -> creationTimestamp

/cc @Clee2691 @cahartma
/assign @jcantrill

Links

Depending on PR(s):

GitHub issue:

JIRA: https://redhat.atlassian.net/browse/LOG-8697

Enhancement proposal:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

jcantrill · 2026-04-02T19:33:20Z

/hold cancel

jcantrill · 2026-04-02T19:33:25Z

/lgtm

openshift-merge-bot · 2026-04-02T20:22:18Z

/retest-required

Remaining retests: 0 against base HEAD fd8d47d and 2 for PR HEAD 1eee056 in total

openshift-ci · 2026-04-02T22:33:27Z

@vparfonov: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot requested review from Clee2691 and alanconway March 19, 2026 09:25

vparfonov changed the title ~~Use event.lastTimestamp instead of metadata.creationTimestamp for Kubernetes event logs~~ LOG-8697: Use event.lastTimestamp instead of metadata.creationTimestamp for Kubernetes event logs Mar 21, 2026

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 21, 2026

openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 21, 2026

vparfonov force-pushed the log8697 branch from c404caa to 8bd75b6 Compare March 21, 2026 14:03

openshift-ci bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 21, 2026

jcantrill reviewed Mar 23, 2026

View reviewed changes

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 23, 2026

openshift-ci bot assigned jcantrill Mar 23, 2026

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Mar 23, 2026

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 23, 2026

vparfonov force-pushed the log8697 branch from 8bd75b6 to 1eee056 Compare March 23, 2026 19:43

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Mar 23, 2026

vparfonov requested a review from jcantrill March 23, 2026 19:45

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 2, 2026

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Apr 2, 2026

openshift-merge-bot bot merged commit a85c907 into openshift:master Apr 2, 2026
7 checks passed

vparfonov deleted the log8697 branch April 3, 2026 04:04

vparfonov mentioned this pull request Apr 3, 2026

[release-6.5]LOG-8697:Use event.lastTimestamp instead of metadata.creationTimestamp for Kubernetes #3244

Open

Conversation

vparfonov commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Links

Uh oh!

openshift-ci-robot commented Mar 21, 2026 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Links

Uh oh!

vparfonov commented Mar 21, 2026

Uh oh!

jcantrill Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

vparfonov Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

jcantrill commented Mar 23, 2026

Uh oh!

jcantrill commented Mar 23, 2026

Uh oh!

jcantrill commented Mar 23, 2026

Uh oh!

openshift-ci bot commented Mar 23, 2026

Uh oh!

openshift-ci-robot commented Mar 23, 2026 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Links

Uh oh!

jcantrill commented Apr 2, 2026

Uh oh!

jcantrill commented Apr 2, 2026

Uh oh!

openshift-merge-bot bot commented Apr 2, 2026

Uh oh!

openshift-ci bot commented Apr 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vparfonov commented Mar 19, 2026 •

edited

Loading

openshift-ci-robot commented Mar 21, 2026 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Mar 23, 2026 •

edited by openshift-ci bot

Loading