Backend Stopped Processing Data After Memory Increase – High CPU Usage (>500%) on event-processor-manager, event-processor-worker

[karlbrown87]

Acknowledgements

• I have searched (https://github.com/utmstack/UTMStack/issues) for past instances of this issue
• I have verified that my UTMStack version is up-to-date

Describe the bug

After increasing memory limits for the following Docker Swarm services:

• utmstack_backend
• utmstack_opensearch
• utmstack_web-pdf
  the backend stopped processing incoming data.
  No logs or events have been ingested or processed in the last 12 hours.
  During investigation, we observed abnormal CPU utilization on the following services:
• event-processor-manager
• event-processor-worker (if applicable, adjust if needed)
  Both services were consistently consuming over 500% CPU

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

• Backend should continue processing and indexing data after memory reallocation
• No interruption in log ingestion pipeline

Current Behavior

• No data ingestion in the last 12 hours
• No new logs indexed in OpenSearch
• Backend API responsive but not processing events
• event-processor-manager/worker CPU usage > 500%
• System load significantly elevated
• No automatic recovery observed

Reproduction Steps

docker service update utmstack_backend --limit-memory 4096M --reserve-memory 4096M
docker service update utmstack_node1 --limit-memory 10240M --reserve-memory 10240M
docker service update utmstack_web-pdf --limit-memory 4096M --reserve-memory 4096M

Possible Solution

No response

Additional Information/Context

No response

UTMStack Version

11.2.3

Operating System and version

Ubuntu 24.04

Hypervisor and Version | Server Vendor and Model

AWS

Browser and version

Chrome

[karlbrown87]

Backend Error: Aggregate.dateHistogram() Null Pointer – Ingestion Recovered After Restarting Event Processor Services

Description
During troubleshooting of the ingestion outage, the following backend error was observed in the logs:

LogAnalyzerResource.dateHistogramView:
LogAnalyzerService.getChartViewData:
Cannot invoke "org.opensearch.client.opensearch._types.aggregations.Aggregate.dateHistogram()"
because "agg" is null

This error was repeatedly logged while the system was not processing incoming data.
At the same time:

• No new data was being indexed
• Event processing services showed abnormal CPU utilization (>500%)
• Backend remained reachable but ingestion pipeline was stalled

Observed Behavior

• Repeated Aggregate.dateHistogram() null reference error
• Backend unable to generate histogram aggregations
• No data ingestion for ~12 hours
• High CPU usage on:
  • event-processor-manager
  • event-processor-worker

Remediation Performed
We manually restarted the following services:

docker stop id_utmstack_event-processor-manager
docker stop id_utmstack_event-processor-worker

(or equivalent service restart)
After restarting these services:

• CPU utilization normalized
• Event processing resumed
• Data ingestion recovered
• OpenSearch indexing resumed normally

Initial Technical Assessment
The error suggests:

1. A null aggregation object returned from OpenSearch