Increase in Job Queue Length

Incident Report for OvationCXM

Resolved

The recent impact to our job queue has been resolved. The system is operational.
Posted Jun 30, 2025 - 10:54 PDT

Update

We identified the root cause which led to a backup in the job queue. Job queues recovered quickly and have remained steady. The incident will remain open over the weekend as the team continue to monitor the situation.
Posted Jun 27, 2025 - 18:12 PDT

Update

Following our last update, while the fix we implemented addressed one of the root causes, we are still observing higher than normal response times in some services.

Users may experience intermittent slowness across the platform. We are working to identify all affected areas and understand the full impact.

Resolving this has our full attention. Our engineering team is actively investigating the remaining issues to restore normal performance as quickly as possible. We sincerely apologize for the continued disruption and will provide another update as soon as we have more information.
Posted Jun 27, 2025 - 14:21 PDT

Update

We have deployed the fix, and our initial monitoring shows that performance for Tasks, Actions, and Automations has returned to normal.

Our team will continue to monitor the situation to ensure a full and stable recovery. We will post a final update once we can confirm the issue is fully resolved.
Posted Jun 27, 2025 - 08:44 PDT

Update

We have confirmed that the performance issues identified earlier may be affecting Tasks, Actions, and Automations. We have identified the initial cause and are working on developing a solution to avoid this in the future.
Posted Jun 26, 2025 - 15:46 PDT

Update

Following up on our previous notice, at this time we have not observed evidence yet that these backend slowdowns are impacting users. All user-facing services are operating normally. Our team is continuing to monitor the situation and will provide further updates as necessary.
Posted Jun 26, 2025 - 12:30 PDT

Monitoring

We are looking into a possible slowdown in our service and a fix is being implemented. As we know more and performance is restored we will update this page. Thank you for your patience.
Posted Jun 26, 2025 - 10:41 PDT

Investigating

We are currently investigating this issue.
Posted Jun 26, 2025 - 10:31 PDT
This incident affected: Production Platform (OLTP Database Cluster, Core/APIs, Microservices, Messaging, Bots, Search, Analytics).