Delayed processing of bulk message status updates
Resolved·Degraded performance

We have deployed a fix to permanently resolve issues with delayed bulk status update jobs.

Tue, Oct 15, 2024, 02:42 PM
(8 months ago)
·
Affected components
API
Updates

Resolved

We have deployed a fix to permanently resolve issues with delayed bulk status update jobs.

Tue, Oct 15, 2024, 02:42 PM

Monitoring

We have finished clearing the queue of backlogged bulk status update jobs. We are still seeing intermittent delays processing bulk status update jobs of no more than 1 minute. We are working on shipping a permanent solution to resolve any residual queueing delays for bulk status updates.

Tue, Oct 15, 2024, 02:02 PM(39 minutes earlier)

Monitoring

We have implemented a fix causing bulk status update jobs to become stuck in a queued state. We are now processing thru the backlog of bulk status update jobs that have become stuck over the past few hours.

Tue, Oct 15, 2024, 01:16 PM(45 minutes earlier)

Identified

We have identified an issue causing deferred jobs used to executing bulk message status updates to get stuck in a queued state. We are first working to resolve the issue for newly issued bulk status updates. We will then focus on clearing out the jobs stuck in a queued state.

Tue, Oct 15, 2024, 12:27 PM(49 minutes earlier)

Investigating

We are investigating reports of delayed processing of bulk message status updates from the V1 API.

Tue, Oct 15, 2024, 12:13 PM(13 minutes earlier)