Delayed processing of bulk message status updates
Incident Report for Knock
Resolved
We have deployed a fix to permanently resolve issues with delayed bulk status update jobs.
Posted Oct 15, 2024 - 14:42 UTC
Update
We have finished clearing the queue of backlogged bulk status update jobs. We are still seeing intermittent delays processing bulk status update jobs of no more than 1 minute. We are working on shipping a permanent solution to resolve any residual queueing delays for bulk status updates.
Posted Oct 15, 2024 - 14:02 UTC
Monitoring
We have implemented a fix causing bulk status update jobs to become stuck in a queued state. We are now processing thru the backlog of bulk status update jobs that have become stuck over the past few hours.
Posted Oct 15, 2024 - 13:16 UTC
Identified
We have identified an issue causing deferred jobs used to executing bulk message status updates to get stuck in a queued state. We are first working to resolve the issue for newly issued bulk status updates. We will then focus on clearing out the jobs stuck in a queued state.
Posted Oct 15, 2024 - 12:27 UTC
Investigating
We are investigating reports of delayed processing of bulk message status updates from the V1 API.
Posted Oct 15, 2024 - 12:13 UTC
This incident affected: Knock systems (API).