Affected components
Logs
Updates
Resolved
This incident has been resolved.
Tue, Oct 15, 2024, 04:56 PM
Monitoring
The cluster has recovered and we're continuing to monitor.
Tue, Oct 15, 2024, 03:31 PM(1 hour earlier)
Identified
We've identified backpressure coming from our Clickhouse cluster due to a large number of mutation queries. We've scaled the cluster up and the backlog of workflow run logs has been processed. Logs are no longer delayed.
Log query latencies are still elevated.
Tue, Oct 15, 2024, 02:54 PM(36 minutes earlier)
Investigating
We're currently investigating a delay in workflow run log ingestion. This affects the logs shown in the "Workflow run" views. Workflow run execution and notification delivery are not impacted
Tue, Oct 15, 2024, 01:58 PM(56 minutes earlier)