Resolved
This incident has been resolved.
Monitoring
The cluster has recovered and we're continuing to monitor.
Identified
We've identified backpressure coming from our Clickhouse cluster due to a large number of mutation queries. We've scaled the cluster up and the backlog of workflow run logs has been processed. Logs are no longer delayed.
Log query latencies are still elevated.
Investigating
We're currently investigating a delay in workflow run log ingestion. This affects the logs shown in the "Workflow run" views. Workflow run execution and notification delivery are not impacted