Workflow run logs are delayed

Incident Report for Knock

Resolved

This incident has been resolved.

Posted Oct 15, 2024 - 16:56 UTC

Monitoring

The cluster has recovered and we're continuing to monitor.

Posted Oct 15, 2024 - 15:31 UTC

Identified

We've identified backpressure coming from our Clickhouse cluster due to a large number of mutation queries. We've scaled the cluster up and the backlog of workflow run logs has been processed. Logs are no longer delayed.

Log query latencies are still elevated.

Posted Oct 15, 2024 - 14:54 UTC

Investigating

We're currently investigating a delay in workflow run log ingestion. This affects the logs shown in the "Workflow run" views. Workflow run execution and notification delivery are not impacted

Posted Oct 15, 2024 - 13:58 UTC

This incident affected: Knock systems (API and workflow logs).