Dopt - experiencing intermittent high latencies – Incident details experiencing intermittent high latencies

Degraded performance
Started 10 months agoLasted about 17 hours



Degraded performance from 8:49 AM to 1:55 AM

Degraded performance from 8:49 AM to 1:55 AM

  • Resolved

    We've deployed updates to our internal block(s) state machine(s) which suffered from high latencies in the case where flows contained large (> 100k) numbers of cycles within them.

    We've also rolled back any load balancer policies related to this incident.

  • Monitoring

    At 11:40 AM, we started proactively rolling out a more protective load balancer policy to isolate the requests which were triggering latency spikes, and have not observed any latency spikes since then.

  • Identified

    We've identified a set of mitigations and are planning a load balancer policy change.

  • Investigating

    We noticed that (the blocks API) experienced high latencies (> 1s) during the following periods (pacific time):

    • 1:50 - 1:55 AM
    • 2:10 - 2:15 AM
    • 2:30 - 2:35 AM
    • 10:50 - 10:55 AM
    • 11:10 - 11:15 AM