Dopt - blocks.dopt.com experiencing intermittent high latencies – Incident details

blocks.dopt.com experiencing intermittent high latencies

Resolved
Degraded performance
Started 10 months agoLasted about 17 hours

Affected

APIs

Degraded performance from 8:49 AM to 1:55 AM

blocks.dopt.com

Degraded performance from 8:49 AM to 1:55 AM

Updates
  • Resolved
    Resolved

    We've deployed updates to our internal block(s) state machine(s) which suffered from high latencies in the case where flows contained large (> 100k) numbers of cycles within them.

    We've also rolled back any load balancer policies related to this incident.

  • Monitoring
    Monitoring

    At 11:40 AM, we started proactively rolling out a more protective load balancer policy to isolate the requests which were triggering latency spikes, and have not observed any latency spikes since then.

  • Identified
    Identified

    We've identified a set of mitigations and are planning a load balancer policy change.

  • Investigating
    Investigating

    We noticed that blocks.dopt.com (the blocks API) experienced high latencies (> 1s) during the following periods (pacific time):

    • 1:50 - 1:55 AM
    • 2:10 - 2:15 AM
    • 2:30 - 2:35 AM
    • 10:50 - 10:55 AM
    • 11:10 - 11:15 AM