blocks.dopt.com experiencing intermittent high latencies

Resolved
Degraded performance
Started 7 months ago Lasted about 17 hours

Affected

APIs
blocks.dopt.com
Updates
  • Resolved
    Resolved

    We've deployed updates to our internal block(s) state machine(s) which suffered from high latencies in the case where flows contained large (> 100k) numbers of cycles within them.

    We've also rolled back any load balancer policies related to this incident.

  • Monitoring
    Monitoring

    At 11:40 AM, we started proactively rolling out a more protective load balancer policy to isolate the requests which were triggering latency spikes, and have not observed any latency spikes since then.

  • Identified
    Identified

    We've identified a set of mitigations and are planning a load balancer policy change.

  • Investigating
    Investigating

    We noticed that blocks.dopt.com (the blocks API) experienced high latencies (> 1s) during the following periods (pacific time):

    • 1:50 - 1:55 AM
    • 2:10 - 2:15 AM
    • 2:30 - 2:35 AM
    • 10:50 - 10:55 AM
    • 11:10 - 11:15 AM