Dopt - blocks and users API latency spikes – Incident details

blocks and users API latency spikes

Resolved
Partial outage
Started 5 months agoLasted 10 minutes

Affected

APIs

Partial outage from 4:50 AM to 5:00 AM

blocks.dopt.com

Partial outage from 4:50 AM to 5:00 AM

users.dopt.com

Partial outage from 4:50 AM to 5:00 AM

Updates
  • Resolved
    Resolved

    A previously reported automated failure on health-check has been resolved.

    At 8:50 PM, we received a health-check failure notice (and internal incident alarms) that our blocks and users APIs were experiencing higher than expected internal error rates and latencies.

    This was due to the slow rollout of underlying GCP infrastructure undergoing routine maintenance upgrades. Our infrastructure self-healed at 9:00 PM once the slow rollout completed; subsequently, latencies returned to normal levels, and internal error rates dropped back to ~0.

    For the ten minutes between 8:50 PM and 9:00 PM, we identified that roughly 0.43% of requests experienced higher than expected latencies (and potentially timeouts). Our clients and SDK exponentially retry these requests, and we have not recorded any errors after 9:00 PM.

  • Investigating
    Investigating

    We're investigating an incident which was automatically triggered by a health-check failure.