Dopt - blocks.dopt.com and users.dopt.com APIs are returning 502s – Incident details

blocks.dopt.com and users.dopt.com APIs are returning 502s

Resolved
Major outage
Started 9 months agoLasted about 2 hours

Affected

APIs

Major outage from 2:32 AM to 4:19 AM

blocks.dopt.com

Major outage from 2:32 AM to 4:19 AM

users.dopt.com

Major outage from 2:32 AM to 4:19 AM

Updates
  • Resolved
    Resolved

    We're resolving this incident after redeploying and monitoring our redis clusters. We'll update this notice with next steps we're taking to improve our redis cluster and client health and resiliency.

  • Monitoring
    Monitoring

    We redeployed our redis clusters (worker and sentinels) and all downstream dependencies.

    All 502 errors were resolved at 8:49:24 PDT.

    We'll continue to monitor.

  • Identified
    Identified

    We've identified the issue.

    Several of our redis clusters which are responsible for caching and message passing did not come up correctly after a GKE scheduled maintenance.

    We're rolling out restarts to our redis clusters now.

  • Investigating
    Investigating

    We are actively investigating the Blocks and Users APIs returning 502s.