Dopt - blocks.dopt.com is reporting degraded performance on transitions and intents – Incident details

blocks.dopt.com is reporting degraded performance on transitions and intents

Resolved
Partial outage
Started 9 months agoLasted about 1 hour

Affected

APIs

Partial outage from 10:01 PM to 10:56 PM

blocks.dopt.com

Partial outage from 10:01 PM to 10:56 PM

Updates
  • Resolved
    Resolved

    This incident has been resolved.

    After monitoring, we determined that a version of the Blocks API which began rolling out at approximately 3:00 PM PDT had an incorrectly configured client for one of our new internal services. This client began throwing errors because it could not reach its internal service due to a wrong internal DNS reference.

    We first rolled back to a previous version of the Blocks API (from this morning at 9:00 AM) and confirmed that the API was functional (no 5xx reported and e2e tests all passing).

    We subsequently rolled out updates to correct the wrong internal DNS reference and also re-configured the client to not crash the API when incorrectly configured.

  • Monitoring
    Monitoring

    We've completed all our rollbacks.

    Additionally, we're beginning to redeploy our earlier performance and logging updates in a sequential manner. We've added further redundancy checks in our code to prevent the Blocks API from crashing on transitions and intents.

  • Identified
    Identified

    We have started rolling back several potentially problematic updates we made this afternoon to the Blocks API to mitigate the degraded performance.

    We are updating the status of this incident to a partial outage until we can fully remedy the 5xx errors.

  • Investigating
    Investigating

    On block transitions and flow intents, Blocks API is reporting degraded performance with intermittent 5xx errors.