We're investigating an incident which was automatically triggered by a health-check failure.

We're investigating an incident which was automatically triggered by a health-check failure.

This incident has been resolved.

After monitoring, we determined that a version of the Blocks API which began rolling out at approximately 3:00 PM PDT had an incorrectly configured client for one of our new internal services. This client began throwing errors because it could not reach its internal service due to a wrong internal DNS reference.

We first rolled back to a previous version of the Blocks API (from this morning at 9:00 AM) and confirmed that the API was functional (no 5xx reported and e2e tests all passing).

We subsequently rolled out updates to correct the wrong internal DNS reference and also re-configured the client to not crash the API when incorrectly configured.

We've completed all our rollbacks.

Additionally, we're beginning to redeploy our earlier performance and logging updates in a sequential manner. We've added further redundancy checks in our code to prevent the Blocks API from crashing on transitions and intents.

We have started rolling back several potentially problematic updates we made this afternoon to the Blocks API to mitigate the degraded performance.

We are updating the status of this incident to a partial outage until we can fully remedy the 5xx errors.

We're resolving this incident after redeploying and monitoring our redis clusters. We'll update this notice with next steps we're taking to improve our redis cluster and client health and resiliency.

We redeployed our redis clusters (worker and sentinels) and all downstream dependencies.

All 502 errors were resolved at 8:49:24 PDT.

We'll continue to monitor.

We've identified the issue.

Several of our redis clusters which are responsible for caching and message passing did not come up correctly after a GKE scheduled maintenance.

We're rolling out restarts to our redis clusters now.

We've deployed updates to our internal block(s) state machine(s) which suffered from high latencies in the case where flows contained large (> 100k) numbers of cycles within them.

We've also rolled back any load balancer policies related to this incident.

At 11:40 AM, we started proactively rolling out a more protective load balancer policy to isolate the requests which were triggering latency spikes, and have not observed any latency spikes since then.

We've identified a set of mitigations and are planning a load balancer policy change.

At 2:58 PM, once the revert was completed, we confirmed that app.dopt.com functions as expected. We've also rolled back the broken nginx image and its assets internally.

At 2:55 PM, we began rolling out a revert to the previous version of app.dopt.com's client.