This incident has been resolved.

After monitoring, we determined that a version of the Blocks API which began rolling out at approximately 3:00 PM PDT had an incorrectly configured client for one of our new internal services. This client began throwing errors because it could not reach its internal service due to a wrong internal DNS reference.

We first rolled back to a previous version of the Blocks API (from this morning at 9:00 AM) and confirmed that the API was functional (no 5xx reported and e2e tests all passing).

We subsequently rolled out updates to correct the wrong internal DNS reference and also re-configured the client to not crash the API when incorrectly configured.

We've completed all our rollbacks.

Additionally, we're beginning to redeploy our earlier performance and logging updates in a sequential manner. We've added further redundancy checks in our code to prevent the Blocks API from crashing on transitions and intents.

We have started rolling back several potentially problematic updates we made this afternoon to the Blocks API to mitigate the degraded performance.

We are updating the status of this incident to a partial outage until we can fully remedy the 5xx errors.