At 10:59 AM, we re-deployed services that had gone into dead-lock waiting for other services to come up. This resolved all issues with higher than expected latencies and errors.

At the peak (~10:58 AM), less than 3% of requests had to be retried. All systems are back to normal post re-deploy.

We're actively working on mitigating dead-locking and k8s coordination.

A previously reported automated failure on health-check has been resolved.

At 8:50 PM, we received a health-check failure notice (and internal incident alarms) that our blocks and users APIs were experiencing higher than expected internal error rates and latencies.

This was due to the slow rollout of underlying GCP infrastructure undergoing routine maintenance upgrades. Our infrastructure self-healed at 9:00 PM once the slow rollout completed; subsequently, latencies returned to normal levels, and internal error rates dropped back to ~0.

For the ten minutes between 8:50 PM and 9:00 PM, we identified that roughly 0.43% of requests experienced higher than expected latencies (and potentially timeouts). Our clients and SDK exponentially retry these requests, and we have not recorded any errors after 9:00 PM.

A previously reported automated failure on health-check has been resolved.

A previously reported automated failure on health-check has been resolved.