Functionality and normal redundancy is now restored.
Posted Jan 04, 2024 - 16:14 UTC
Update
Initial mitigation is now complete. The SRE team will keep working to make sure every deployment has full redundancy. We will post an update once this is complete.
Posted Jan 03, 2024 - 22:55 UTC
Identified
The cause of the storage instability is identified and there's no risk of re-occurrence currently. Meanwhile, mitigation procedures for the remaining deployments to prevent future data loss are in progress.
Posted Jan 03, 2024 - 17:24 UTC
Investigating
Our monitoring has alerted about an increase in storage-related errors in the Redis Clusters in the aws-us-east-1 zone.
This has been caused by a rolling of the storage nodes, the mitigation procedure is being identified and put in action.
Please avoid re-deploying or restarting deployments in this zone until the incident is resolved.
Posted Jan 03, 2024 - 12:37 UTC
This incident affected: Deployment Zones (Controller at aws-us-east-1).