Google Cloud's London outage: the generators never got the signal to start
On 20 October, Google Cloud's London zone europe-west2-b went dark for about six hours, disrupting around 40 Google Cloud products and, downstream, services at Lloyds, HSBC, Revolut and Royal Mail. Google's incident report is a sobering read on layered redundancy: one of the site's two utility feeds was already offline for repairs when an underground cable fault took out the other. The diesel generators should then have started, but the undervoltage sensors feeding the dual-redundant PLCs were wired upstream of the isolated breaker, so the controllers saw voltage on a disconnected path and never sent a start signal. Because primary and standby PLCs ran identical logic, the redundancy shared the same flaw. The UPS batteries, rated for roughly five minutes, ran down before operators could manually transfer the load. Google is now reviewing PLC code across similar sites, freezing maintenance there, and committing to service-level fixes for regional disks, Cloud SQL and AlloyDB failover, GKE autoscaling and BigQuery isolation. The takeaway for architects on any cloud: a single zone is a single failure domain, even inside a well-run hyperscaler, so critical workloads need tested multi-zone or multi-region failover rather than trust in the provider's backup power.
This is a summary by our content curator. Read the original at Google Cloud Service Health: https://status.cloud.google.com/incidents/Xs2XTHzAVC9YnafnPyzj.