A three-month-old bug and one failover: how Microsoft 365 went dark for a day
Microsoft has explained the nearly 24-hour Microsoft 365 outage of 12–13 November that took down Outlook, Exchange Online, Teams and Copilot for users worldwide. The root cause was a latent lock-contention bug in a fix to M365's networking gateway servers that had been running quietly in production since July; load tests never exposed it because the lab used smaller, fewer-core machines than the newer hardware in the fleet. It surfaced when Teams failed over onto the gateway that serves Exchange Online while about 14% of gateway capacity was already offline for an unrelated reason. Requests piled up, traffic shifted to the surviving servers, and those failed too: within two hours more than 80% of the fleet was unresponsive and regional isolation did not contain the spread. Restarted servers were immediately swamped by the backlog, so recovery waited on a code fix. Teams, OneDrive and SharePoint could fail over and came back sooner; Exchange Online could not. Microsoft says it is adding capacity, monitoring and safeguards so traffic redirection cannot set off another cascade. The takeaways for any platform team: load-test on production-class hardware, treat failover as a load event the receiving side must be sized for, and plan how a recovering fleet sheds a backlog instead of just restarting into it.
This is a summary by our content curator. Read the original at BleepingComputer: https://www.bleepingcomputer.com/news/microsoft/microsoft-says-months-old-bug-caused-global-microsoft-365-outage/.