On 12 and 13 November 2026, Microsoft 365 had a global outage on the networking gateway that fronts many of its services. The preliminary post-incident report, dated 15 November 2026, says the most significant impact was on Exchange Online: most users were unable to access their mailbox through Outlook or other mail clients, and email delivery was delayed for many users. Elements of Copilot, Teams, SharePoint, OneDrive, Defender and Purview that interact with Exchange Online were also affected, though the report says most of these services largely remained online and accessible. The report gives the window as 16:20 UTC on 12 November to 16:10 UTC on 13 November. That is 23 hours and 50 minutes.
Charles Lamanna, Microsoft's EVP of Copilot, Agents and Platform, posted an abridged root cause on X on 16 November 2026 at 06:06 UTC. He pointed readers to the preliminary report in the Microsoft 365 Admin center, under incident IDs MO1550821 and EX1550543, and quoted an @MSFT365Status post that announced the same. The full preliminary PIR is not on a public Microsoft page; it sits behind that Admin center login. Lamanna's post is the public source for the root cause below. The report date, the incident window and the user-impact wording come from an unofficial copy of the report, whose title page reads "Preliminary Post Incident Report For Microsoft 365, Report Date: November 15, 2026" and whose root-cause section matches Lamanna's post. The report says its information is preliminary and subject to change, and that a final PIR will supersede it. Read the details below that way.
According to the preliminary report, three factors combined.
First, a latent lock-contention defect. The change first started deployment in July 2026, went through safe deployment, and, in the report's words, ran worldwide "for more than three months... without impact."
Second, approximately fifteen hours before the outage, Teams failed over onto the primary gateway that also serves Exchange Online, increasing load and changing traffic patterns. The report describes this kind of failover as part of normal operations.
Third, nearly 14% of that gateway's server capacity was out of service because of an unrelated issue from the previous day.
Taken together, the increased load and the reduced capacity caused the gateway to hit the July defect. At its lowest, more than 80% of servers were unresponsive. The report says that, due to a separate issue, the normal bulkheads between partitions of the gateway did not stop the cascade, which let the impact spread globally. As servers came back, they were immediately overwhelmed by waiting traffic.
The report also says ordinary load testing missed the bug. The labs used a different traffic and SKU mix, and the contention "is worsened on machines with more cores and newer hardware", while load testing "tended to focus on" weaker hardware with fewer cores.
And because the defect was a latent bug from several months earlier, "normal rollback, flighting, and feature flags were unable to quickly revert the change."
That last point is the one I want to sit with, because it shows both the use and the limit of a canary.
One of the calmer production migrations I've done was also one of the highest-risk on paper: moving a legacy PHP application from 7.3 to 8.4 while it served roughly 100,000 daily active users.
I couldn't pause the application while I found compatibility issues. A one-shot cutover would have turned every unknown into a production incident. A rewrite would have blown up the change surface and delayed the runtime upgrade for months.
The plan optimized for one thing: reversibility. Keep the existing traffic-control layer, run both runtimes in parallel, shift traffic gradually, and make rollback a routing change instead of a new deployment.
I wrote the engineering detail in Migrating a high-traffic PHP application to 8.4. The case study is the shorter version.
The migration unit was traffic, not code
The application already ran behind Cloudflare Load Balancer. I set up two origin pools: a stable pool of PHP 7.3 nodes, and a candidate pool of PHP 8.4 nodes.
The load balancer gave me a weighted split. The rollout started at 95% of traffic on PHP 7.3 and 5% on PHP 8.4. The candidate pool grew, more nodes and more traffic, only after production behavior held up.
If a route failed on PHP 8.4 but worked on PHP 7.3, the response was already decided: disable the candidate pool, send traffic back to PHP 7.3, isolate the issue, deploy the fix to the PHP 8.4 pool, and resume the canary.
Nobody had to reverse a database migration or restore an image. They changed the traffic weights.
A real rollback plan specifies the operator and permission required, the exact control-plane action, the observable condition that triggers rollback, the state that does and does not need reverting, and the verification required before traffic resumes. Rollback is safest when it is a normal, rehearsed operation, not a move invented mid-incident.
Why staging was not enough
The compatibility failures that scared me were not syntax errors. A legacy application carries assumptions about runtime behavior across a long tail of code paths: warnings that become exceptions or change severity, weak typing exposed by particular user inputs, dependency behavior staging never exercises, framework and library edges under the newer runtime, scheduled jobs and admin flows with uneven traffic, and routes whose production data shape differs from test fixtures.
Staging catches the obvious failures. It is not proof that every production path is compatible. I'd rather let reality tell me what's wrong than trust staging to have covered it. I designed the rollout around that uncertainty, rather than pretending I could eliminate it before real traffic was involved.
The canary ran the application under real request distribution and real data shape. The signals that mattered were HTTP error rate, application logs, route-level behavior (so a healthy median could not hide one broken endpoint), background jobs, resource saturation, and business-level symptoms: failures that return technically valid responses.
A small canary only helps if you can tell why a candidate request failed, and whether the failure is isolated or systemic. The traffic split was a risk-control mechanism. It was not a substitute for being able to see that.
The preliminary Microsoft report supports that caution. Its load tests, on a different traffic and SKU mix and on weaker hardware, missed a defect that showed up on production machines.
The limit of a canary
A canary of the kind used in that PHP migration is built to catch problems that appear under real traffic while most of the system is still on the old runtime. This bug, on Microsoft's own preliminary account, went through safe deployment and ran worldwide for more than three months without impact. A 95/5 split at rollout would not have caught a defect that stayed quiet that long. Once a change has been the only version in production for months, there is no older pool left to switch back to. I am not claiming that method would have prevented this outage.
The report points to a different gap. Microsoft wrote that because the defect was a latent bug from several months earlier, its normal rollback, flighting and feature flags could not quickly revert the change, and engineers developed an update that returned the code path to the previously known good state. That is a requirement a canary does not cover: being able to revert a change after the rollout is over, not only in the first hours.
Why I didn't build a migration platform
The load balancer already expressed the control I needed: route a percentage of requests to one pool and the rest to another. A custom migration service would have added a second traffic-control plane, a second operational interface, and another failure mode.
This worked because the migration boundary was the runtime pool. It did not need request-level session pinning, a data rewrite, or a new orchestration system. The existing architecture was enough. More machinery isn't more safety when it puts untested machinery on the critical path.
What to settle before you start
These questions come from how this migration was set up and from the rollback plan it needed. They apply to any high-traffic runtime upgrade, not only a PHP one.
Can the old and new runtimes run side by side behind the traffic-control layer you already have? If they cannot, rollback will probably be a deployment, not a switch.
Is there state the new version will write that the old version cannot read? In this migration, rollback did not require reversing a database change or restoring an image, because the migration boundary was the runtime pool and there was no data rewrite. As a general rule, if an upgrade does write state the old version cannot read, shifting traffic back will not undo it. The rollback plan has to name the state that does and does not need reverting before the first canary starts.
What will you watch on the 5% that staging will not show? Route-level errors, background jobs, and business symptoms belong on that list, not only a pool-wide error rate.
Who can change the weights, and under what condition do they do it? That answer needs to exist before the candidate pool sees traffic.
For high-traffic runtime migrations, the hard part isn't installing the new version. It's building a path where real traffic can teach you what's broken while the old version stays one traffic-weight away.
The application moved from PHP 7.3 to 8.4 while serving about 100,000 daily active users. The result I cared about wasn't only that PHP 8.4 ran fine. It was that compatibility failures could be isolated to the candidate pool without taking the application down with them.
That design is for problems that show up under real traffic while you can still switch back. It is not an answer to a change that has already been live for months, which is the failure Microsoft's preliminary report describes.
If you have a problem like this, I help businesses turn engineering problems into working systems through Developer Labs.