Massive Azure Outage Is Over, But Problems Linger - Here's What Happened

picture alliance/Contributor/picture confederation via Getty Images

Follow ZDNET: Add america arsenic a preferred source on Google.

ZDNET's cardinal takeaways

• Microsoft Azure knowledgeable a world outage connected October 29.
• Microsoft customer-facing services were affected.
• Recovery came later that aforesaid day, but immoderate problems linger.

Last week, Amazon Web Services (AWS) went down, and galore of america were miserable. This week, it's Microsoft Azure's move to autumn down and spell boom, and erstwhile more, we're beautiful darn unhappy astir it. 

Microsoft states that nan latest Azure outage began astatine astir noon ET connected October 29. However, Downdetector, which relies connected personification reports, shows nan problems surfaced earlier, astir 11:40 a.m. 

Also: The monolithic AWS outage that collapsed half nan net is yet complete - here's what happened

ThousandEyes, nan Cisco web information company, "detected HTTP timeouts, server correction codes, and elevated packet nonaccomplishment astatine nan separator of Microsoft's network, preventing successful connections to affected services and often timing retired aliases returning service-related errors."

The latest position update

As of 5:30 p.m. ET, October 29, Microsoft reported, "We initiated nan deployment of our 'last known good' configuration, which has now successfully completed. We are presently recovering nodes and re-routing postulation done patient nodes."

Don't get excessively excited, though. We're not done yet. Microsoft continued, "As betterment progresses, immoderate requests whitethorn still onshore connected unhealthy nodes, resulting successful intermittent failures aliases reduced readiness until much nodes are afloat restored. This betterment effort involves reloading configurations and rebalancing postulation crossed a ample measurement of nodes to reconstruct afloat operational scale. The process is gradual by design, ensuring stableness and preventing overload arsenic limited services recover. We expect continued betterment crossed affected regions. This intends we expect betterment to hap by 23:20 UTC connected 29 October 2025." 

That's 7:30 p.m. ET.

In reality, it took a spot longer. Azure reported that it was backmost to normal by 8:05 p.m. yesterday. Even then, Microsoft warned that customer configuration changes to Azure Front Door (AFD) would stay temporarily blocked. Microsoft promised it would notify customers erstwhile this artifact has been lifted. In addition, while "error rates and latency are backmost to pre-incident levels, a mini number of customers whitethorn still beryllium seeing issues, and we are still moving to mitigate this agelong tail."

If you're still having problem today, talk to Azure. If things are really fouled up, Microsoft recommends you see implementing existing failover strategies utilizing Azure Traffic Manager to redirect postulation from Azure Front Door to their root servers arsenic an interim measure." This is acold from an easy fix. If your unit isn't knowledgeable pinch Azure postulation routing, I'd grit my teeth and hold for Azure to travel wholly backmost online.

Also: No 1 pays ransomware demands anymore - truthful attackers person a caller goal

Unlike nan AWS failure, which -- while monolithic successful its harm -- was constricted to a azygous region (AWS East), according to nan Azure Status page arsenic of 1:30 p.m. ET, each Azure regions were down. 

Tracing nan faulty deployment

We still don't person a last study connected what happened. At first, Microsoft only said, "Starting astatine astir 16:00 UTC, we began experiencing Azure Front Door (AFD) issues resulting successful a nonaccomplishment of readiness of immoderate services. We fishy that an inadvertent configuration alteration was nan trigger arena for this issue. We are taking 2 concurrent actions wherever we are blocking each changes to nan AFD services and, astatine nan aforesaid time, rolling backmost to our past known bully state."

Microsoft's first study connected nan incident stated, "An inadvertent tenant configuration alteration wrong AFD triggered a wide work disruption affecting some Microsoft services and customer applications limited connected AFD for world contented delivery." The alteration caused an invalid configuration state, which, successful turn, resulted successful a important number of AFD nodes failing to load properly, including accrued latencies, timeouts, and relationship errors for downstream services. In different words, it was a complete mess.

Also: Best VPN services 2025: Our apical picks for velocity and security

As unhealthy nodes dropped retired of nan world pool, postulation distribution crossed patient nodes became imbalanced, amplifying nan effect and causing intermittent readiness moreover successful partially patient regions. Microsoft instantly "blocked each further configuration changes to forestall further propagation of nan faulty authorities and began deploying a 'last known good' configuration crossed nan world fleet. Recovery required reloading configurations crossed a ample number of nodes and rebalancing postulation gradually to debar overload conditions arsenic nodes returned to service. This deliberate, phased betterment was basal to stabilize nan strategy while restoring standard and ensuring nary recurrence of nan issue."

The responsibility has been traced backmost to a faulty tenant configuration deployment process. "Our protection mechanisms, to validate and artifact immoderate erroneous deployments, grounded owed to a package defect that allowed nan deployment to bypass information validations. Safeguards person since been reviewed, and further validation and rollback controls person been instantly implemented to forestall akin issues successful nan future."

Although it's not mentioned successful this document, early Azure reports put immoderate of nan blasted connected -- you guessed it! -- a Domain Name System (DNS) problem. Say it pinch me: When there's a web problem, "It's ever DNS!"

It's ever DNS. 

sjvn

Which sites and services were affected?

Ordinary group felt nan symptom arsenic well. Popular services specified arsenic Microsoft 365 and Microsoft Intune for business users and Xbox Live and Minecraft for group conscionable wanting to person nosy person besides been down. Others reported that Microsoft logins were besides slowing to a crawl aliases failing entirely.  

The pursuing services were affected:

• Microsoft 365
• Microsoft Azure
• Microsoft Copilot
• Microsoft Entra
• Microsoft Store
• Microsoft Teams
• Minecraft
• Xbox 

It was a bad time if you relied connected Microsoft.

Alaska Airlines suffered interruptions to its captious soul systems, including its website and operational infrastructure. Vodafone successful nan UK and Heathrow Airport were besides reported to person been affected by nan outage.

Behind nan scences, Microsoft now reports that nan pursuing Azure services were affected: App Service, Azure Active Directory B2C, Azure Communication Services, Azure Databricks, Azure Healthcare APIs, Azure Maps, Azure Portal, Azure SQL Database, Container Registry, Media Services, Microsoft Defender External Attack Surface Management, Microsoft Entra ID, Microsoft Purview, Microsoft Sentinel, Video Indexer, and Virtual Desktop. 

Earlier, Ookla telecom expert Luke Kehoe said, "Microsoft Azure has knocked galore services offline worldwide, pinch a wide blast radius crossed airlines, banks, and authorities agencies. It is nan 2nd specified arena this month, highlighting nan systemic risks of attraction and azygous points of logical failure, sloppy of really physically hardened nan infrastructure is." 

Also: Microsoft's revamped Windows 11 Start paper is rolling retired - but I'll instrumentality pinch my favourite alternative

He's sewage a point. We trust excessively heavy connected AWS, Azure, and different unreality services, which, erstwhile nan going gets tough, move retired to beryllium azygous points of failure. 

Be that arsenic it may, successful its latest quarterly report, which came aft nan doorbell connected nan aforesaid day, Microsoft reported that it hit Wall Street estimates and that Azure's income grew by astir 40%. Still, pinch this ongoing nonaccomplishment and Microsoft admitting that it can't support up pinch AI and unreality demands, Microsoft's banal sank little successful after-market trading.

Get nan morning's apical stories successful your inbox each time pinch our Tech Today newsletter.