Microsoft down — LIVE updates Azure back up; Xbox and Minecraft still down
Multiple Microsoft services suffering
Azure update - Recovery
According to the Azure status page, the team there has rolled back to the "last known good" configuration. This means that recovery signs should start appearing. However, it may not be a full one yet.
We have pushed our ‘last known good’ configuration, and customers may begin to see initial signs of recovery. We are currently recovering nodes and routing traffic through healthy nodes, and as we make progress in this workstream, customers will continue to see improvement.
Customer configuration changes will remain temporarily blocked while we continue mitigation efforts. We will notify customers once this block has been lifted.
Xbox struggling
With Xbox Live and Minecraft still showing outage issues, I tried checking the Xbox status page and can't access it at all.
All I can see is a tiny Xbox logo with the spinning loading circle.
I have reached out to Xbox about the outage, but have yet to receive a response.
Reports down
Over on Down Detector the main Microsoft services from Azure to 365 and Teams have steadily declined. It's not quite over but Azure's reports are under 2,500.
However, the gaming services Minecraft and Xbox Live remain seeing high reports.
We are currently waiting for the Azure mitigation to go through. That should happen in about 10 or 15 minutes.
Mitigation deployed
According to the Azure status page, the Microsoft team has deployed its fix, which should be ready in about 30 minutes.
We have initiated the deployment of our 'last known good' configuration. This is expected to be fully deployed in about 30 minutes from which point customers will start to see initial signs of recovery. Once this is completed, the next stage is to start to recover nodes while we route traffic through these healthy nodes.
Customer configuration changes will remain blocked during this time as we work towards mitigation. We will communicate to customers when this block is reverted.
A statement from Microsoft
I reached out to Microsoft toward the beginning of the outage, and we just received a response from the company.
"We are working to address an issue affecting Azure Front Door that is impacting the availability of some services. Customers should continue to check their Service Health Alerts and the latest update on this issue can be found on the Azure status page.”
It's not an explanation and directs us to the status page we've already been reviewing. But if you haven't looked there, please do.
I will note that the "failover strategies" page it currently links to is broken.
Microsoft Status page down
We have been using the Microsoft Azure status site to see what the Azure team is saying about updates and how it appears to be going.
However, even that page is down now!
As you can see in the screenshot above the only thing I can see is a "We'll be back soon!" message. It's not clear how long that will last.
Every region affected
You'll notice that the Azure status page has changed to show issues across every region of that they service. This includes AFD a portion of the cloud distribution service being down.
This will likely have widespread issues as problems spike up and down for services that use Azure.
What readers are telling us
Tom's Guide readers have reached out to confirm that some services they use aren't working properly.
Some of these include the Chris Hemsworth fitness app Centr, Workfront via the Adobe Experience Platform, the Pearson Vue testing system for schools, and at least one university where a student said they couldn't upload documents.
Another Microsoft update
Microsoft Azure's status page has a new update showing that the team there is trying to get back to a version that wasn't facing issues. However, it appears to be partially working and the rollback may not be ready yet:
We suspect that an inadvertent configuration change as the trigger event for this issue. We are taking two concurrent actions where we are blocking all changes to the AFD services and at the same time rolling back to our last known good state.
We have failed the portal away from AFD to mitigate the portal access issues. Customers should be able to access the Azure management portal directly.
We do not have an ETA for when the rollback will be completed.
Steep drop in reports
Since the solutions message appeared on Azure reports on Down Detector have taken a steep drop.
Here's the thing, though, other servics from Microsoft like Minecraft and Xbox are going up.
Additionally, places that use Azure like Starbucks are still totally down. Starbucks.com is inaccessible as of this writing.
Azure update
The Microsoft Azure status page has another update and it looks like there may be some solutions.
Customers can attempt to use programmatic methods (PowerShell, CLI, etc.) to access/utilize resources if they are unable to access the portal directly. We have failed the portal away from Azure Front Door (AFD) to attempt to mitigate the portal access issues and are continuing to assess the situation.
We are actively assessing failover options of internal services from our AFD infrastructure. Our investigation into the contributing factors and additional recovery workstreams continues. More information will be provided within 60 minutes or sooner.
That's specifically for Azure customers but it may help bring things back online.
Have you been affected?
Azure, unlike AWS, has more government contracts and business ones. Though it does power some services like Walmart.
Have you noticed a degradation of service because of the Microsoft problems today?
Let us know by emailing me at scott.younker@futurenet.com
Some services coming back online?
I am seeing a drop in reports on Down Detector.
Specifically for Azure, the Microsoft Store and 365.
However, services like Minecraft and Xbox Live are going up. It's not clear at this moment if those are separate issues.
Azure status notice
Microsoft has acknowledged the outage on the Azure status page. Here's what it currently says:
Starting at approximately 16:00 UTC, we began experiencing DNS issues resulting in availability degradation of some services. Customers may experience issues accessing the Azure Portal. We have taken action that is expected to address the portal access issues here shortly. We are actively investigating the underlying issue and additional mitigation actions. More information will be provided within 60 minutes or sooner.
It seems to be a similar problem that affected and is affecting AWS. Unlike that outage, Microsoft's appears more global.
AWS is also down...again
Simultaneously, Amazon Web Services has gone down again. Specifically the US-Region-1 site.
We're already seeing that impact on multiple services.
If the internet isn't working for you today, these simultaneous outages may be why.
Affected Microsoft services
Here's what Microsoft-owned services we've seen so far as having issues.
We're certain there are more plus other companies that use Azure like Starbucks but this is what we've specifically seen spike at the moment.
- Microsoft 365
- Microsoft Azure
- Micrsoft Copilot
- Microsoft Entra
- Microsoft Store
- Microsoft Teams
- Minecraft
- Xbox
