Some European hosted websites experiencing intermittent service
Incident
Report for Optimizely Service
Resolved
All services have been successfully restored to a normal state.
Summary of impact: From 17:44 on 19 Jun 2018 to 04:30 UTC on 20 Jun 2018 customers using Azure services in North Europe may have experienced connection failures when attempting to access resources hosted in the region. Customers leveraging a subset of Azure services may have experienced residual impact for a sustained period post-mitigation of the underlying issue. We are communicating with these customers directly in their Management Portal.
Preliminary root cause: Engineers identified that an underlying temperature issue in one of the datacenters in the region triggered an infrastructure alert, which in turn caused a structured shutdown of a subset of Storage and Network devices in this location to ensure hardware and data integrity.
Mitigation: Engineers addressed the temperature issue, and performed a structured recovery of the affected devices and the affected downstream services.
Next Steps: A full root cause analysis will be performed and published to https://azure.microsoft.com/en-us/status/ in approximately 72 hours.
Any questions can be sent to support@episerver.com or call:
Worldwide: +46 8 555 827 50
United States: +1 888 726 81 27 (Toll Free)
United Kingdom: +44 800 066 4784 (Toll Free)
Australia: +61 280 363 161
Summary of impact: From 17:44 on 19 Jun 2018 to 04:30 UTC on 20 Jun 2018 customers using Azure services in North Europe may have experienced connection failures when attempting to access resources hosted in the region. Customers leveraging a subset of Azure services may have experienced residual impact for a sustained period post-mitigation of the underlying issue. We are communicating with these customers directly in their Management Portal.
Preliminary root cause: Engineers identified that an underlying temperature issue in one of the datacenters in the region triggered an infrastructure alert, which in turn caused a structured shutdown of a subset of Storage and Network devices in this location to ensure hardware and data integrity.
Mitigation: Engineers addressed the temperature issue, and performed a structured recovery of the affected devices and the affected downstream services.
Next Steps: A full root cause analysis will be performed and published to https://azure.microsoft.com/en-us/status/ in approximately 72 hours.
Any questions can be sent to support@episerver.com or call:
Worldwide: +46 8 555 827 50
United States: +1 888 726 81 27 (Toll Free)
United Kingdom: +44 800 066 4784 (Toll Free)
Australia: +61 280 363 161
Update
Azure update: Information Service availability issue in North Europe - Final validation
Current Status: Many services have recovered, and some customers reported mitigated. As we work toward final stage of recovery, engineers continue to investigate an temperature issue in one of the data centers which caused some Storage and Network devices to shutdown.
Customer Impact: Starting at approximately 17:44 UTC on 19 Jun 2018 a subset of customers in North Europe may experience connection failures when trying to access multiple Azure resources hosted in the region. Next update will be provided in 60 minutes.
Last Updated at: 03:30 UTC on 20 Jun 2018
Current Status: Many services have recovered, and some customers reported mitigated. As we work toward final stage of recovery, engineers continue to investigate an temperature issue in one of the data centers which caused some Storage and Network devices to shutdown.
Customer Impact: Starting at approximately 17:44 UTC on 19 Jun 2018 a subset of customers in North Europe may experience connection failures when trying to access multiple Azure resources hosted in the region. Next update will be provided in 60 minutes.
Last Updated at: 03:30 UTC on 20 Jun 2018
Monitoring
Azure Update: Service availability issue in North Europe - Mitigation applied and validating recovery
Update
Warning Service availability issue in North Europe - Mitigation applied and validating recovery
Current Status: Engineers are seeing recovery. As we work to recover and validate improvements, engineers continue to investigate an underlying temperature issue in one of the data centers which caused Storage, Network device equipment to fail.
Customer Impact: Starting at approximately 17:44 UTC on 19 Jun 2018 a subset of customers in North Europe may experience connection failures when trying to access multiple Azure resources hosted in the region. Next update will be provided in 60 minutes.
Last Updated at: 23:50 UTC on 19 Jun 2018
Current Status: Engineers are seeing recovery. As we work to recover and validate improvements, engineers continue to investigate an underlying temperature issue in one of the data centers which caused Storage, Network device equipment to fail.
Customer Impact: Starting at approximately 17:44 UTC on 19 Jun 2018 a subset of customers in North Europe may experience connection failures when trying to access multiple Azure resources hosted in the region. Next update will be provided in 60 minutes.
Last Updated at: 23:50 UTC on 19 Jun 2018
Update
Latest Update from Azure:
"Service availability issue in North Europe - Applying Mitigation and Recovery in Progress
Starting at approximately 17:44 UTC on 19 Jun 2018 a subset of customers using Virtual Machines, Storage, SQL Database, Key Vault, App Service, Site Recovery, Automation, Service Bus, Event Hubs, Data Factory, Backup, API management, Log Analytics, Application Insight, Azure Batch Azure Search, Redis Cache, Media Services, IoT Hub, Stream Analytics, Power BI, Azure Monitor, Azure Cosmo DB, Azure Container Registry, or Logic Apps in North Europe may experience connection failures when trying to access resources hosted in the region. Engineers have identified the root cause, and actively working to mitigate the issue. Our telemetry has shown improvement, and next update will be provided in 60 minutes or as events warrant."
You can read more here: https://azure.microsoft.com/en-us/status/
*** Please note *** This outage may affect some clients attempting to use paasportal.episerver.net to perform deployments within their DXC environments.
"Service availability issue in North Europe - Applying Mitigation and Recovery in Progress
Starting at approximately 17:44 UTC on 19 Jun 2018 a subset of customers using Virtual Machines, Storage, SQL Database, Key Vault, App Service, Site Recovery, Automation, Service Bus, Event Hubs, Data Factory, Backup, API management, Log Analytics, Application Insight, Azure Batch Azure Search, Redis Cache, Media Services, IoT Hub, Stream Analytics, Power BI, Azure Monitor, Azure Cosmo DB, Azure Container Registry, or Logic Apps in North Europe may experience connection failures when trying to access resources hosted in the region. Engineers have identified the root cause, and actively working to mitigate the issue. Our telemetry has shown improvement, and next update will be provided in 60 minutes or as events warrant."
You can read more here: https://azure.microsoft.com/en-us/status/
*** Please note *** This outage may affect some clients attempting to use paasportal.episerver.net to perform deployments within their DXC environments.
Update
Latest Update from Azure:
" Warning Service availability issue in North Europe - Applying Mitigation and Recovery in Progress
Starting at approximately 17:44 UTC on 19 Jun 2018 a subset of customers using Virtual Machines, Storage, SQL Database, Key Vault, App Service, Site Recovery, Automation, Service Bus, Event Hubs, Data Factory, Backup, API management, Log Analytics, Application Insight, Azure Batch Azure Search, Redis Cache, Media Services, IoT Hub, Stream Analytics, Power BI, or Logic Apps in North Europe may experience connection failures when trying to access resources hosted in the region. Engineers have identified the root cause, and actively working to mitigate the issue. Our telemetry has shown improvement, and next update will be provided in 60 minutes or as events warrant."
You can read more here: https://azure.microsoft.com/en-us/status/
" Warning Service availability issue in North Europe - Applying Mitigation and Recovery in Progress
Starting at approximately 17:44 UTC on 19 Jun 2018 a subset of customers using Virtual Machines, Storage, SQL Database, Key Vault, App Service, Site Recovery, Automation, Service Bus, Event Hubs, Data Factory, Backup, API management, Log Analytics, Application Insight, Azure Batch Azure Search, Redis Cache, Media Services, IoT Hub, Stream Analytics, Power BI, or Logic Apps in North Europe may experience connection failures when trying to access resources hosted in the region. Engineers have identified the root cause, and actively working to mitigate the issue. Our telemetry has shown improvement, and next update will be provided in 60 minutes or as events warrant."
You can read more here: https://azure.microsoft.com/en-us/status/
Identified
From Azure:
Warning Service availability issue in North Europe
Starting at approximately 17:44 UTC on 19 Jun 2018 a subset of customers using Virtual Machines, Storage, Key Vault, App Service, or Site Recovery in North Europe may experience connection failures when trying to access resources hosted in the region. Engineers are aware of this issue and are actively investigating. The next update will be provided in 60 minutes or as events warrant.
Details can be seen here:
https://azure.microsoft.com/en-us/status/
Warning Service availability issue in North Europe
Starting at approximately 17:44 UTC on 19 Jun 2018 a subset of customers using Virtual Machines, Storage, Key Vault, App Service, or Site Recovery in North Europe may experience connection failures when trying to access resources hosted in the region. Engineers are aware of this issue and are actively investigating. The next update will be provided in 60 minutes or as events warrant.
Details can be seen here:
https://azure.microsoft.com/en-us/status/
Update
We are continuing to investigate this issue.
Investigating
We are currently investigating the issue and will update this status accordingly.
This incident affected: Everweb (Datacenter SE1 (Stockholm), Datacenter SE2 (Stockholm)).