Network connectivity issues in Sweden
Incident Report for Optimizely Service
Postmortem

SUMMARY

Between 2020-06-04 12:08 UTC and 2020-06-04 12:32 UTC Everweb Datacenter SE1 experienced an outage.

What triggered this issue was an incorrect configuration change that was pushed to the datacenter switches. As soon as this was identified, the applied changes were reverted and service was fully operational at 12:32 UTC.

TIMELINE

2020-06-04 12:08 UTC– Alerts triggered and investigation initiated

2020-06-04 12:27 UTCSTATUSPAGE updated

2020-06-04 12:30 UTC – Cause identified and a configuration change is reverted

2020-06-04 12:32 UTC – Alerts cleared and Service Operational.

2020-06-04 12:40 UTCIncident closed

ANALYSIS

The root cause of this incident was due to a human error when performing a configuration change to Datacenter switches as part of a decommission procedure.

IMPACT

During the event, clients located in the SE1 datacenter would have experienced network timeouts (5xx-errors) when trying to connect to their service.

CORRECTIVE MEASURES

Short-term mitigation

  • Configuration change was reverted to restore service as soon as possible.

Long-term mitigation

  • We are constantly reviewing our internal process in an effort to identify and drive opportunities for continuous improvement. To improve our processes and to mitigate the risk of this happening again, we will improve documentation and add additional sanity checks to avoid similar events in the future.

FINAL WORDS

We place the utmost importance and pride on achieving and sustaining the highest level of availability for our customers and we regret any disruption in service you have experienced. We continue to work tirelessly to ensure any and all service disruptions are prevented and or mitigated and we will use this incident to further these efforts to help ensure you receive a reliable and positive experience.

Posted Jun 16, 2020 - 08:53 UTC

Resolved
This incident has been resolved.
Posted Jun 04, 2020 - 14:13 UTC
Monitoring
The issue has been identified and a fix implemented.

We are monitoring the results.
Posted Jun 04, 2020 - 12:34 UTC
Investigating
We are currently investigating a network connectivity issue in one of our Swedish Datacenters.

We apologize for the inconvenience and will share an update once we have more information.
Posted Jun 04, 2020 - 12:28 UTC
This incident affected: Everweb (Datacenter SE1 (Stockholm)).