Summary

The Episerver NuGet feed (nuget.episerver.com) is the central source for downloading packages for releases and upgrades to the Episerver platform. On Monday the 11th of March 2019 we had an event impacting the functionality of the Episerver NuGet site, and it was not possible to access it to download any packages. This report describes additional details around the event.

Incident duration: 256 minutes

Timeline

March 11th, 2019

17:49 CET - First alert is triggered by global monitoring system.

17:53 CET - Ticket was raised to Managed Services to start investigation.

18:27 CET - Server reboot and IIS reset performed but without desired outcome to mitigate the issue and troubleshooting efforts are continued.

20:05 CET - Escalation to on-duty Reliability Engineering (third-line) and troubleshooting efforts are initiated.

20:14 CET - On-duty confirms that there is no issue with the infrastructure nuget.episer.com is hosted on but issues with the application itself.

20:18 CET - Escalation to Software Engineering for further investigation.

21:59 CET - Identified new uploaded corrupted NuGet package which was removed.

22.05 CET - Service fully operational

Root cause

The root cause was the uploading to the feed of corrupt NuGet packages, which caused the site to break.

Corrective and preventative measures

A package verification procedure will be implemented for checking packages before upload to the actual live feed.

Final Words

We apologize for the impact to affected customers. We have a strong commitment to delivering high availability for our services and we will do everything we can to learn from the event and to avoid a recurrence in the future.