Do Staging Procedures Need a Rethink?

“Has everyone started owning discussions with their CIO/CEO about transferring back to an in-household mail server? I advocate for it”

Specified the scale of its consumer base and with a agreement truly worth up to $10 billion in the bag to operate the back-conclude of a superpower’s navy, Microsoft could possibly want to start out contemplating about how it can set up a staging process for its Azure cloud that enables it to deploy alterations and reliably roll back these alterations when factors break.

(We know, it is uncomplicated to say so from a protected distance…)

Redmond was at it once again late Monday, knocking an (apparently significant) “subset of consumers in the Azure General public and Azure Government clouds” offline for 3 hrs with swathes of users globally encountering problems carrying out authentication functions multiple services were being impacted, which includes Microsoft 365.

The enterprise blamed the challenge on a “recent configuration change [that] impacted a backend storage layer, which caused latency to authentication requests.” (Read through, users could not login to Groups, Azure and extra for hrs mainly because of the snafu).

A whole root result in assessment is pending. (We will update this piece when we see it). The blockage was felt for users from 22:twenty five BST on Sep 28 2020 to 01:23 BST.

The challenge will come a fortnight after a protracted outage in Microsoft’s United kingdom South area triggered by a cooling program failure in a information centre. With temperatures mounting, automatic programs shut down all community, compute, and storage methods “to shield information durability” as engineers rushed to take handbook control.

Previously this thirty day period meanwhile Gartner stated it “continues to have fears related to the overall architecture and implementation of Azure, even with resilience-concentrated engineering attempts and improved provider availability metrics throughout the previous year”.

Microsoft Azure CTO Mark Russinovich in July 2019 stated that Azure had fashioned a new High-quality Engineering workforce within just his CTO business office, operating along with Microsoft’s Site Dependability Engineering (SRE) workforce to “pioneer new approaches to produce an even extra responsible platform” subsequent buyer worry at a string of outages.

He wrote at the time: “Outages and other provider incidents are a obstacle for all public cloud vendors, and we continue on to improve our knowledge of the elaborate approaches in which things such as operational processes, architectural designs, components troubles, software package flaws, and human things can align to result in provider incidents.

“Has everyone started owning discussions with their CIO/CEO about transferring back to an in-household mail server? I advocate for it” a single frustrated consumer mentioned on a world Outages mailing record meanwhile… If cloud is your compressed audio stream that you are not confident you personal, it may well not be extended just before in-household mail servers become the vintage high-quality vinyl of the IT earth old, but quite a great deal back in desire.

Stranger factors have occurred.