Rackspace Hosted Exchange Outage Due to Security Incident
Rackspace hosted Exchange experienced a catastrophic outage starting December 2, 2022 and is nevertheless ongoing as of 12:37 AM December 4th. Initially described as connectivity and login issues, the guidance was at some point up to date to announce that they had been working with a safety incident.
Rackspace Hosted Trade Challenges
The Rackspace technique went down in the early early morning hrs of December 2, 2022. To begin with there was no term from Rackspace about what the trouble was, a great deal fewer an ETA of when it would be fixed.
Shoppers on Twitter reported that Rackspace was not responding to support emails.
This has been quite the day with #Rackspace. Every hosted trade shopper has been down for 14 several hours or so. Assist is not looking through/responding to tickets. Updates are unhelpful.
I am involved now that they fell victim to one thing undesirable like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace customer privately messaged me over social media on Friday to relate their practical experience:
“All hosted Trade consumers down above the previous 16 hours.
Not certain how several organizations that is, but it’s significant.
They are serving a 554 long hold off bounce so people today emailing in are not conscious of the bounce for numerous hrs.”
The official Rackspace status web site offered a operating update of the outage but the first posts had no information and facts other than there was an outage and it was remaining investigated.
The initial official update was on December 2nd at 2:49 AM:
“We are investigating an challenge that is affecting our Hosted Trade environments. Extra particulars will be posted as they become obtainable.”
Thirteen minutes later Rackspace commenced calling it a “connectivity concern.”
“We are investigating reviews of connectivity issues to our Trade environments.
Buyers may perhaps practical experience an mistake on accessing the Outlook Internet Application (Webmail) and syncing their e-mail customer(s).”
By 6:36 AM the Rackspace updates described the ongoing dilemma as “connectivity and login issues” then later that afternoon at 1:54 PM Rackspace announced they have been nevertheless in the “investigation phase” of the outage, continue to trying to determine out what went erroneous.
And they ended up nonetheless contacting it “connectivity and login issues” in their Cloud Office environments at 4:51 PM that afternoon.
Rackspace Endorses Migrating to Microsoft 365
Four hrs afterwards Rackspace referred to the scenario as a “significant failure”and commenced providing their consumers cost-free Microsoft Trade Program 1 licenses on Microsoft 365 as a workaround until they comprehended the issue and could convey the method back again on the web.
The formal steering stated:
“We expert a sizeable failure in our Hosted Exchange ecosystem. We proactively shut down the surroundings to steer clear of any additional problems when we carry on work to restore assistance. As we keep on to get the job done through the root induce of the concern, we have an alternate solution that will re-activate your means to deliver and get emails.
At no expense to you, we will be delivering you obtain to Microsoft Trade Program 1 licenses on Microsoft 365 right up until even further discover.”
Rackspace Hosted Exchange Stability Incident
It was not till virtually 24 hrs later on at 1:57 AM on December 3rd that Rackspace officially introduced that their hosted Exchange assistance was struggling from a security incident.
The announcement even more exposed that the Rackspace technicians had run down and disconnected the Exchange surroundings.
Rackspace posted:
“After further analysis, we have determined that this is a protection incident.
The regarded impression is isolated to a part of our Hosted Trade system. We are getting important steps to consider and shield our environments.”
Twelve several hours later that afternoon they up to date the position site with additional information that their safety staff and outside specialists ended up however functioning on resolving the outage.
Was Rackspace Assistance Influenced by a Vulnerability?
Rackspace has not launched information of the protection party.
A protection occasion frequently involves a vulnerability and there are two extreme vulnerabilities presently in the wile that had been patched in November 2022.
These are the two most latest vulnerabilities:
- CVE-2022-41040
Microsoft Trade Server Server-Aspect Request Forgery (SSRF) Vulnerability
A Server Facet Ask for Forgery (SSRF) assault enables a hacker to go through and change knowledge on the server. - CVE-2022-41082
Microsoft Trade Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is a single in which an attacker is capable to operate malicious code on a server.
An advisory released in Oct 2022 described the impact of the vulnerabilities:
“An authenticated distant attacker can execute SSRF attacks to escalate privileges and execute arbtirary PowerShell code on susceptible Microsoft Exchange servers.
As the attack is targeted from Microsoft Exchange Mailbox server, the attacker can likely obtain access to other sources by means of lateral motion into Trade and Lively Listing environments.”
The Rackspace outage updates have not indicated what the precise challenge was, only that it was a protection incident.
The most current standing update as of December 4th mentioned that the services is nevertheless down and shoppers are encouraged to migrate to the Microsoft 365 service.
Rackspace posted the adhering to on December 4, 2022 at 12:37 AM:
“We go on to make development in addressing the incident. The availability of your service and stability of your details is of substantial worth.
We have dedicated intensive interior resources and engaged entire world-course exterior skills in our initiatives to lessen negative impacts to clients.”
It is attainable that the above observed vulnerabilities are associated to the protection incident influencing the Rackspace Hosted Trade company.
There has been no announcement of regardless of whether client details has been compromised. This function is however ongoing.
Featured picture by Shutterstock/Orn Rin
