Rackspace hosted Exchange suffered a devastating blackout beginning December 2, 2022 and is still continuous since 12:37 AM December 4th. Initially referred to as connection and login issues, the assistance was eventually upgraded to announce that they were dealing with a security occurrence.
Rackspace Hosted Exchange Issues
The Rackspace system went down in the morning hours of December 2, 2022. Initially there was no word from Rackspace about what the problem was, much less an ETA of when it would be dealt with.
Consumers on Buy Twitter Verified reported that Rackspace was not reacting to support e-mails.
This has actually been quite the day with #Rackspace. Every hosted exchange customer has actually been down for 14 hours or two. Assistance isn’t reading/responding to tickets. Updates are unhelpful.
I am worried now that they came down with something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace customer independently messaged me over social networks on Friday to relate their experience:
“All hosted Exchange clients down over the previous 16 hours.
Not exactly sure the number of business that is, but it’s substantial.
They’re serving a 554 long delay bounce so people emailing in aren’t aware of the bounce for numerous hours.”
The main Rackspace status page offered a running upgrade of the blackout however the initial posts had no info besides there was a blackout and it was being investigated.
The very first authorities upgrade was on December second at 2:49 AM:
“We are investigating an issue that is impacting our Hosted Exchange environments. More details will be published as they become available.”
Thirteen minutes later Rackspace began calling it a “connectivity concern.”
“We are investigating reports of connectivity issues to our Exchange environments.
Users may experience an error upon accessing the Outlook Web App (Webmail) and syncing their e-mail customer(s).”
By 6:36 AM the Rackspace updates described the continuous problem as “connectivity and login issues” then later that afternoon at 1:54 PM Rackspace revealed they were still in the “investigation stage” of the blackout, still trying to find out what went wrong.
And they were still calling it “connectivity and login issues” in their Cloud Workplace environments at 4:51 PM that afternoon.
Rackspace Recommends Migrating to Microsoft 365
4 hours later on Rackspace referred to the scenario as a “significant failure”and started using their consumers complimentary Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround until they understood the issue and might bring the system back online.
The official assistance stated:
“We experienced a significant failure in our Hosted Exchange environment. We proactively closed down the environment to prevent any more concerns while we continue work to restore service. As we continue to overcome the root cause of the concern, we have an alternate solution that will re-activate your ability to send and get e-mails.
At no cost to you, we will be supplying you access to Microsoft Exchange Strategy 1 licenses on Microsoft 365 up until further notification.”
Rackspace Hosted Exchange Security Incident
It was not up until almost 24 hours later on at 1:57 AM on December 3rd that Rackspace formally revealed that their hosted Exchange service was suffering from a security occurrence.
The statement even more revealed that the Rackspace service technicians had powered down and detached the Exchange environment.
“After more analysis, we have actually determined that this is a security event.
The recognized impact is separated to a part of our Hosted Exchange platform. We are taking essential actions to examine and secure our environments.”
Twelve hours later that afternoon they upgraded the status page with more details that their security team and outside professionals were still dealing with resolving the outage.
Was Rackspace Service Impacted by a Vulnerability?
Rackspace has actually not launched details of the security event.
A security event normally involves a vulnerability and there are two serious vulnerabilities presently in the wile that were patched in November 2022.
These are the two most current vulnerabilities:
Microsoft Exchange Server Server-Side Demand Forgery (SSRF) Vulnerability
A Server Side Request Forgery (SSRF) attack permits a hacker to read and alter data on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an enemy is able to run destructive code on a server.
An advisory published in October 2022 described the effect of the vulnerabilities:
“A verified remote assailant can carry out SSRF attacks to escalate opportunities and carry out arbtirary PowerShell code on susceptible Microsoft Exchange servers.
As the attack is targeted versus Microsoft Exchange Mail box server, the opponent can potentially gain access to other resources via lateral movement into Exchange and Active Directory site environments.”
The Rackspace outage updates have actually not suggested what the specific problem was, just that it was a security incident.
The most current status update as of December 4th stated that the service is still down and consumers are motivated to move to the Microsoft 365 service.
Rackspace posted the following on December 4, 2022 at 12:37 AM:
“We continue to make development in dealing with the event. The availability of your service and security of your data is of high importance.
We have actually devoted substantial internal resources and engaged first-rate external competence in our efforts to decrease negative effects to consumers.”
It’s possible that the above noted vulnerabilities belong to the security event affecting the Rackspace Hosted Exchange service.
There has actually been no statement of whether consumer details has actually been jeopardized. This event is still ongoing.
Featured image by Best SMM Panel/Orn Rin