This Sunday marks one year since the May 17 service disruption that forced Salesforce to block access to thousands of its customers for just over 15 hours. To put that into perspective, a single hour of server downtime can cost enterprises between $300,000 and $400,000.
The incident stemmed from the deployment of a database script in Pardot that inadvertently changed user profiles and permissions in some orgs. To protect their customers, Salesforce blocked access to all instances that contained affected customers until they were able to remove the inadvertent permissions.
While Salesforce is and continues to be the world’s most secure and reliable SaaS platform, this incident caused many customers to consider their backup and recovery strategy, and in particular, the accessibility of their data. In the unlikely event of another service disruption, how would they be able to access their critical customer information?
In this post we’ll examine why data accessibility is such an important consideration in your backup and recovery strategy, how Own helped customers during last year’s service disruption, and some key tips moving forward.
Why Data Accessibility Is So Important
Imagine for a moment you’re sitting at home and your power suddenly goes out. You can no longer access the Internet, TV, appliances, and all of the things that make your house run. If you’re in a part of the country where this happens a lot (or just an ultra-prepared type of person) you might invest in a generator. This way, you can still power your house during an outage.
This is the same reason why storing data backups outside of Salesforce is a best practice. If Salesforce has an outage, you’ve got redundancy in your data. However, where you store it outside of Salesforce is just as important. Backing the data up to a personal hard drive or S3 server isn’t the answer. Yes, it’s accessible outside of Salesforce, but the number of people who have access will be limited, and providing access to those who might need the data poses a major security risk.
Own customers have access to their data and metadata at all times through a controlled user interface should Salesforce become temporarily unavailable. We give customers the ability to search their entire backup archive to easily find specific records in compliance or recovery situations. You can also export to SQL anytime you need, in order to maintain legacy business continuity and data recovery procedures.
How Customers Leveraged Own During Last Year’s Service Disruption
Around 12:30 PM EST on May 17th, Own received its first support call related to the permission error. And then another. And then another. And the calls kept coming, until we had helped 64 of our customers recover their profiles and permission sets. Many customers were able to use the metadata restore tool directly through the Own user interface to restore the permissions themselves. Own was able to help its customers minimize the impact of the disruption in several ways.
Before Salesforce Access Was Blocked
Own customers were among the first to know about the permission issue. How? They received a Smart Alert that there were some unusual changes within their metadata. Own Smart Alerts notify users when data or metadata, including profiles and user permissions, is changed, deleted, or corrupted, based on their set rules or statistical outliers. With the help of Own, customers who had set up Smart Alerts were able to restore user profiles and permissions before Salesforce blocked access.
While Salesforce Access Was Blocked
When Salesforce became aware of the inadvertent change to profiles and permissions, they blocked access to all orgs that were potentially impacted. Since Own stores its customers’ backups outside of Salesforce, users were still able to access their data through the Own user interface. Many customers used Own’s data export tool to export data to their on-premise database, as well as Own Sandbox Seeding and Replicate to move their critical data to a sandbox.
After Salesforce Access Was Unblocked
When Salesforce restored access to admins, but not profiles and permissions, they initially advised customers to manually restore profiles and permissions from a sandbox. Salesforce would run scripts to restore profiles and permissions over the coming days. Meanwhile, Own’s Customer Success team helped customers quickly recover profile object permissions and audit any data changes or deletions that took place during elevated permissions. With the help of the entire Own team, our customers were able to have peace of mind and maintain business continuity throughout the incident.
3 Lessons We Took Away from the May 17th Service Disruption
For many customers, the incident shined a spotlight on the need for a comprehensive backup and recovery solution that sits outside of Salesforce. Even for us at Own, the outage further emphasized several things:
Having a documented disaster recovery plan is a must. According to the 2020 State of Salesforce Data Protection survey results, Salesforce customers said that they could not maintain business continuity if they lost their most critical Salesforce data, which included customer information, accounts, opportunities, and contracts.
Losing any of this data can impact your entire company with increased labor costs, data recovery fees, lost reputation, revenue impact, compliance fines, and loss in productivity. If you’ve identified Salesforce as a critical component of your business, then it’s important to define your disaster recovery plan to account for any Salesforce data loss or corruption.
Metadata backup is critical. Over the course of the service disruption, many Salesforce customers using the Weekly Export found out the hard way that it does not include metadata. Not only does Own backup all of a customer’s data, metadata, and attachments, our Compare tool allows customers to review data and metadata side-by-side to see what has been added, deleted, or changed.
Maintain Business Continuity with Own
In the unlikely event of another Salesforce outage, will you be able to access your data? What about the simple mistakes and human errors that happen each and every day?
With the upcoming retirement of Salesforce’s Data Recovery service, now is the perfect opportunity to take a step back and put a comprehensive backup and recovery plan in place to ensure that your Salesforce data has the same level of protection as your other critical systems.
