Our Business Growth Was Almost Derailed After Losing a Day’s Worth of Customer Information

In this anonymous story, originally published in 2019, the CIO of a financial technology company recounts his experience with a user-inflicted Salesforce data loss and how it impacted his entire company. Then, this CIO explains how the company leveraged OwnBackup to enable accelerated business growth by safeguarding critical business information.

I am the CIO for a large, US financial technology company that helps everyday consumers manage their investments without a stock broker. We’re dedicated to meeting the needs of our customers and are, therefore, in a constant state of digital transformation. We've implemented numerous IT-related projects, from artificial intelligence to marketing automation, to accelerate our business growth.

My team owns our Salesforce platform, in which we store our lead and customer account information. Our entire application runs on top of Salesforce, and the data is refreshed whenever a customer logs in to their account.

Upon my return from a business trip, I was bombarded with several urgent emails from multiple departments. I was also forwarded hundreds of complaints from customers who were claiming they were no longer able to log in to their accounts. I immediately called my IT team in for an emergency meeting.

During a nightly batch job with an ETL tool, a bug got into the system. So, instead of updating 38,000 customer records in Salesforce, 38,000 loan collateral records were deleted. We had built our entire product on top of a CRM system that we had not yet equipped with the proper data protection foundation.

Data integrity is required for our company to deliver uninterrupted service and an exceptional customer experience. After this incident, our application not only stopped working for 38,000 customers but also had massive repercussions on my company's day-to-day departmental operations.

After further assessing the situation, my IT team had some good news and bad news. The good news was that we were backing up with the Salesforce Weekly Export, which produced .CSV files containing our data. My Salesforce admin had these saved on his hard drive. The bad news was that we’ve never had to recover lost data using the Weekly Export files. After examining the process, my team informed me that it could take weeks to identify all of the accounts and relationships that were lost. Even then, we’d never be able to completely recover all of the lost data.

Marketing has spent months developing customer engagement campaigns with personalized emails, mobile messages, social experiences, and advertisements. After the customer data loss, the marketing team’s campaign efforts have led to a jump in customer account cancellations and an even faster spread of bad publicity. Even worse, our enraged customers have fueled the fire on social media, encouraging new prospects to seek out competitors. An explanation email and press release seemed imminent, followed by an apology tour by our CEO.

As we attempted to recover the lost customer accounts, finance was hounding us for customer payment information. Besides being unable to properly track which customers were asking to terminate their accounts, finance had no idea which customers would still need to pay for their subscriptions.

Customer service had to man the front lines of this un-winnable battle. With no information on 38,000 customers, they couldn't even begin to respond to the bombardment of complaint calls and angry customer emails. Their hands were tied until we could recover the missing customer account information.

A data loss of this magnitude dropped us within firing range of lawsuits and regulatory compliance fines. Specifically, SEC 17a-4, which requires companies to retain, for three to six years, originals and copies of data and communications related to their business. As a fintech company, we are responsible for protecting the confidentiality, integrity, and availability of customer information. Failure to do so puts both our company and our customers at major risk.

Just when I thought things couldn’t get worse, my CEO called me into her office. She was in the midst of preparing for our annual board meeting and numbers in her Salesforce reports weren’t making sense. When I explained the situation to her and how it was impacting our entire company, I swear her head looked like it was about to explode. A data loss of this magnitude had the potential to cause a complete organizational failure. Understandably, she wouldn’t take “we don’t know how to recover the lost data” for an answer. If I couldn’t recover these 38,000 customer accounts before the end of the week, I would be fired.

I was at a loss for words. My entire company was about to be derailed by one data loss incident.

Now what if I told you there was a devoted team on a mission to help companies accelerate business growth and minimize business risk? Let’s replay the above story again and I’ll show you what I mean…

I am the CIO for a large, US financial technology company that helps everyday consumers manage their investments without a stock broker. We’re dedicated to meeting the needs of our customers and are, therefore in a constant state of digital transformation. From artificial intelligence to marketing automation, we’ve implemented numerous IT-related projects to accelerate our business growth.

My team owns our Salesforce platform, in which we store our lead and customer account information. Our entire application runs on top of Salesforce and the data is refreshed whenever a customer logs in to their account.

Early one morning, one of my Salesforce admins checked his email and noticed an unusual OwnBackup Smart Alert…38,000 removed customer records. Without hesitation, he signed into OwnBackup, used OwnBackup’s visual graphs to identify exactly when the data loss occurred, and recovered the lost data and attachments in less than 24 hours.

By identifying the exact point in time when the data loss occurred, my IT team was also able to figure out what caused the data loss. During a nightly batch job with an ETL tool, a bug got into the system. They promptly resolved the issue. Then, using the OwnBackup API, our developers coded an automatic backup to kick off right before each nightly batch job.

OwnBackup has empowered our company to minimize data loss risk and costs, comply with regulations, and unleash digital transformation. Furthermore, Blockchain Verify, helps us meet compliance and regulatory objectives for electronic storage, record-keeping, and backup integrity of regulated Salesforce records.

 Learn more about how OwnBackup supports the compliance needs of financial organizations like the one in this anonymous CIO story.