sf-deity is a data-recovery testing solution presented by the Cash App Salesforce Team at Block. This tool allows a Salesforce team to simulate data loss at a massive scale to determine necessary remediation steps in the event of catastrophe. This tool introduces the use of a Queueable Jobs Framework (provided by DX_QueueableJobChain.cls and DX_ChainedQueueable.cls) that allows a developer flexibility surrounding the limits of the Queueable Interface provided by Salesforce by enqueueing the next job when the current job is completed.
This code will cause mass data loss through modification and hard deletion, and, as a result, there are failsafes to keep it from running in a production environment. This code should never be deployed to production. If you do run it in prod, like is the case with all Salesforce development, there will be a paper trail and there will likely be hefty consequences. That being said, make sure you have a backup solution in place before this code is run in a sandbox environment or be prepared to refresh from production. This code is recommended for use in a full or partial copy sandbox where development work is not completed so that it does not disrupt day to day operations.
-
Related sObjects and child sObjects will be corrupted if there are Master-Detail relationships or roll-up summary fields. This is expected.
-
There may be automations or validations that prevent some records from being modified or deleted. THIS IS A GOOD THING! If you don't see exactly the number of records corrupted that you specified, it means your org is doing its job to prevent data loss.
-
In order to fully restore records from a backup, all automations and validations will need to be turned off. This is achievable through a hierarchical custom setting. This remediation should be completed before running this tabletop exercise on a large scale.
-
If there are picklist values on an sObject that are no longer used, they should be Deactivated rather than Deleted. This is so a backup service can restore historical data. If these picklist values are deleted, there will be an error on restore attempt.
-
There are some sObjects, such as
FeedItem, that do not allow for aggregate queries per SOQL Object Limits and Limitations. This functionality relies on an aggregate query, so these items are not corruptable. -
NOT EVERYTHING CAN BE RESTORED. Check with your backup provider to ensure that the sObjects you are testing against can be restored.
After taking remediation steps, such as adding bypasses for automations and validation rules for the bot-user that your backup tool utilizes, you are ready to complete your tabletop exercise.
In order to complete this exercise, the running user must have Setup access in Sandbox, as well as the Author Apex permission to ensure Apex and necessary components can be deployed.
To deploy the code to a Sandbox environment, complete the following steps:
-
Ensure that Git, Visual Studio Code, the Salesforce Extension Pack for VSCode, and the Salesforce CLI and the Salesforce are installed and configured for Apex development.
-
Clone the repository via the following command in the desired directory, likely a directory away from your internal codebase to ensure that the code will not be promoted to production:
git clone https://github.com/block/sf-deity.git
-
Connect your VSCode to your desired Salesforce Sandbox via
CMD+ ShiFT + P(CTRL + SHIFT + Pon Windows) and selectingSFDX: Authorize An Org. The project will default to usingtest.salesforce.comas the default login. -
Login to your desired Sandbox.
-
Run a validated deployment by issuing the following command:
sf project deploy validate --source-dir force-app
-
Depending on the size of your org and the average time to Run All Tests, this may take a while! As written, this code is designed to fail tests in Production, meaning this validation will fail in a Developer Edition Org.
-
If all goes well, there should be 11 Test Failures off the bat - this is expected. This is because we have not deployed and assigned the permissions necessary to run this code successfully. This is to ensure that there are multiple layers of protection against deploying and running this code in a production environment. The errors you should see should be along the lines of:
You do not have the permissions necessary to complete this action.
-
If you see an error about not running in prod: do not pass go, do not collect $200, stop what you're doing. This action can cause irreversible data loss, even if you have a backup solution.
-
Once you have validated and ensured that the only test failures are due to permissions, you are good to deploy with the following command:
sf project deploy start --source-dir force-app
-
Assign the Permission Set
Corrupt Data Via SF-Deityto the user that will be running the code (likely yourself) in the Salesforce UI. I recommend assigning with an expiration date of 1 week. -
Revalidate with the following command to ensure that the 11 failed tests are now passing:
sf project deploy validate --source-dir force-app
-
Et voila. You are ready to configure!
-
Through the use of Salesforce Custom Metadata records on the
Data Corruption ObjectCMDT, a developer can specify a number of Standard or Custom sObjects to corrupt, Standard or Custom number or text fields to corrupt on each sObject, whether to corrupt by deletion, modification, or a 50-50 split of each, and a percentage of each sObject to corrupt.
-
For example, in the sample data as deployed, there are three Standard sObjects (
Account,Case, andContact) and one Custom sObject (My_Custom_Object__c). Note that the CMDT record requires leaving__coff of the custom sObject label per Salesforce API name requirements, so it is appended in the code based on whether the Custom Object box is checked. Also note that the__cis required on a custom field, as is indicated in the example metadata record. -
Each of the example sObjects is currently specified to be corrupted by deletion and modification at 100% (with a 50-50 split between deletion and modification, as both options are selected). Only objects with the Active box checked will be corrupted, and they will be corrupted in the order specified in the Corruption Order field on the CMDT records, with nulls being corrupted last. After you have examined and noted the format of the example metadata records, you can modify or delete them.
-
Once you have deployed the folder structure and configured your Custom Metadata records, the code is ready to run.
-
In order to run the code, you will enter the following Anonymous Apex snippet, substituting your own email address:
new DX_DataCorruptionQueueableChain('[email protected]').runJob();
-
Once each job is complete, you will receive an email notifying you that the job is complete so you don't have to keep checking your Apex Job Logs. In order to receive this email, ensure deliverability is on and there is a verified org-wide email address set. There is a risk of sending emails out when running any job, so ensure your org is compatible with turning on deliverability for the duration of this exercise.
-
Once the final job is complete, you are good to test your recovery tooling.
This project was recognized by Own Company for their Inaugural Innovator of the Year award in 2024. It has since been iterated on and improved upon so other teams can prepare their orgs for the worst.
| Resource | Description |
|---|---|
| CODEOWNERS | Outlines the project lead(s) |
| CODE_OF_CONDUCT.md | Expected behavior for project contributors, promoting a welcoming environment |
| CONTRIBUTING.md | Developer guide to build, chat, discuss, contribute, and file issues |
| LICENSE | Apache License, Version 2.0 |