Feature Request: Resource locks

[bricrsa]

Given that we would like to encourage CI/CD, it would be a useful to have some protection on key resources, to discourage them from being deleted by rogue pipelines.

Suggest a resource lock may be a good way to achieve this, particularly for storage accounts and key vaults.

[marvinbuss]

Thanks @bricrsa for submitting this feature request. For Key Vaults we are using Purge protection and Soft Delete. That is certainly a better way compared to resource locks. Do you agree?
For storage accounts, we will evaluate whether we should add this to the reference implementation. 👍
Would it be ok from your standpoint to just add this to the storage accounts?

[bricrsa]

Agree that this is best placed for storage accounts and soft delete and purge protection is better for AKV.

[AnalyticJeremy]

Preventing customers from accidentally deleting their storage accounts is a very good idea, and placing a cannot-delete lock on the storage accounts seems like a logical way to achieve this. However, these locks have unexpected effects that make it very difficult to use them.

Description of Issue

A cannot-delete lock does, in fact, prevent a storage account from being deleted. But it also prevents the deletion of the storage account's child ARM resources. These include:

• RBAC Assignments: If a storage account has a cannot-delete lock, you can still add RBAC assignments to the account. However, you can't remove assignments. This makes it very difficult to properly manage permissions on your storage accounts.
• Event Subscriptions: In Azure, you can create event subscriptions on a storage account so your code can react whenever a blob is created, deleted, etc. These event subscriptions are used by Azure Data Factory for the event-based pipeline triggers. However, this functionality is broken by cannot-delete locks on a storage account because ADF is unable to recreate the event triggers whenever the pipelines are updated.

Alternative Solutions

Preventing data loss is obviously very important so we still need a way to help customers secure their data platforms. I have two alternative solutions that protect data without using cannot-delete locks on the storage accounts.

• Storage Account Recovery: Customers do not have to worry about accidental account deletion as much as they may think because modern storage accounts can be recovered for 14 days after they are deleted. This feature is available on all storage accounts created with ARM; you don't have to activate it. This means that if a customer accidentally deletes their storage account, all they have to do is recover it. This approach can be paired with the "Soft Delete for Containers" feature to provide the same peace-of-mind at the container level.
• Immutability Policy: The approach above protects customers' data from being permanently lost. In practice, though, it would be disruptive to the business if a storage account was deleted and it took a few hours for the cloud admin team to recover it. One way to prevent a storage account from being deleted in the first place is to use an immutability policy. If a storage account contains any objects with an immutability policy, it cannot be deleted. Any attempts to delete the storage account will fail.
  However, immutability policies place a lot of restrictions on the data that would prevent many common data management patterns. Therefore, we don't want to put the immutability on a container that we will actually be using as part of our data platform. Instead, you can create a container within the storage account that is solely for the purpose of hosting the immutability policy. It can be helpful to name this container something like "immutable-account-lock" so that its purpose is clear.

Solution for Enterprise-Scale Analytics

Since Enterprise-Scale Analytics is meant to provide best practice guidance, we should rely heavily on the Storage Account Recovery feature of Azure. While using an immutability policy will work, it feels a little bit like a hack so I would not recommend that we include it as part of the deployment templates. Instead, we can include it in the documentation and make sure that CSA's at Microsoft are aware of this option.

[marvinbuss]

The above mentioned Storage Account feature "Soft Delete for Containers" is already enabled on all data lakes that are deployed as part of the Data Landing Zone.