Deploying blob storage with CMK and rbac-enabled keyVault fails due to race condition

[torfmaster]

Bicep version
Bicep CLI version 0.33.93 (7a77c7f)

Describe the bug

Deploying blob storage with CMK and rbac-enabled keyVault fails due to a race condition. It seems that role assignments are applied inside storage managed identity only after a small delay. An explicit wait (using a deployment script) works around the issue (but is, of course, fragile).

To Reproduce

• Deploy an rbac-enabled keyVault containing a CMK
• give an MI permissions (rbac) to the keyVault
• create a key
• deploy blob storage using the key and with the MI as identity

The deployment will fail with the error message

[{"code":"KeyVaultAuthenticationFailure","message":"The operation failed because of authentication issue on the keyvault.

Additional context

• the issue does not occur if access policies are used instead of rbac on the keyVault
• the deployment is successful if the deployment is retried

This code does not work:

@description('Specifies the name of the environment.')
param environment string

@description('Specifies the name of the deployment.')
@minLength(1)
@maxLength(10)
param customerAsset string

@description('Set soft delete retention in days for key vault')
param softDeleteRetentionInDays int

@description('We only use 2 letters as a unique suffix so that we adhere to the Naming Rules and Character Restrictions, see https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/resource-name-rules')
var uniqueSuffix = substring(uniqueString(resourceGroup().id), 0, 2)
var region = 'gwc'
var storageAccountName = 'st${customerAsset}${environment}${region}${uniqueSuffix}'
var storageIdentityName = 'id-${customerAsset}-${environment}-${region}-storage${uniqueSuffix}'
var storageAccountKeyName = 'key-${customerAsset}-${environment}-${region}-storage${uniqueSuffix}'
var keyVaultName = 'kv-${customerAsset}-${environment}-${region}${uniqueSuffix}'

resource storageIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2022-01-31-preview' = {
  name: storageIdentityName
  location: resourceGroup().location
  tags: resourceGroup().tags
}

resource keyVault 'Microsoft.KeyVault/vaults@2023-07-01' = {
  name: keyVaultName
  location: resourceGroup().location
  tags: resourceGroup().tags
  properties: {
    tenantId: subscription().tenantId
    sku: {
      name: 'standard'
      family: 'A'
    }
    enablePurgeProtection: true
    enableSoftDelete: true
    softDeleteRetentionInDays: softDeleteRetentionInDays
    enableRbacAuthorization: true
  }
}

@description('This is the built-in Key Vault Crypto Service Encryption User role ')
resource keyVaultCryptoServiceEncryptionUserRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
  name: 'e147488a-f6f5-4113-8e2d-b22465e65bf6'
}

resource keyVaultCryptoServiceEncryptionUserAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(keyVaultCryptoServiceEncryptionUserRole.id, storageIdentity.id, keyVault.id)
  scope: keyVault
  properties: {
    description: 'Allow access to perform cryptographic operations using keys.'
    roleDefinitionId: keyVaultCryptoServiceEncryptionUserRole.id
    principalId: storageIdentity.properties.principalId
    principalType: 'ServicePrincipal'
  }
}

@description('This is the built-in Key Vault Reader role. See https://docs.microsoft.com/azure/role-based-access-control/built-in-roles/security#key-vault-reader')
resource keyVaultReaderRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
  name: '21090545-7ca7-4776-b22c-e363652d74d2'
}

resource keyVaultReaderRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(keyVaultReaderRole.id, storageIdentity.id, keyVault.id)
  scope: keyVault
  properties: {
    description: 'Read metadata of key vaults and its certificates, keys, and secrets.'
    roleDefinitionId: keyVaultReaderRole.id
    principalId: storageIdentity.properties.principalId
    principalType: 'ServicePrincipal'
  }
}

resource storageAccountKey 'Microsoft.KeyVault/vaults/keys@2023-07-01' = {
  parent: keyVault
  name: storageAccountKeyName
  tags: resourceGroup().tags
  properties: {
    kty: 'RSA'
    keySize: 4096
    rotationPolicy: {
      lifetimeActions: [
        {
          trigger: {
            timeAfterCreate: 'P1Y'
          }
          action: {
            type: 'rotate'
          }
        }
        {
          trigger: {
            timeBeforeExpiry: 'P30D'
          }
          action: {
            type: 'notify'
          }
        }
      ]
    }
    attributes: {
      enabled: true
    }
  }
}

resource storageAccount 'Microsoft.Storage/storageAccounts@2023-05-01' = {
  name: storageAccountName
  location: resourceGroup().location
  tags: resourceGroup().tags
  sku: {
    name: 'Standard_LRS'
  }
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: { '${storageIdentity.id}': {} }
  }
  kind: 'StorageV2'
  properties: {
    accessTier: 'Hot'
    supportsHttpsTrafficOnly: true
    minimumTlsVersion: 'TLS1_2'
    encryption: {
      services: {
        blob: {
          enabled: true
          keyType: 'Account'
        }
        file: {
          enabled: true
          keyType: 'Account'
        }
        queue: {
          enabled: true
          keyType: 'Account'
        }
        table: {
          enabled: true
          keyType: 'Account'
        }
      }
      keySource: 'Microsoft.Keyvault'
      identity: {
        userAssignedIdentity: storageIdentity.id
      }
      keyvaultproperties: {
        keyname: storageAccountKey.name
        keyvaulturi: keyVault.properties.vaultUri
      }
    }
  }
  dependsOn: [
    keyVaultReaderRoleAssignment
    keyVaultCryptoServiceEncryptionUserAssignment
  ]
}

This is working code:

@description('Specifies the name of the environment.')
param environment string

@description('Specifies the name of the deployment.')
@minLength(1)
@maxLength(10)
param customerAsset string

@description('Set soft delete retention in days for key vault')
param softDeleteRetentionInDays int

@description('We only use 2 letters as a unique suffix so that we adhere to the Naming Rules and Character Restrictions, see https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/resource-name-rules')
var uniqueSuffix = substring(uniqueString(resourceGroup().id), 0, 2)
var region = 'gwc'
var storageAccountName = 'st${customerAsset}${environment}${region}${uniqueSuffix}'
var storageIdentityName = 'id-${customerAsset}-${environment}-${region}-storage${uniqueSuffix}'
var storageAccountKeyName = 'key-${customerAsset}-${environment}-${region}-storage${uniqueSuffix}'
var keyVaultName = 'kv-${customerAsset}-${environment}-${region}${uniqueSuffix}'

resource storageIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2022-01-31-preview' = {
  name: storageIdentityName
  location: resourceGroup().location
  tags: resourceGroup().tags
}

resource keyVault 'Microsoft.KeyVault/vaults@2023-07-01' = {
  name: keyVaultName
  location: resourceGroup().location
  tags: resourceGroup().tags
  properties: {
    tenantId: subscription().tenantId
    sku: {
      name: 'standard'
      family: 'A'
    }
    /* networkAcls: networkAcls */
    enablePurgeProtection: true
    enableSoftDelete: true
    softDeleteRetentionInDays: softDeleteRetentionInDays
    enableRbacAuthorization: true
  }
}

@description('This is the built-in Key Vault Crypto Service Encryption User role ')
resource keyVaultCryptoServiceEncryptionUserRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
  name: 'e147488a-f6f5-4113-8e2d-b22465e65bf6'
}

resource keyVaultCryptoServiceEncryptionUserAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(keyVaultCryptoServiceEncryptionUserRole.id, storageIdentity.id, keyVault.id)
  scope: keyVault
  properties: {
    description: 'Allow access to perform cryptographic operations using keys.'
    roleDefinitionId: keyVaultCryptoServiceEncryptionUserRole.id
    principalId: storageIdentity.properties.principalId
    principalType: 'ServicePrincipal'
  }
}

@description('This is the built-in Key Vault Reader role. See https://docs.microsoft.com/azure/role-based-access-control/built-in-roles/security#key-vault-reader')
resource keyVaultReaderRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
  name: '21090545-7ca7-4776-b22c-e363652d74d2'
}

resource keyVaultReaderRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(keyVaultReaderRole.id, storageIdentity.id, keyVault.id)
  scope: keyVault
  properties: {
    description: 'Read metadata of key vaults and its certificates, keys, and secrets.'
    roleDefinitionId: keyVaultReaderRole.id
    principalId: storageIdentity.properties.principalId
    principalType: 'ServicePrincipal'
  }
}

resource storageAccountKey 'Microsoft.KeyVault/vaults/keys@2023-07-01' = {
  parent: keyVault
  name: storageAccountKeyName
  tags: resourceGroup().tags
  properties: {
    kty: 'RSA'
    keySize: 4096
    rotationPolicy: {
      lifetimeActions: [
        {
          trigger: {
            timeAfterCreate: 'P1Y'
          }
          action: {
            type: 'rotate'
          }
        }
        {
          trigger: {
            timeBeforeExpiry: 'P30D'
          }
          action: {
            type: 'notify'
          }
        }
      ]
    }
    attributes: {
      enabled: true
    }
  }
}

resource deploymentScript 'Microsoft.Resources/deploymentScripts@2023-08-01' = {
  name: 'inlineCLI'
  location: resourceGroup().location
  kind: 'AzureCLI'
  properties: {
    azCliVersion: '2.52.0'
    scriptContent: '''
      sleep 10s
    '''
    retentionInterval: 'PT1H'
  }
  dependsOn: [
    keyVaultReaderRoleAssignment
    keyVaultCryptoServiceEncryptionUserAssignment
  ]
}

resource storageAccount 'Microsoft.Storage/storageAccounts@2023-05-01' = {
  name: storageAccountName
  location: resourceGroup().location
  tags: resourceGroup().tags
  sku: {
    name: 'Standard_LRS'
  }
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: { '${storageIdentity.id}': {} }
  }
  kind: 'StorageV2'
  properties: {
    accessTier: 'Hot'
    supportsHttpsTrafficOnly: true
    minimumTlsVersion: 'TLS1_2'
    encryption: {
      services: {
        blob: {
          enabled: true
          keyType: 'Account'
        }
        file: {
          enabled: true
          keyType: 'Account'
        }
        queue: {
          enabled: true
          keyType: 'Account'
        }
        table: {
          enabled: true
          keyType: 'Account'
        }
      }
      keySource: 'Microsoft.Keyvault'
      identity: {
        userAssignedIdentity: storageIdentity.id
      }
      keyvaultproperties: {
        keyname: storageAccountKey.name
        keyvaulturi: keyVault.properties.vaultUri
      }
    }
  }
  dependsOn: [
    deploymentScript
  ]
}