Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: resolve Azure Linux nvidia-container-toolkit posttrans error #5321

Open
wants to merge 2 commits into
base: dev
Choose a base branch
from
+27 −23

fix: resolve Azure Linux nvidia-container-toolkit posttrans error

3581414
Select commit
Loading
Failed to load commit list.
Refresh
Open

fix: resolve Azure Linux nvidia-container-toolkit posttrans error #5321

fix: resolve Azure Linux nvidia-container-toolkit posttrans error
3581414
Select commit
Loading
Failed to load commit list.
Azure Pipelines / AKS Linux VHD Build - PR check-in gate failed Dec 4, 2024 in 37m 56s

Build #20241204.12_merge_109540556 had test failures

1 errors / 0 warnings

Details

Tests

  • Failed: 8 (12.90%)
  • Passed: 54 (87.10%)
  • Other: 0 (0.00%)
  • Total: 62

Annotations

Check failure on line 1416 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / AKS Linux VHD Build - PR check-in gate

Build log #L1416 

Bash exited with code '1'.

Check failure on line 1 in Test_MarinerV2_AirGap

See this annotation in the file changed.

@azure-pipelines azure-pipelines / AKS Linux VHD Build - PR check-in gate

Test_MarinerV2_AirGap 

Failed
Raw output 
    scenario_helpers_test.go:169: running scenario vhd: "/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/CBLMarinerV2gen2/versions/1.1733277976.9867", tags {Name:Test_MarinerV2_AirGap ImageName:CBLMarinerV2gen2 OS:mariner Arch:amd64 Airgap:true GPU:false WASM:false ServerTLSBootstrapping:false Scriptless:false KubeletCustomConfig:false}
    vmss.go:39: creating VMSS "2024-12-04-qtju-marinerv2airgap" in resource group "MC_abe2e-westus2_abe2e-kubenet-airgap-07688_westus2"
    types.go:160: using "/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/CBLMarinerV2gen2/versions/1.1733277976.9867" for VHD
    scenario_helpers_test.go:125: vmss 2024-12-04-qtju-marinerv2airgap creation succeeded
    pollers.go:29: waiting for node 2024-12-04-qtju-marinerv2airgap to be ready
    pollers.go:57: node 2024-12-04-qtju-marinerv2airgap000000 is ready
    scenario_helpers_test.go:128: node 2024-12-04-qtju-marinerv2airgap is ready
    pollers.go:120: -- pod metadata --
    pollers.go:121:    Name: debug-mariner-zdx9k
                       Namespace: default
                       Node: aks-nodepool1-76102184-vmss000000
                       Status: Pending
                       Start Time: 2024-12-04 03:01:43 +0000 UTC
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:123:    Reason: ContainersNotReady
    pollers.go:124:    Message: containers with unready status: [mariner]
    pollers.go:123:    Reason: ContainersNotReady
    pollers.go:124:    Message: containers with unready status: [mariner]
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:126: -- container(s) info --
    pollers.go:128:    Container: mariner
                       Image: privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0
                       Ports: []
    pollers.go:130: -- pod events --
    pollers.go:133:    Reason: Scheduled, Message: Successfully assigned default/debug-mariner-zdx9k to aks-nodepool1-76102184-vmss000000, Count: 0, Last Timestamp: 0001-01-01 00:00:00 +0000 UTC
    pollers.go:133:    Reason: Pulling, Message: Pulling image "privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0", Count: 4, Last Timestamp: 2024-12-04 03:03:32 +0000 UTC
    pollers.go:133:    Reason: Failed, Message: Failed to pull image "privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0": rpc error: code = NotFound desc = failed to pull and unpack image "privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0": failed to resolve reference "privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0": privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0: not found, Count: 4, Last Timestamp: 2024-12-04 03:03:32 +0000 UTC
    pollers.go:133:    Reason: Failed, Message: Error: ErrImagePull, Count: 4, Last Timestamp: 2024-12-04 03:03:32 +0000 UTC
    pollers.go:133:    Reason: BackOff, Message: Back-off pulling image "privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0", Count: 19, Last Timestamp: 2024-12-04 03:06:46 +0000 UTC
    pollers.go:133:    Reason: Failed, Message: Error: ImagePullBackOff, Count: 6, Last Timestamp: 2024-12-04 03:03:47 +0000 UTC
    pollers.go:77: time before timeout: 8m32.594468194s
        
    pollers.go:120: -- pod metadata --
    pollers.go:121:    Name: debug-mariner-zdx9k
                       Namespace: default
                       Node: aks-nodepool1-76102184-vmss000000
                       Status: Pending
                       Start Time: 2024-12-04 03:01:43 +0000 UTC
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:123:    Reason: ContainersNotReady
    pollers.go:124:

Check failure on line 1 in Test_Ubuntu2204_AirGap

See this annotation in the file changed.

@azure-pipelines azure-pipelines / AKS Linux VHD Build - PR check-in gate

Test_Ubuntu2204_AirGap 

Failed
Raw output 
    scenario_helpers_test.go:169: running scenario vhd: "/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/2204gen2containerd/versions/1.1733277968.12093", tags {Name:Test_Ubuntu2204_AirGap ImageName:2204gen2containerd OS:ubuntu Arch:amd64 Airgap:true GPU:false WASM:false ServerTLSBootstrapping:false Scriptless:false KubeletCustomConfig:false}
    vmss.go:39: creating VMSS "2024-12-04-7m12-ubuntu2204airgap" in resource group "MC_abe2e-westus2_abe2e-kubenet-airgap-07688_westus2"
    types.go:160: using "/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/2204gen2containerd/versions/1.1733277968.12093" for VHD
    scenario_helpers_test.go:125: vmss 2024-12-04-7m12-ubuntu2204airgap creation succeeded
    pollers.go:29: waiting for node 2024-12-04-7m12-ubuntu2204airgap to be ready
    pollers.go:57: node 2024-12-04-7m12-ubuntu2204airgap000000 is ready
    scenario_helpers_test.go:128: node 2024-12-04-7m12-ubuntu2204airgap is ready
    pollers.go:120: -- pod metadata --
    pollers.go:121:    Name: debug-mariner-zdx9k
                       Namespace: default
                       Node: aks-nodepool1-76102184-vmss000000
                       Status: Pending
                       Start Time: 2024-12-04 03:01:43 +0000 UTC
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:123:    Reason: ContainersNotReady
    pollers.go:124:    Message: containers with unready status: [mariner]
    pollers.go:123:    Reason: ContainersNotReady
    pollers.go:124:    Message: containers with unready status: [mariner]
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:126: -- container(s) info --
    pollers.go:128:    Container: mariner
                       Image: privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0
                       Ports: []
    pollers.go:130: -- pod events --
    pollers.go:133:    Reason: Scheduled, Message: Successfully assigned default/debug-mariner-zdx9k to aks-nodepool1-76102184-vmss000000, Count: 0, Last Timestamp: 0001-01-01 00:00:00 +0000 UTC
    pollers.go:133:    Reason: Pulling, Message: Pulling image "privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0", Count: 4, Last Timestamp: 2024-12-04 03:03:32 +0000 UTC
    pollers.go:133:    Reason: Failed, Message: Failed to pull image "privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0": rpc error: code = NotFound desc = failed to pull and unpack image "privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0": failed to resolve reference "privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0": privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0: not found, Count: 4, Last Timestamp: 2024-12-04 03:03:32 +0000 UTC
    pollers.go:133:    Reason: Failed, Message: Error: ErrImagePull, Count: 4, Last Timestamp: 2024-12-04 03:03:32 +0000 UTC
    pollers.go:133:    Reason: BackOff, Message: Back-off pulling image "privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0", Count: 19, Last Timestamp: 2024-12-04 03:06:46 +0000 UTC
    pollers.go:133:    Reason: Failed, Message: Error: ImagePullBackOff, Count: 6, Last Timestamp: 2024-12-04 03:03:47 +0000 UTC
    pollers.go:77: time before timeout: 5m7.26375796s
        
    pollers.go:120: -- pod metadata --
    pollers.go:121:    Name: debug-mariner-zdx9k
                       Namespace: default
                       Node: aks-nodepool1-76102184-vmss000000
                       Status: Pending
                       Start Time: 2024-12-04 03:01:43 +0000 UTC
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:123:    Reason: ContainersNotReady
    pollers.

Check failure on line 1 in Test_Ubuntu2204_WASMAirGap

See this annotation in the file changed.

@azure-pipelines azure-pipelines / AKS Linux VHD Build - PR check-in gate

Test_Ubuntu2204_WASMAirGap 

Failed
Raw output 
    scenario_helpers_test.go:169: running scenario vhd: "/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/2204gen2containerd/versions/1.1733277968.12093", tags {Name:Test_Ubuntu2204_WASMAirGap ImageName:2204gen2containerd OS:ubuntu Arch:amd64 Airgap:true GPU:false WASM:false ServerTLSBootstrapping:false Scriptless:false KubeletCustomConfig:false}
    vmss.go:39: creating VMSS "2024-12-04-rwxz-ubuntu2204wasmairgap" in resource group "MC_abe2e-westus2_abe2e-kubenet-airgap-07688_westus2"
    types.go:160: using "/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/2204gen2containerd/versions/1.1733277968.12093" for VHD
    scenario_helpers_test.go:125: vmss 2024-12-04-rwxz-ubuntu2204wasmairgap creation succeeded
    pollers.go:29: waiting for node 2024-12-04-rwxz-ubuntu2204wasmairgap to be ready
    pollers.go:57: node 2024-12-04-rwxz-ubuntu2204wasmairgap000000 is ready
    scenario_helpers_test.go:128: node 2024-12-04-rwxz-ubuntu2204wasmairgap is ready
    pollers.go:120: -- pod metadata --
    pollers.go:121:    Name: debug-mariner-zdx9k
                       Namespace: default
                       Node: aks-nodepool1-76102184-vmss000000
                       Status: Pending
                       Start Time: 2024-12-04 03:01:43 +0000 UTC
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:123:    Reason: ContainersNotReady
    pollers.go:124:    Message: containers with unready status: [mariner]
    pollers.go:123:    Reason: ContainersNotReady
    pollers.go:124:    Message: containers with unready status: [mariner]
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:126: -- container(s) info --
    pollers.go:128:    Container: mariner
                       Image: privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0
                       Ports: []
    pollers.go:130: -- pod events --
    pollers.go:133:    Reason: Scheduled, Message: Successfully assigned default/debug-mariner-zdx9k to aks-nodepool1-76102184-vmss000000, Count: 0, Last Timestamp: 0001-01-01 00:00:00 +0000 UTC
    pollers.go:133:    Reason: Pulling, Message: Pulling image "privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0", Count: 4, Last Timestamp: 2024-12-04 03:03:32 +0000 UTC
    pollers.go:133:    Reason: Failed, Message: Failed to pull image "privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0": rpc error: code = NotFound desc = failed to pull and unpack image "privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0": failed to resolve reference "privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0": privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0: not found, Count: 4, Last Timestamp: 2024-12-04 03:03:32 +0000 UTC
    pollers.go:133:    Reason: Failed, Message: Error: ErrImagePull, Count: 4, Last Timestamp: 2024-12-04 03:03:32 +0000 UTC
    pollers.go:133:    Reason: BackOff, Message: Back-off pulling image "privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0", Count: 19, Last Timestamp: 2024-12-04 03:06:46 +0000 UTC
    pollers.go:133:    Reason: Failed, Message: Error: ImagePullBackOff, Count: 6, Last Timestamp: 2024-12-04 03:03:47 +0000 UTC
    pollers.go:77: time before timeout: 5m3.845581719s
        
    pollers.go:120: -- pod metadata --
    pollers.go:121:    Name: debug-mariner-zdx9k
                       Namespace: default
                       Node: aks-nodepool1-76102184-vmss000000
                       Status: Pending
                       Start Time: 2024-12-04 03:01:43 +0000 UTC
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:123:    Reason: Contai

Check failure on line 1 in Test_Ubuntu2204Gen2_ContainerdAirgappedK8sNotCached

See this annotation in the file changed.

@azure-pipelines azure-pipelines / AKS Linux VHD Build - PR check-in gate

Test_Ubuntu2204Gen2_ContainerdAirgappedK8sNotCached 

Failed
Raw output 
    scenario_helpers_test.go:169: running scenario vhd: "/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/2204Gen2/versions/1.1725612526.29638", tags {Name:Test_Ubuntu2204Gen2_ContainerdAirgappedK8sNotCached ImageName:2204Gen2 OS:ubuntu Arch:amd64 Airgap:true GPU:false WASM:false ServerTLSBootstrapping:false Scriptless:false KubeletCustomConfig:false}
    aks_model.go:205: Creating private Azure Container Registry in rg abe2e-westus2
    aks_model.go:209: Private ACR already exists at id /subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/abe2e-westus2/providers/Microsoft.ContainerRegistry/registries/privateacre2ewestus2, skipping creation
    cluster.go:161: cluster abe2e-kubenet-airgap-07688 already exists in rg abe2e-westus2
    cluster.go:118: node resource group: MC_abe2e-westus2_abe2e-kubenet-airgap-07688_westus2
    cluster.go:124: ensuring debug daemonsets
    aks_model.go:70: Adding network settings for airgap cluster abe2e-kubenet-airgap-07688 in rg MC_abe2e-westus2_abe2e-kubenet-airgap-07688_westus2
    aks_model.go:154: Checking if private endpoint for private container registry is in rg MC_abe2e-westus2_abe2e-kubenet-airgap-07688_westus2
    aks_model.go:195: Private Endpoint already exists with ID: /subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus2_abe2e-kubenet-airgap-07688_westus2/providers/Microsoft.Network/privateEndpoints/PE-for-ABE2ETests
    aks_model.go:163: Private Endpoint already exists, skipping creation
    aks_model.go:106: updated cluster abe2e-kubenet-airgap-07688 subnet with airgap settings
    vmss.go:39: creating VMSS "2024-12-04-b00e-ubuntu2204gen2containerdairgappedk8snotca" in resource group "MC_abe2e-westus2_abe2e-kubenet-airgap-07688_westus2"
    types.go:160: using "/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/2204Gen2/versions/1.1725612526.29638" for VHD
    scenario_helpers_test.go:125: vmss 2024-12-04-b00e-ubuntu2204gen2containerdairgappedk8snotca creation succeeded
    pollers.go:29: waiting for node 2024-12-04-b00e-ubuntu2204gen2containerdairgappedk8snotca to be ready
    pollers.go:57: node 2024-12-04-b00e-ubuntu2204gen2containerdairgappedk8snotca000000 is ready
    scenario_helpers_test.go:128: node 2024-12-04-b00e-ubuntu2204gen2containerdairgappedk8snotca is ready
    pollers.go:120: -- pod metadata --
    pollers.go:121:    Name: debug-mariner-zdx9k
                       Namespace: default
                       Node: aks-nodepool1-76102184-vmss000000
                       Status: Pending
                       Start Time: 2024-12-04 03:01:43 +0000 UTC
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:123:    Reason: ContainersNotReady
    pollers.go:124:    Message: containers with unready status: [mariner]
    pollers.go:123:    Reason: ContainersNotReady
    pollers.go:124:    Message: containers with unready status: [mariner]
    pollers.go:123:    Reason: 
    pollers.go:124:    Message: 
    pollers.go:126: -- container(s) info --
    pollers.go:128:    Container: mariner
                       Image: privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0
                       Ports: []
    pollers.go:130: -- pod events --
    pollers.go:133:    Reason: Scheduled, Message: Successfully assigned default/debug-mariner-zdx9k to aks-nodepool1-76102184-vmss000000, Count: 0, Last Timestamp: 0001-01-01 00:00:00 +0000 UTC
    pollers.go:133:    Reason: Pulling, Message: Pulling image "privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/core:2.0", Count: 4, Last Timestamp: 2024-12-04 03:03:32 +0000 UTC
    pollers.go:133:    Reason: Failed, Message: Failed to pull image "privateacre2ewestus2.azurecr.io/aks/cbl-mariner/base/
View more details on Azure Pipelines