Microsoft.NET.Build.Containers.IntegrationTests are pulling from Docker Hub

[akoeplinger]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=733130
Build error leg or test failing: Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.WriteToPrivateBasicRegistry
Pull request: https://github.com/dotnet/sdk.git/pull/42019

We need to stop pulling from upstream Docker Hub since we're hitting rate limits.

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "You have reached your pull rate limit",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=733130
Error message validated: [You have reached your pull rate limit]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 7/8/2024 7:20:41 PM UTC

Report

Build	Definition	Test	Pull Request
880538	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45169
880118	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45157
880099	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#44570
880085	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45042
879699	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45143
879638	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateImageIndexTests.CreateImageIndex_Baseline	#43085
878388	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45108
878367	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45106
878361	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45105
878289	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateImageIndexTests.CreateImageIndex_Baseline	#43085
877637	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45071
877627	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44786
877502	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.ApiEndToEndWithLocalLoad	#45052
875785	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44786
875754	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45009
875752	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45015
875748	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.ApiEndToEndWithLocalLoad	#43085
875728	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45014
874542	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.EndToEnd_MultiProjectSolution	#44786
874047	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#44786
873924	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44927
873400	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44805
873332	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44949
873330	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#44948
873211	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#44802
873191	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#43085
872042	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#44810
871495	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#44906
855463	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44494
855453	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44477

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	18	30

[akoeplinger]

@marcpopMSFT who owns Microsoft.NET.Build.Containers?

[baronfel]

@dotnet/sdk-container-builds-maintainers

[baronfel]

@akoeplinger the image in question is registry:2, which we need/use to validate our logic for communicating with container registries via the container registry APIs. Do you know if this image is mirrored anywhere that we can pull from? It's not currently on mcr.microsoft.com.

[nagilson]

Bumping this thread since it's affecting more PRs

[nagilson]

We don't have a path forward to fix this yet - Aspire moved to using their own ACR for storing the images they use, but we need an engineering-systems-managed solution for our entire stack. There's an issue tracking this request but I can't find it at the moment.

[baronfel]

Could be fixed at a root cause level by dotnet/dnceng#3389

[MichaelSimons]

@akoeplinger the image in question is registry:2, which we need/use to validate our logic for communicating with container registries via the container registry APIs. Do you know if this image is mirrored anywhere that we can pull from? It's not currently on mcr.microsoft.com.

@baronfel - It sounds like the test has this dependency and is what is pulling the image. One possible option to workaround scenarios where tests are pulling images directly from DockerHub would be to create a wrapper buildtools image. The Dockerfile would just be a FROM statement. The buildtools images are stored on MCR and therefore you would be able to get around this rate limiting issue.

[nagilson]

@dotnet/sdk-container-builds-maintainers Do you agree with @MichaelSimons suggestion? This has a pretty large impact on PR failure rates. If this is still an issue going forward a month or two down the line, I will sadly suggest that we just disable these tests

[marcpopMSFT]

@donJoseLuis @MichalPavlik I'm told by baronfel that there may be a solution to this of setting up our own ACR mirror of these so we don't get blocked by docker from pulling too often. He said to talk to @eerhardt about that. Is that something you can potentially drive getting set up?

As you can see above, this fails a couple of dozen PRs every week that have to be rerun. It's not a crazy blocker but that's high enough to be something we want eyes on to get fixed eventually.