slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2721883	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1057172	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
2721627	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2721465	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
1056128	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#116070
2720451	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2720371	dotnet-runtime	Performance ios_scenarios iOSNativeAOT JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1055356	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#116157
2720095	dotnet-runtime	Performance ios_scenarios iOSNativeAOT JIT osx x64 perfiphone12mini net10.0	Log
2719657	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2719546	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1054348	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#116109
2719411	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2719292	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2718928	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2718689	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2718652	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2718521	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2718501	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2718416	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2718142	dotnet-runtime	Performance ios_scenarios iOSNativeAOT JIT osx x64 perfiphone12mini net10.0	Log
2717800	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1050951	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1419
1050896	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1418
2717656	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2717537	dotnet-runtime	Performance ios_scenarios iOSNativeAOT JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1051573	dotnet/arcade	Android_Simulators Build_Debug	Log	dotnet/arcade#15853
2717391	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2717147	dotnet-runtime	Performance ios_scenarios iOSNativeAOT JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1049688	dotnet/arcade	Android_Simulators Build_Debug	Log	dotnet/arcade#15858
2717021	dotnet-runtime	Performance ios_scenarios iOSNativeAOT JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1050821	dotnet/runtime	osx-x64 Debug PALTests	Log	dotnet/runtime#115957
1050818	dotnet/runtime	osx-x64 Debug PALTests	Log
2716864	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
1050720	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#115977
1050725	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#115947
2716799	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1050715	dotnet/runtime	osx-x64 Debug PALTests	Log
1050710	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#115974
1050691	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#115658
1050686	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#115948
1050675	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#115959
1050671	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#115976
1050662	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#115975
1050518	dotnet/arcade	Android_Simulators Build_Debug	Log	dotnet/arcade#15853
2716439	dotnet-install-scripts	MacOSTests	Log
2716376	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1050144	dotnet/performance	Performance roslyn windows 22H2 x64 Open 9.0	Log
2715612	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2715327	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1049152	dotnet/performance	Performance scenarios windows 22H2 x64 Open 8.0	Log
2715137	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
1048782	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1416
1048645	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1416
2714932	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2714920	dotnet-performance	Performance mlnet ubuntu 2204 arm64 Ampere 9.0	Log
2714788	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1047936	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1047482	dotnet/performance	Performance micro windows 22H2 x86 Open main	Log
1047135	dotnet/performance	Performance micro windows 22H2 x86 Open main	Log	dotnet/performance#4859
1047062	dotnet/performance	Performance micro ubuntu 2204 x64 Open main	Log
2713736	dotnet-performance	Performance maui_scenarios_android R2RComposite CoreCLR windows 22H2 x64 Pixel main	Log
2713673	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2713242	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1045991	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log	dotnet/runtime#115756
2713121	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2713036	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2712923	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2712931	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2712749	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
1045347	dotnet/arcade	Linux Build_Debug	Log	dotnet/arcade#15847
1044814	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1414
1045210	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#115688
1045205	dotnet/performance	Performance micro ubuntu 2204 x64 Open 9.0	Log	dotnet/performance#4841
1045192	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1417
1045194	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#115723
1045188	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#115658
2712615	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1045162	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#115699
1045150	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#115652
1044387	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#115687
1045125	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#110472
1045118	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#115722
1045105	dotnet/performance	Performance fsharpmicro ubuntu 2204 x64 Open 8.0	Log
2712568	dotnet-performance	Performance bepuphysics ubuntu 2204 x64 Tiger 8.0	Log
1045091	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1416
1045073	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#110692
1045025	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#115609
1045019	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#114148
1045001	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#115408
1044998	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log
1044988	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#114619
1044985	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#115553
1044967	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log
2712394	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
1044940	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#114741
1044312	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#115659
1044780	dotnet/arcade	Android_Devices Build_Release	Log	dotnet/arcade#15845
1044898	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#115710
1044874	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#115589
Displaying 100 of 239 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
3	24	239

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.