(Confirmed Incident) GitHub Actions workflows stuck in queued state

[cfarrend]

UPDATE: GitHub have confirmed this as an incident

https://www.githubstatus.com/incidents/9yk1fbk0qjjc

Background

Hi all,

Noticed in the last 10-20 minutes or so all of our workflows are now pending. Some runners actually show output despite them being in a Queued state.

I notice there has been no updates to https://www.githubstatus.com/.

Just checking here quickly to see if anyone else has a similar problem?

This is the ubuntu-latest runner, I haven't tested other runners.

[sbhadr]

Same is happening to me. Runner is super simple so, it has to be Github issues. Interesting, when you cancel all the runners in this pending state, it causes issues and it hangs.

[yannxaver]

Same problem

[Tisawesomeness]

I can confirm, seems to be a widespread issue.

[rtau-cca]

Github Status Incident Report on this: https://www.githubstatus.com/incidents/9yk1fbk0qjjc

[xxl4]

https://www.githubstatus.com/

[sweetapple911]

same now solved

[cfarrend]

In that case I'll consider it an outage yet to be reported. Thanks all

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[dongho-jung]

I'm also suffering from the same issue right now. 😢

[Somnusochi]

me too.

[mpdunlop]

Same issue here. Seems like the GitHub Status page is lying about there being no issues...

[PrabhuSaruvamalai]

same here

[davidtrading]

SAME

[kimjisub]

same here. I'm using Self-hosted runner but it also stuck in queued state.

[HwangBBang]

SAME

[Zhaohe-Ji]

same here, seems the scheduler get stucked

[dmmarkov]

same here

[SON-JUYOUNG]

Same

[Tisawesomeness]

Trying to force-cancel a job through the API results in a 500 error:

{
  "message": "Failed to cancel workflow run",
  "documentation_url": "https://docs.github.com/rest/actions/workflow-runs#force-cancel-a-workflow-run",
  "status": "500"
}

[HwangBBang]

SAMEEEE!

Waiting for a runner to pick up this job...
Job is waiting for a hosted runner to come online.

[Pixis-Akshay-Gopani]

Jobs getting stuck in queued state

[Mustafa1100]

Issue is resolved!

[samus-aran]

Hi everyone! 👋

Current Status: This incident has been resolved. ✅

Thanks for sharing and we appreciate your patience. This particular incident is documented here 👈 You can check our GitHub Status Page to see for real-time and historical data on system performance + subscribe to issues for updates.

Thank you again for your patience and understanding.

[shunhuaiyao]

Hi all,
Do you have the error issue below?

OS: macOS 14.7 23H124
Runner image: macos-14-arm64
Version: 20241022.361

bundler: failed to load command: fastlane (/opt/homebrew/lib/ruby/gems/3.0.0/bin/fastlane)

SystemStackError: stack level too deep

/opt/homebrew/lib/ruby/site_ruby/3.0.0/rubygems/stub_specification.rb:230:in `loaded_spec'

/opt/homebrew/lib/ruby/site_ruby/3.0.0/rubygems/stub_specification.rb:186:in `spec'

/opt/homebrew/lib/ruby/gems/3.0.0/gems/bundler-2.1.4/lib/bundler/stub_specification.rb:78:in `_remote_specification'

/opt/homebrew/lib/ruby/gems/3.0.0/gems/bundler-2.1.4/lib/bundler/remote_specification.rb:104:in `method_missing'

[shunhuaiyao]

Fixed the issue by updating the bundler version, 2.5.22, and fastlane version, 2.220.0.

[samus-aran]

Well done @shunhuaiyao. Thanks for sharing 👍

[danilodesolesonar]

We are still facing issue with python, in particular for some dependencies it seems that the path is missing, although we did not change anything

Successfully created virtual environment!
....
  File "/home/runner/work/..../..../scripts/repackage_dependencies.py", line 16, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'
make: *** [Makefile:26: ....] Error 1

[danilodesolesonar]

And another example

Alternatively, run a command inside the virtualenv with pipenv run.
Installing dependencies from Pipfile...
Installing dependencies from Pipfile...
./test.sh
Error: the command pytest could not be found within PATH or Pipfile's [scripts].
make: *** [Makefile:19: test] Error 1

[danilodesolesonar]

This is related to pipenv regression. Please ignore.

[cfarrend]

Closing this thread as this was specifically around the outage which is now resolved. Thanks all.

[jihoon-overthehand]

Issue is still occurred...

[samus-aran]

Have you experienced this since the incident announced or has this begun after the incident was resolved? Are these new jobs?

[mangeshmandlik1]

We are still facing this issue "Waiting for a runner to pick up this job..."

name: Workflow to trigger pipeline for helm changes to be promoted to k8s
on:
push:
branches: ["main"]
pull_request:
branches: ["main"]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Test First Job
run: echo "Job triggered !!"

[h4ckroot]

@samus-aran We are currently experiencing the same issue (all jobs are in queues state).

Note: Enterprise account.

[h4ckroot]

never mind, it is fixed now.

[ChintanRajpara]

Issue is still occurred