Switch jenkins OIDC from Amazon federate to GitHub

[gaiksaya]

Is your feature request related to a problem? Please describe

Coming from opensearch-project/opensearch-build#5171
The first milestone is to make release process CI executable by anyone including external community. As of today, the jenkins CI at https://build.ci.opensearch.org/ using Amazon Federate, an internal authentication service to authenticate users.

Describe the solution you'd like

Possible options:

• Switch from internal tool to GitHub as OIDC.
• Basic Auth
• Something else?

Describe alternatives you've considered

Keep using amazon federate and ask the users with right access to execute workflows everytime.

Additional context

We recently integrated nightly playgrounds https://playground.nightly.opensearch.org/ with GitHub as OIDC.
Though GitHub is not an OpenID Connect Provider, mediator such as keycloak can be used to break the identity and manage access accordingly.

[gaiksaya]

Approach 1: Use github-oauth plugin [Recommended]

Jenkins has multiple plugins that it uses for its efficiency, integration with other tools and even basic functioning. One such plugin that can be used to integrated GitHub with jenkins as a OIDC is github-oauth plugin.
See https://plugins.jenkins.io/github-oauth/

Pros:

• Direct integration
• Minimal set up
• Less to no maintenance

Cons:

• Need to map roles to users which can be added as a code.

Approach 2: Use existing keycloak set up

Github is not an identity provider. Hence an intermediate broker is required to break the identity. We recently integrated keycloak with nightly plyagrounds to use GitHub as an OIDC. This approach proposes to use the same set up for jenkins authentication as well.

Create a new realm for jenkins exclusively.

Pros:

• So extra set up required.

Cons:

• Need to create new roles, map them to jenkins roles and then map users to those roles.
• Too much maintenance
• Single point of failure in case keycloak goes down
• Dual permission management both on keycloak side and jenkins side.

Approach 3: Use basic auth

Have user and password basic authentication. Strictly not recommended.

[gaiksaya]

Did a small POC with approach 1 and seems to be working without any issue. Easy integration as well without any complications or major changes.

Adding @prudhvigodithi @rishabh6788 @peterzhuamazon @Divyaasm to get some input on the approaches.

[gaiksaya]

Posting more details about GitHub Oauth app. Required if we move with approach 1 and 2.

Following the set up mentioned here: https://plugins.jenkins.io/github-oauth/

1. Visit https://github.com/settings/applications/new to create a GitHub application registration.
2. The values for application name, homepage URL, or application description don't matter. They can be customized however desired.
3. However, the authorization callback URL takes a specific value. It must be https://jenkins.example.com/securityRealm/finishLogin where jenkins.example.com is the location of the Jenkins server.
4. The important part of the callback URL is /securityRealm/finishLogin
5.  Finish by clicking Register application.

While creating GitHub Oauth apps, looks like we cannot have more than one authorizarion call-back URLs. See documentation here.

 OAuth apps cannot have multiple callback URLs, unlike GitHub Apps.

GitHub Apps on the other hand does support multiple callback URLs. However, jenkins does not have a way to specify the redirect_uri option. The PR is open for add this feature but is stale at this point. jenkinsci/github-oauth-plugin#198. Related jenkins issue: https://issues.jenkins.io/browse/JENKINS-43214

As of Nov 18,2024 we have 2 endpoints: beta and prod. However in future with #382 there might be more. Assuming all of them need to use GH as OIDC and after researching for solutions, we have below options:

1. Create a new app for each endpoint. This will have separate maintenance for each app but we avoid single point of failure. Also I believe it will help with the point 4 in above set up at this point.
2. Have an external server that acts as a proxy to redirect the requests based on URL. Not sure how and if we would able to handle the requirement from Point 4. Need to test it.

[gaiksaya]

Hi everyone,
There was another workaround to use the about redirect-uri feature from the open PR. See the details below:

1. As mentioned in one of the PR comments, I tried installing the plugin directly using the recent build https://ci.jenkins.io/job/Plugins/job/github-oauth-plugin/job/PR-198/34/ hpi file. However, due to the requirement of jenkins version need to be latest, the installation failed with below error:

java.io.IOException: Failed to load: GitHub Authentication plugin (github-oauth 647.v2ff2987066b_f)
 - Jenkins (2.479.1) or higher required
 - Update required: Git plugin (git 5.2.2) to be updated to 5.6.0 or higher
 - Update required: Jenkins Mailer Plugin (mailer 470.vc91f60c5d8e2) to be updated to 489.vd4b_25144138f or higher
 - Update required: GitHub API Plugin (github-api 1.318-461.v7a_c09c9fa_d63) to be updated to 1.321-478.vc9ce627ce001 or higher
 - Update required: Matrix Project Plugin (matrix-project 822.824.v14451b_c0fd42) to be updated to 840.v812f627cb_578 or higher
 - Update required: GitHub Branch Source Plugin (github-branch-source 1789.v5b_0c0cea_18c3) to be updated to 1807.v50351eb_7dd13 or higher
 - Update required: Pipeline: Multibranch (workflow-multibranch 773.vc4fe1378f1d5) to be updated to 795.ve0cb_1f45ca_9a_ or higher
	at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:988)
	at hudson.PluginManager.dynamicLoad(PluginManager.java:943)
Caused: java.io.IOException: Failed to install github-oauth plugin
	at hudson.PluginManager.dynamicLoad(PluginManager.java:957)
	at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2247)
Caused: java.io.IOException: Failed to dynamically deploy this plugin
	at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2251)
	at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1897)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:121)
	at java.base/java.lang.Thread.run(Thread.java:1583)

2. I tried upgrading jenkins and plugins (manually using UI) and tried installing the above build plugin again. It was successful. 🎉
3. Created a GitHub App (not an OAuth App) where you can mention multiple callback URLs. See github's documentation for authorization callback URLs and how it can be customized using redirect_uri option.
4. Enabled securityRealm to use GitHub authentication and it was able to redirect and use GitHub as OIDC.
   

Option A:

If we want to move ahead with this approach, the action items on our end would be:

1. Upgrade jenkins from 2.426.3 to 2.479.1
2. Create one single GitHub App (not OAuth app) that can configure multiple callback URLs.
3. Find a way to install github-oauth plugin using URL from above build.
4. Enable the same on jenkins side via code.

Option B:

If not, another approach would be:

1. Create GitHub OAuth app for each endpoint with unique callback URLs (count is 2 at this point).
2. Enable GitHub as OIDC via code.
3. Once jenkins is upgraded move/switch to above approach of GitHub App.

@rishabh6788 @prudhvigodithi @peterzhuamazon @getsaurabh02 let me know if you guys have preferences. Moving fast we can goahead with option B and then once we have upgraded jenkins and hopefully that feature gets merged officially in github-oauth repo we can switch to Option A.

Thanks!

[Divyaasm]

Thanks for listing out the possible aproaches Sayali, the windows issue related to durable-task plugin is not resolved yet. Same issue was opened recently here. I would vote for option B now as GitHub Authentication plugin is not compatible with our jenkins version.

[peterzhuamazon]

Second to @Divyaasm that Option 1 has the issue of existing windows runner failure.
We are still evaluating the CICD options as part of moving to LF.

If we entirely move to potential GitHub Actions + GitHub ARC then this might not needed. Thanks.

[gaiksaya]

Thank you for the inputs @Divyaasm and @peterzhuamazon
Will proceed with Option B for now and create an issue with Option A for future migration. The maintenance overhead will be drastically reduced with Option A once we upgrade or plugin supports backward compatibility with the version in use in prod.

[gaiksaya]

Just to update,
Enabling Github as OIDC also allows to call jenkins API using GitHub Personal Access Token. See https://plugins.jenkins.io/github-oauth/#plugin-content-other-usage
This is a great solution for our future usecase where the release manager can trigger workflows (manually or via tool) using GitHub PATs.

[gaiksaya]

The authentication for https://build.ci.opensearch.org/ has been switched from internal auth to GitHub using GitHub-oauth plugin. The GitHub OAuth apps are currently residing in my developer account. Will be transferring the ownership of the same to opensearch-project org based on new process. See opensearch-project/.github#236

Thanks!