Skip to content

Conversation

pan3793
Copy link
Member

@pan3793 pan3793 commented Aug 26, 2025

Description of PR

Rocky Linux is supposed to be a drop-in replacement for the discontinued CentOS. See more details at https://rockylinux.org/about

How was this patch tested?

$ ./start-build-env.sh rockylinux_8
...

 _   _           _                    ______
| | | |         | |                   |  _  \
| |_| | __ _  __| | ___   ___  _ __   | | | |_____   __
|  _  |/ _` |/ _` |/ _ \ / _ \| '_ \  | | | / _ \ \ / /
| | | | (_| | (_| | (_) | (_) | |_) | | |/ /  __/\ V /
\_| |_/\__,_|\__,_|\___/ \___/| .__/  |___/ \___| \_(_)
                              | |
                              |_|

This is the standard Hadoop Developer build environment.
This has all the right tools installed required to build
Hadoop from source.

[chengpan@4af99dc981b9 hadoop]$
$ mvn clean install -DskipTests -Pnative -Pyarn-ui -DskipShade
...
[INFO] Reactor Summary for Apache Hadoop Main 3.5.0-SNAPSHOT:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [  0.675 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  1.518 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  0.651 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  0.686 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  0.072 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.077 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  1.476 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  0.316 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  2.002 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  0.535 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 18.943 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  1.071 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [  1.085 s]
[INFO] Apache Hadoop Registry ............................. SUCCESS [  1.282 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.046 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 10.259 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 17.829 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [02:26 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [  1.772 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  1.020 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.041 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [  5.855 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [  4.910 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.038 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [  3.994 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [  1.619 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  1.207 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  1.058 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 10.150 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 28.435 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  1.134 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [  1.959 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.616 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [  4.440 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [  2.062 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  1.414 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  2.901 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  1.887 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  3.319 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  1.954 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  1.113 s]
[INFO] Apache Hadoop Federation Balance ................... SUCCESS [  1.234 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [  5.733 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.034 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  0.886 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  0.866 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [  0.035 s]
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [  1.291 s]
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [  1.709 s]
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [  0.036 s]
[INFO] Apache Hadoop YARN TimelineService HBase Server 2.5  SUCCESS [  1.677 s]
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [  1.493 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [  1.912 s]
[INFO] Apache Hadoop YARN TimelineService DocumentStore ... SUCCESS [  1.018 s]
[INFO] Apache Hadoop YARN GlobalPolicyGenerator ........... SUCCESS [  1.175 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.036 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  1.166 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  0.689 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [  0.034 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [  2.393 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [  1.552 s]
[INFO] Apache Hadoop YARN Application Catalog ............. SUCCESS [  0.034 s]
[INFO] Apache Hadoop YARN Application Catalog Webapp ...... SUCCESS [ 10.674 s]
[INFO] Apache Hadoop YARN Application Catalog Docker Image  SUCCESS [  0.046 s]
[INFO] Apache Hadoop YARN Application MaWo ................ SUCCESS [  0.034 s]
[INFO] Apache Hadoop YARN Application MaWo Core ........... SUCCESS [  0.781 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.036 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  0.439 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 51.592 s]
[INFO] Apache Hadoop YARN CSI ............................. SUCCESS [  2.776 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [  1.155 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  0.854 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [ 19.795 s]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [  0.777 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  1.427 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  1.079 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  1.533 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [  0.642 s]
[INFO] Apache Hadoop Dynamometer Workload Simulator ....... SUCCESS [  1.110 s]
[INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SUCCESS [  2.098 s]
[INFO] Apache Hadoop Dynamometer Block Listing Generator .. SUCCESS [  0.994 s]
[INFO] Apache Hadoop Dynamometer Dist ..................... SUCCESS [  1.063 s]
[INFO] Apache Hadoop Dynamometer .......................... SUCCESS [  0.034 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  0.908 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  1.125 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  1.466 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  1.450 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.002 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  0.995 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  3.333 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  4.599 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [  0.636 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  4.082 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [  0.878 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  1.312 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [  0.868 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  0.844 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [  0.893 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  0.624 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  0.042 s]
[INFO] Apache Hadoop Common Benchmark ..................... SUCCESS [  9.870 s]
[INFO] Apache Hadoop Compatibility Benchmark .............. SUCCESS [  0.598 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.032 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [  0.851 s]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [  0.639 s]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [  0.373 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [  0.690 s]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [  0.171 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [  0.639 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [  0.447 s]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [  0.032 s]
[INFO] Apache Hadoop Tencent COS Support .................. SUCCESS [  0.640 s]
[INFO] Apache Hadoop OBS support .......................... SUCCESS [  1.068 s]
[INFO] Apache Hadoop Volcano Engine Services support ...... SUCCESS [  1.601 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  0.401 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.033 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  07:35 min
[INFO] Finished at: 2025-08-26T10:05:06Z
[INFO] ------------------------------------------------------------------------
[chengpan@4af99dc981b9 hadoop]$

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 0s Docker mode activated.
-1 ❌ docker 0m 21s Dockerfile '/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7900/rockylinux-8/src/dev-support/docker/Dockerfile_rockylinux_8' not found.
Subsystem Report/Notes
GITHUB PR #7900
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7900/1/console
versions git=2.34.1
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 0s Docker mode activated.
-1 ❌ docker 0m 21s Dockerfile '/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7900/rockylinux-8/src/dev-support/docker/Dockerfile_rockylinux_8' not found.
Subsystem Report/Notes
GITHUB PR #7900
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7900/2/console
versions git=2.34.1
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 0s Docker mode activated.
-1 ❌ docker 0m 21s Dockerfile '/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7900@2/rockylinux-8/src/dev-support/docker/Dockerfile_rockylinux_8' not found.
Subsystem Report/Notes
GITHUB PR #7900
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7900/3/console
versions git=2.34.1
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@pan3793
Copy link
Member Author

pan3793 commented Aug 26, 2025

Looks like Yetus finds Dockerfile from the trunk branch instead of the PR branch, I'm not very familiar with this part. If the behavior is not easy to change, I'm afraid this PR must be committed first, then we will know what is going to happen.

[2025-08-26T10:13:14.863Z] Already on 'trunk'
[2025-08-26T10:13:14.863Z] Your branch is up to date with 'origin/trunk'.
[2025-08-26T10:13:14.863Z] HEAD is now at 9d2a83d18ba HADOOP-19636. [JDK17] Remove CentOS 7 Support and Clean Up Dockerfile. (#7822)
[2025-08-26T10:13:15.444Z] 
[2025-08-26T10:13:15.444Z] Testing https://github.com/apache/hadoop/pull/7900 patch on trunk.
[2025-08-26T10:13:15.444Z] ERROR: Dockerfile '/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7900@2/rockylinux-8/src/dev-support/docker/Dockerfile_rockylinux_8' not found.

@slfan1989
Copy link
Contributor

Looks like Yetus finds Dockerfile from the trunk branch instead of the PR branch, I'm not very familiar with this part. If the behavior is not easy to change, I'm afraid this PR must be committed first, then we will know what is going to happen.

[2025-08-26T10:13:14.863Z] Already on 'trunk'
[2025-08-26T10:13:14.863Z] Your branch is up to date with 'origin/trunk'.
[2025-08-26T10:13:14.863Z] HEAD is now at 9d2a83d18ba HADOOP-19636. [JDK17] Remove CentOS 7 Support and Clean Up Dockerfile. (#7822)
[2025-08-26T10:13:15.444Z] 
[2025-08-26T10:13:15.444Z] Testing https://github.com/apache/hadoop/pull/7900 patch on trunk.
[2025-08-26T10:13:15.444Z] ERROR: Dockerfile '/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7900@2/rockylinux-8/src/dev-support/docker/Dockerfile_rockylinux_8' not found.

@pan3793 Currently, Hadoop's Yetus performs the build twice: first with the trunk code, and then again after applying the patch. At this point, I’m not certain whether what you pointed out is actually an issue.

@aajisaka @ayushtkn @GauthamBanasandra Could you please take a look at this issue? Thank you very much!

@aajisaka
Copy link
Member

Would you push the commit to a new branch and then create a test PR based on the branch to check Yetus is working fine?

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 0s Docker mode activated.
-1 ❌ docker 0m 20s Dockerfile '/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7900/rockylinux-8/src/dev-support/docker/Dockerfile_rockylinux_8' not found.
Subsystem Report/Notes
GITHUB PR #7900
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7900/4/console
versions git=2.34.1
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@pan3793
Copy link
Member Author

pan3793 commented Aug 27, 2025

Would you push the commit to a new branch and then create a test PR based on the branch to check Yetus is working fine?

@aajisaka I don't have the permission to commit hadoop repo ...

@slfan1989
Copy link
Contributor

Would you push the commit to a new branch and then create a test PR based on the branch to check Yetus is working fine?

@aajisaka I don't have the permission to commit hadoop repo ...

@pan3793 let’s keep the file name unchanged as Dockerfile_centos_8. If needed, I can help create a branch.

@pan3793
Copy link
Member Author

pan3793 commented Aug 27, 2025

with the help of @slfan1989, branch HADOOP-19661 was created at apache/hadoop repo, I opened #7898 targeting branch HADOOP-19661, the Dockerfile_rockylinux_8 not found issue is gone, but it seems platform change is not detected thus the Docker building test was skipped ...

image

@aajisaka do you have other suggestions?

@slfan1989
Copy link
Contributor

with the help of @slfan1989, branch HADOOP-19661 was created at apache/hadoop repo, I opened #7898 targeting branch HADOOP-19661, the Dockerfile_rockylinux_8 not found issue is gone, but it seems platform change is not detected thus the Docker building test was skipped ...

image @aajisaka do you have other suggestions?

@pan3793 Although we can compile locally, there are some issues with Yetus, so I cannot confirm whether this PR will affect other team members' code submissions. I still need @aajisaka @GauthamBanasandra to help verify it.

cc: @steveloughran @ayushtkn

@hadoop-yetus
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7900/5/console in case of problems.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 23m 24s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 shellcheck 0m 0s Shellcheck was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+0 🆗 hadolint 0m 0s hadolint was not available.
+0 🆗 jsonlint 0m 0s jsonlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 44m 28s trunk passed
+1 💚 mvnsite 23m 54s trunk passed
+1 💚 shadedclient 106m 45s branch has no errors when building and testing our client artifacts.
-0 ⚠️ patch 107m 19s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 💚 mvninstall 34m 26s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 20m 8s the patch passed
+1 💚 shadedclient 60m 45s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 16m 17s root in the patch passed.
+1 💚 asflicense 1m 45s The patch does not generate ASF License warnings.
245m 14s
Subsystem Report/Notes
Docker ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7900/5/artifact/out/Dockerfile
GITHUB PR #7900
Optional Tests dupname asflicense codespell detsecrets shellcheck shelldocs hadolint mvnsite unit jsonlint
uname Linux 5842e2ca7744 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / b2e0372
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7900/5/testReport/
Max. process+thread count 554 (vs. ulimit of 5500)
modules C: . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7900/5/console
versions git=2.43.7 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@pan3793
Copy link
Member Author

pan3793 commented Aug 29, 2025

I have to rename it back to workaround the Dockerfile_rockylinux_8 not found issue in b2e0372.

@slfan1989 @aajisaka, please take a look, and ping me to revert the renaming change if you think it's ready to go.

@slfan1989
Copy link
Contributor

@pan3793 LGTM. I don't see any issues. I checked the compilation results and didn't find any problems.

@slfan1989
Copy link
Contributor

@cnauroth @stoty Could you please help review this PR? Thank you very much!

Copy link
Contributor

@cnauroth cnauroth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. I'll commit this.

This reverts commit b2e0372.
@cnauroth cnauroth closed this in 5666603 Aug 29, 2025
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 0s Docker mode activated.
-1 ❌ patch 0m 20s #7900 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.
Subsystem Report/Notes
GITHUB PR #7900
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7900/6/console
versions git=2.34.1
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@pan3793
Copy link
Member Author

pan3793 commented Aug 29, 2025

@cnauroth I need to revert the renaming change before committing to trunk ... let me send a follow up to address it

@cnauroth
Copy link
Contributor

@cnauroth I need to revert the renaming change before committing to trunk ... let me send a follow up to address it

@pan3793 , oops, I committed. :-D LMK, and I'll watch for more patches required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants