Skip to content

HADOOP-19659. Upgrade Debian 10 to 11 in build env Dockerfile #7898

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: HADOOP-19661
Choose a base branch
from
Open

HADOOP-19659. Upgrade Debian 10 to 11 in build env Dockerfile #7898

wants to merge 1 commit into from
+67 −66

Conversation

pan3793
Copy link
Member

@pan3793 pan3793 commented Aug 26, 2025

Description of PR

Debian 10 EOL, and the apt repo is unavailable, this PR upgrades it to Debian 11

docker run --rm -it debian:10 bash
root@bc2a4c509cb3:/# apt update
Ign:1 http://deb.debian.org/debian buster InRelease
Ign:2 http://deb.debian.org/debian-security buster/updates InRelease
Ign:3 http://deb.debian.org/debian buster-updates InRelease
Err:4 http://deb.debian.org/debian buster Release
  404  Not Found [IP: 151.101.90.132 80]
Err:5 http://deb.debian.org/debian-security buster/updates Release
  404  Not Found [IP: 151.101.90.132 80]
Err:6 http://deb.debian.org/debian buster-updates Release
  404  Not Found [IP: 151.101.90.132 80]
Reading package lists... Done
E: The repository 'http://deb.debian.org/debian buster Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://deb.debian.org/debian-security buster/updates Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://deb.debian.org/debian buster-updates Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
root@bc2a4c509cb3:/#

Debian 11 will be EOL after 31 Aug 2026. I didn't switch it to Debian 12 or 13 because the new version does not have openjdk-11-jdk in the apt repo.

How was this patch tested?

$ ./start-build-env.sh debian_11
...
 _   _           _                    ______
| | | |         | |                   |  _  \
| |_| | __ _  __| | ___   ___  _ __   | | | |_____   __
|  _  |/ _` |/ _` |/ _ \ / _ \| '_ \  | | | / _ \ \ / /
| | | | (_| | (_| | (_) | (_) | |_) | | |/ /  __/\ V /
\_| |_/\__,_|\__,_|\___/ \___/| .__/  |___/ \___| \_(_)
                              | |
                              |_|

This is the standard Hadoop Developer build environment.
This has all the right tools installed required to build
Hadoop from source.

chengpan@c85a4426ba52:~/hadoop$

$ mvn clean install -DskipTests -Pnative -Pyarn-ui -DskipShade
...
[INFO] Reactor Summary for Apache Hadoop Main 3.5.0-SNAPSHOT:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [  0.662 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  1.010 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  0.592 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  0.618 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  0.069 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.109 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  1.754 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  0.416 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  2.437 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  0.581 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 22.868 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  1.273 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [  1.393 s]
[INFO] Apache Hadoop Registry ............................. SUCCESS [  1.533 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.040 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 11.693 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 22.103 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [02:13 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [  2.320 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  1.312 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.042 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [  7.877 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [  6.436 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.038 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [  5.022 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [  2.078 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  1.541 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  1.331 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 13.285 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 24.058 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  1.467 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [  2.682 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.682 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [  5.570 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [  2.672 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  1.767 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  3.697 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  2.380 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  4.119 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  2.581 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  1.206 s]
[INFO] Apache Hadoop Federation Balance ................... SUCCESS [  1.630 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 15.224 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.035 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  1.195 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  1.104 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [  0.034 s]
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [  1.532 s]
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [  4.403 s]
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [  0.039 s]
[INFO] Apache Hadoop YARN TimelineService HBase Server 2.5  SUCCESS [  1.835 s]
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [  1.916 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [  2.665 s]
[INFO] Apache Hadoop YARN TimelineService DocumentStore ... SUCCESS [  1.284 s]
[INFO] Apache Hadoop YARN GlobalPolicyGenerator ........... SUCCESS [  1.474 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.033 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  1.413 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  0.925 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [  0.033 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [  3.138 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [  1.914 s]
[INFO] Apache Hadoop YARN Application Catalog ............. SUCCESS [  0.038 s]
[INFO] Apache Hadoop YARN Application Catalog Webapp ...... SUCCESS [ 11.624 s]
[INFO] Apache Hadoop YARN Application Catalog Docker Image  SUCCESS [  0.048 s]
[INFO] Apache Hadoop YARN Application MaWo ................ SUCCESS [  0.035 s]
[INFO] Apache Hadoop YARN Application MaWo Core ........... SUCCESS [  0.991 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.075 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  0.458 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [02:07 min]
[INFO] Apache Hadoop YARN CSI ............................. SUCCESS [  3.403 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [  1.283 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  1.046 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [ 18.413 s]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [  1.040 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  1.912 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  1.140 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  2.034 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [  0.728 s]
[INFO] Apache Hadoop Dynamometer Workload Simulator ....... SUCCESS [  1.363 s]
[INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SUCCESS [  1.878 s]
[INFO] Apache Hadoop Dynamometer Block Listing Generator .. SUCCESS [  1.241 s]
[INFO] Apache Hadoop Dynamometer Dist ..................... SUCCESS [  1.161 s]
[INFO] Apache Hadoop Dynamometer .......................... SUCCESS [  0.034 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.310 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  1.536 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  1.914 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  1.920 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.347 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  1.377 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  2.679 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  7.690 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [  0.901 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  5.553 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [  1.183 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  1.679 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [  1.213 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  1.156 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [  1.256 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  0.704 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  0.047 s]
[INFO] Apache Hadoop Common Benchmark ..................... SUCCESS [  9.812 s]
[INFO] Apache Hadoop Compatibility Benchmark .............. SUCCESS [  0.853 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.032 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [  0.911 s]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [  0.763 s]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [  0.367 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [  0.756 s]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [  0.174 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [  0.778 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [  0.493 s]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [  0.034 s]
[INFO] Apache Hadoop Tencent COS Support .................. SUCCESS [ 56.113 s]
[INFO] Apache Hadoop OBS support .......................... SUCCESS [  1.335 s]
[INFO] Apache Hadoop Volcano Engine Services support ...... SUCCESS [  2.005 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  0.444 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.032 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  10:25 min
[INFO] Finished at: 2025-08-26T05:03:57Z
[INFO] ------------------------------------------------------------------------

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

pan3793
pan3793 commented Aug 26, 2025
View reviewed changes
start-build-env.sh Outdated
@@ -93,7 +93,7 @@ RUN userdel -r \$(getent passwd ${USER_ID} | cut -d: -f1) 2>/dev/null || :
RUN groupadd --non-unique -g ${GROUP_ID} ${USER_NAME}
RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME} -d "${DOCKER_HOME_DIR}"
RUN echo "${USER_NAME} ALL=NOPASSWD: ALL" > "/etc/sudoers.d/hadoop-build-${USER_ID}"
ENV HOME "${DOCKER_HOME_DIR}"
ENV HOME="${DOCKER_HOME_DIR}"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update because

 1 warning found (use docker --debug to expand):
 - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 7)

@slfan1989
Copy link
Contributor

LGTM.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 22m 6s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 shellcheck 0m 0s Shellcheck was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+0 🆗 hadolint 0m 0s hadolint was not available.
+0 🆗 jsonlint 0m 0s jsonlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 40m 22s trunk passed
-1 ❌ mvnsite 17m 26s /branch-mvnsite-root.txt root in trunk failed.
+1 💚 shadedclient 94m 11s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 32m 16s the patch passed
+1 💚 blanks 0m 1s The patch has no blanks issues.
-1 ❌ mvnsite 11m 20s /patch-mvnsite-root.txt root in the patch failed.
+1 💚 shadedclient 47m 38s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 14m 13s root in the patch passed.
+1 💚 asflicense 1m 57s The patch does not generate ASF License warnings.
214m 23s
Subsystem Report/Notes
Docker ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7898/1/artifact/out/Dockerfile
GITHUB PR #7898
Optional Tests dupname asflicense codespell detsecrets shellcheck shelldocs hadolint mvnsite unit jsonlint
uname Linux 71b5c8e33180 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / f40defe
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7898/1/testReport/
Max. process+thread count 707 (vs. ulimit of 5500)
modules C: . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7898/1/console
versions git=2.27.0 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989
slfan1989 requested changes Aug 26, 2025
View reviewed changes
Copy link
Contributor

@slfan1989 slfan1989 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to solve the compilation error issue.

@pan3793
Copy link
Member Author

pan3793 commented Aug 26, 2025

We need to solve the compilation error issue.

The building error occurs on CentOS 8 stage, looks like I need to fix it first

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 0s Docker mode activated.
-1 ❌ docker 0m 19s Dockerfile '/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7898/rockylinux-8/src/dev-support/docker/Dockerfile_rockylinux_8' not found.
Subsystem Report/Notes
GITHUB PR #7898
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7898/2/console
versions git=2.34.1
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@pan3793 pan3793 changed the base branch from trunk to HADOOP-19661 August 27, 2025 05:57
@hadoop-yetus
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7898/3/console in case of problems.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 20m 58s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 shelldocs 0m 1s Shelldocs was not available.
+0 🆗 jsonlint 0m 1s jsonlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ HADOOP-19661 Compile Tests _
+1 💚 mvninstall 45m 24s HADOOP-19661 passed
+1 💚 mvnsite 22m 56s HADOOP-19661 passed
+1 💚 shadedclient 40m 21s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 36m 26s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 hadolint 0m 2s No new issues.
+1 💚 mvnsite 20m 1s the patch passed
+1 💚 shellcheck 0m 0s No new issues.
+1 💚 shadedclient 40m 8s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 16m 4s root in the patch passed.
+1 💚 asflicense 1m 5s The patch does not generate ASF License warnings.
246m 11s
Subsystem Report/Notes
Docker ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7898/3/artifact/out/Dockerfile
GITHUB PR #7898
Optional Tests dupname asflicense codespell detsecrets hadolint shellcheck shelldocs mvnsite unit jsonlint
uname Linux 1b828c1875fb 5.15.0-144-generic #157-Ubuntu SMP Mon Jun 16 07:33:10 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19661 / 7f02ba4
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7898/3/testReport/
Max. process+thread count 588 (vs. ulimit of 5500)
modules C: . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7898/3/console
versions git=2.25.1 maven=3.6.3 hadolint=1.11.1-0-g0e692dd shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@pan3793 pan3793 mentioned this pull request Aug 27, 2025
Open
4 tasks
@stoty
Copy link
Contributor

stoty commented Aug 27, 2025

I think it would be useful to run at least the native (C code) tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@slfan1989 slfan1989 slfan1989 requested changes

Assignees
No one assigned
Labels
Infra trunk
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@pan3793 @slfan1989 @hadoop-yetus @stoty