HDFS-17456. Fix the incorrect dfsused statistics of datanode when appending a file. #6713

fuchaohong · 2024-04-07T06:46:42Z

JIRA: HDFS-17456.
The effect of this patch is that when the replica is moved from the finalized directory to the rbw directory, dfsused will subtract the replica length.

… file.

ZanderXu

LGTM. Maybe we can use replicaInfo.getNumBytes()

...p-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java

ZanderXu

LGTM.

@fuchaohong please describe this case in the Description window, thanks.

fuchaohong · 2024-04-07T09:25:09Z

@ZanderXu done the describe, thanks.

hadoop-yetus · 2024-04-07T13:14:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	45m 16s		trunk passed
+1 💚	compile	1m 20s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	1m 17s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	checkstyle	1m 11s		trunk passed
+1 💚	mvnsite	1m 26s		trunk passed
+1 💚	javadoc	1m 9s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 40s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	3m 17s		trunk passed
+1 💚	shadedclient	36m 1s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 13s		the patch passed
+1 💚	compile	1m 14s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	1m 14s		the patch passed
+1 💚	compile	1m 10s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	javac	1m 10s		the patch passed
+1 💚	blanks	0m 1s		The patch has no blanks issues.
+1 💚	checkstyle	1m 0s		the patch passed
+1 💚	mvnsite	1m 14s		the patch passed
+1 💚	javadoc	0m 54s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 38s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	3m 18s		the patch passed
+1 💚	shadedclient	35m 26s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	232m 13s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 46s		The patch does not generate ASF License warnings.
		386m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6713/1/artifact/out/Dockerfile
GITHUB PR	#6713
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 30389c7458ec 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `bce57ed`
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6713/1/testReport/
Max. process+thread count	3540 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6713/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-04-07T14:15:21Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	44m 19s		trunk passed
+1 💚	compile	1m 21s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	1m 17s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	checkstyle	1m 10s		trunk passed
+1 💚	mvnsite	1m 26s		trunk passed
+1 💚	javadoc	1m 7s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 44s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	3m 15s		trunk passed
+1 💚	shadedclient	35m 35s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 10s		the patch passed
+1 💚	compile	1m 14s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	1m 14s		the patch passed
+1 💚	compile	1m 8s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	javac	1m 8s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 57s		the patch passed
+1 💚	mvnsite	1m 12s		the patch passed
+1 💚	javadoc	0m 52s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 35s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	3m 14s		the patch passed
+1 💚	shadedclient	35m 29s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	227m 27s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 46s		The patch does not generate ASF License warnings.
		368m 13s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6713/2/artifact/out/Dockerfile
GITHUB PR	#6713
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 07fcd1da07a1 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ecf2c6a`
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6713/2/testReport/
Max. process+thread count	4502 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6713/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-04-24T05:49:10Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
			_ Prechecks _
+1 💚	dupname	0m 00s		No case conflicting files found.
+0 🆗	spotbugs	0m 01s		spotbugs executables are not available.
+0 🆗	codespell	0m 01s		codespell was not available.
+0 🆗	detsecrets	0m 01s		detect-secrets was not available.
+1 💚	@author	0m 00s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 00s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	89m 22s		trunk passed
+1 💚	compile	6m 03s		trunk passed
+1 💚	checkstyle	5m 02s		trunk passed
+1 💚	mvnsite	6m 49s		trunk passed
+1 💚	javadoc	6m 02s		trunk passed
+1 💚	shadedclient	148m 32s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	4m 58s		the patch passed
+1 💚	compile	3m 31s		the patch passed
+1 💚	javac	3m 31s		the patch passed
+1 💚	blanks	0m 01s		The patch has no blanks issues.
+1 💚	checkstyle	2m 22s		the patch passed
+1 💚	mvnsite	4m 12s		the patch passed
+1 💚	javadoc	3m 37s		the patch passed
+1 💚	shadedclient	159m 22s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	5m 21s		The patch does not generate ASF License warnings.
		423m 56s

Subsystem	Report/Notes
GITHUB PR	#6713
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	MINGW64_NT-10.0-17763 ebc41b0c2c32 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys
Build tool	maven
Personality	/c/hadoop/dev-support/bin/hadoop.sh
git revision	trunk / `ecf2c6a`
Default Java	Azul Systems, Inc.-1.8.0_332-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6713/1/testReport/
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6713/1/console
versions	git=2.44.0.windows.1
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Hexiaoqiao · 2024-04-29T03:58:41Z

@fuchaohong Hi, IIUC the dfsused will be refreshed periodically. Do you meet any issues about this metric? Thanks.

fuchaohong · 2024-04-29T06:32:45Z

@fuchaohong Hi, IIUC the dfsused will be refreshed periodically. Do you meet any issues about this metric? Thanks.
@Hexiaoqiao Indeed, the dfsused metric is corrected periodically, but the cycle is very long. Currently, when there are too many append operations, dfsused may expand abnormally, not reflecting the actual usage accurately. Thanks.

Hexiaoqiao

Got it. Make sense to me. +1.

Hexiaoqiao · 2024-04-30T04:23:09Z

Committed to trunk. Thanks @fuchaohong and @ZanderXu !

…ending a file. (apache#6713). Contributed by fuchaohong. Reviewed-by: ZanderXu <[email protected]> Signed-off-by: He Xiaoqiao <[email protected]>

Fix the dfsused statistics of datanode are incorrect when appending a…

bce57ed

… file.

github-actions bot added HDFS trunk labels Apr 7, 2024

fuchaohong changed the title ~~Fix the dfsused statistics of datanode are incorrect when appending a file.~~ HDFS-17456. Fix the dfsused statistics of datanode are incorrect when appending a file. Apr 7, 2024

ZanderXu reviewed Apr 7, 2024

View reviewed changes

...p-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java Outdated Show resolved Hide resolved

Update FsVolumeImpl.java

ecf2c6a

ZanderXu reviewed Apr 7, 2024

View reviewed changes

ZanderXu approved these changes Apr 7, 2024

View reviewed changes

Hexiaoqiao approved these changes Apr 29, 2024

View reviewed changes

Hexiaoqiao changed the title ~~HDFS-17456. Fix the dfsused statistics of datanode are incorrect when appending a file.~~ HDFS-17456. Fix the incorrect dfsused statistics of datanode when appending a file. Apr 30, 2024

Hexiaoqiao merged commit 0c9e0b4 into apache:trunk Apr 30, 2024
3 of 5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-17456. Fix the incorrect dfsused statistics of datanode when appending a file. #6713

HDFS-17456. Fix the incorrect dfsused statistics of datanode when appending a file. #6713

fuchaohong commented Apr 7, 2024 •

edited

Loading

ZanderXu left a comment

ZanderXu left a comment

fuchaohong commented Apr 7, 2024

hadoop-yetus commented Apr 7, 2024

hadoop-yetus commented Apr 7, 2024

hadoop-yetus commented Apr 24, 2024

Hexiaoqiao commented Apr 29, 2024

fuchaohong commented Apr 29, 2024

Hexiaoqiao left a comment

Hexiaoqiao commented Apr 30, 2024

HDFS-17456. Fix the incorrect dfsused statistics of datanode when appending a file. #6713

HDFS-17456. Fix the incorrect dfsused statistics of datanode when appending a file. #6713

Conversation

fuchaohong commented Apr 7, 2024 • edited Loading

ZanderXu left a comment

Choose a reason for hiding this comment

ZanderXu left a comment

Choose a reason for hiding this comment

fuchaohong commented Apr 7, 2024

hadoop-yetus commented Apr 7, 2024

hadoop-yetus commented Apr 7, 2024

hadoop-yetus commented Apr 24, 2024

Hexiaoqiao commented Apr 29, 2024

fuchaohong commented Apr 29, 2024

Hexiaoqiao left a comment

Choose a reason for hiding this comment

Hexiaoqiao commented Apr 30, 2024

fuchaohong commented Apr 7, 2024 •

edited

Loading