HADOOP-19141. Vector IO: Update default values consistently

[dongjoon-hyun]

Description of PR

This PR updates two Vector IO default values.

- public static final int DEFAULT_AWS_S3_VECTOR_READS_MIN_SEEK_SIZE = 4896; // 4K
+ public static final int DEFAULT_AWS_S3_VECTOR_READS_MIN_SEEK_SIZE = 4096; // 4K
- public static final int DEFAULT_AWS_S3_VECTOR_READS_MAX_MERGED_READ_SIZE = 1253376; //1M
+ public static final int DEFAULT_AWS_S3_VECTOR_READS_MAX_MERGED_READ_SIZE = 1048576; //1M

If a user set a configuration like the documented 4K or 1M, Apache Hadoop will use 4096 and 1048576 which are different from the AS-IS defined value of Constants.java. Thus, this PR fixes Constants.java with the values, 4096 and 1048576.

hadoop/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

Lines 779 to 782 in 87fb977

	final int minSeekVectored = (int) longBytesOption(conf, AWS_S3_VECTOR_READS_MIN_SEEK_SIZE,
	DEFAULT_AWS_S3_VECTOR_READS_MIN_SEEK_SIZE, 0);
	final int maxReadSizeVectored = (int) longBytesOption(conf, AWS_S3_VECTOR_READS_MAX_MERGED_READ_SIZE,
	DEFAULT_AWS_S3_VECTOR_READS_MAX_MERGED_READ_SIZE, 0);

How was this patch tested?

1. Apache Hadoop uses 1M as 1048576 = 1024 x 1024 historically like the following.
   

   hadoop/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java

   Line 521 in 87fb977

   public static final int DEFAULT_FAST_BUFFER_SIZE = 1048576; //1MB

   So, the default value of Vector IO is inconsistent from not only the other values but also from its comment, //1M, and its documentation, 1M.
   

   hadoop/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java

   Line 1431 in 87fb977

   public static final int DEFAULT_AWS_S3_VECTOR_READS_MAX_MERGED_READ_SIZE = 1253376; //1M

   hadoop/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md

   Line 79 in 87fb977

   <value>1M</value>

2. The AS-IS default values of Vector IO are also inconsistent from the values of Vector IO test cases.
   

   hadoop/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractVectoredRead.java

   Lines 144 to 147 in 87fb977

   	final int configuredMinSeek = 2 * 1024;
   	final int configuredMaxSize = 10 * 1024 * 1024;
   	conf.set(Constants.AWS_S3_VECTOR_READS_MIN_SEEK_SIZE, "2K");
   	conf.set(Constants.AWS_S3_VECTOR_READS_MAX_MERGED_READ_SIZE, "10M");

3. The inconsistency happens for 4K in the same way.
   

   hadoop/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java

   Line 1426 in 87fb977

   public static final int DEFAULT_AWS_S3_VECTOR_READS_MIN_SEEK_SIZE = 4896; // 4K

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[dongjoon-hyun]

Could you review this, @steveloughran ?

I filed this as a subtask of HADOOP-18855. If it's inappropriate, I'll convert this to an independent JIRA. Please let me know.

[steveloughran]

looks good to me -suspect it was a latent typo or something.

Note that org.apache.hadoop.fs.azure.integration.Sizes defines some of the sizes; there's something else in bits of hadoop-aws tests. really we should have something in hadoop-common.

side issue: have you tested this yet? as right now I know the delegation token mr job fails until a pending pr is merged

[dongjoon-hyun]

Thank you for review. Yes, I've been testing S3 Express One Zone and Vector IO actively with Apache Spark 4.0.0 SNAPSHOT (including ORC 2.0.0) since last week. I hope we can stabilize as much as possible.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	46m 3s		trunk passed
+1 💚	compile	0m 43s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	0m 35s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	checkstyle	0m 31s		trunk passed
+1 💚	mvnsite	0m 43s		trunk passed
+1 💚	javadoc	0m 26s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	0m 32s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	1m 9s		trunk passed
+1 💚	shadedclient	37m 14s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s		the patch passed
+1 💚	compile	0m 35s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	0m 35s		the patch passed
+1 💚	compile	0m 26s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 19s		the patch passed
+1 💚	mvnsite	0m 32s		the patch passed
+1 💚	javadoc	0m 16s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	0m 25s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	1m 9s		the patch passed
+1 💚	shadedclient	35m 28s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 51s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		135m 47s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6702/1/artifact/out/Dockerfile
GITHUB PR	#6702
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 2d4172ad12e5 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 52cbf5c
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6702/1/testReport/
Max. process+thread count	627 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6702/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[dongjoon-hyun]

CI seems to be happy except test4tests part.

[steveloughran]

LGTM
+1

[steveloughran]

merging to trunk and branch 3.4/3.3

[dongjoon-hyun]

Thank you, @steveloughran !

[mukund-thakur]

Not sure how this happened but thanks for fixing this.