HADOOP-19057. S3A: Landsat bucket used in tests no longer accessible (#6515)

[steveloughran]

The AWS landsat data previously used in some S3A tests is no longer accessible

This PR moves to the new external file
s3a://noaa-cors-pds/raw/2024/001/akse/AKSE001x.24_.gz

• Large enough file for scale tests
• Bucket supports anonymous access
• Ends in .gz to keep codec tests happy
• No spaces in path to keep bucket-info happy

Test Code Changes

• Leaves the test key name alone: fs.s3a.scale.test.csvfile
• Rename all methods and fields move remove "csv" from their names and move to "external file" we no longer require it to be CSV.
• Path definition and helper methods have been moved to PublicDatasetTestUtils
• Improve error reporting in ITestS3AInputStreamPerformance if the file is too short

This is the V1 SDK version of the patch; it has deleted ITestAWSStatisticCollection as part of the changes.

With S3 Select removed, there is no need for the file to be a CSV file; there is a test which tries to unzip it; other tests have a minimum file size.

Consult the JIRA for the settings to add to auth-keys.xml to switch earlier builds to this same file.

Contributed by Steve Loughran

How was this patch tested?

s3 london -Dparallel-tests -DtestsThreadCount=8 -Dscale

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[steveloughran]

this is the cherrypick of #6515 to branch-3.3, moving to the v1 sdk.

a couple of places I left the variables as csvPath as is simplified the test, cut some methods and a whole test suite as they were obsolete and not worth reworking.

All tests which weren't skipped passed

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	4m 3s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 18 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	33m 16s		branch-3.3 passed
+1 💚	compile	0m 26s		branch-3.3 passed
+1 💚	checkstyle	0m 20s		branch-3.3 passed
+1 💚	mvnsite	0m 30s		branch-3.3 passed
+1 💚	javadoc	0m 27s		branch-3.3 passed
+1 💚	spotbugs	0m 50s		branch-3.3 passed
+1 💚	shadedclient	20m 57s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 26s		the patch passed
+1 💚	compile	0m 20s		the patch passed
+1 💚	javac	0m 20s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 13s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 2 new + 18 unchanged - 1 fixed = 20 total (was 19)
+1 💚	mvnsite	0m 23s		the patch passed
+1 💚	javadoc	0m 17s		the patch passed
+1 💚	spotbugs	0m 46s		the patch passed
+1 💚	shadedclient	20m 52s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 47s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		88m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6548/1/artifact/out/Dockerfile
GITHUB PR	#6548
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle xmllint
uname	Linux 0e4d2ce2f9c6 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / c079d43
Default Java	Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6548/1/testReport/
Max. process+thread count	657 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6548/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 18 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	35m 0s		branch-3.3 passed
+1 💚	compile	0m 25s		branch-3.3 passed
+1 💚	checkstyle	0m 20s		branch-3.3 passed
+1 💚	mvnsite	0m 32s		branch-3.3 passed
+1 💚	javadoc	0m 27s		branch-3.3 passed
+1 💚	spotbugs	0m 49s		branch-3.3 passed
+1 💚	shadedclient	23m 15s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 26s		the patch passed
+1 💚	compile	0m 22s		the patch passed
+1 💚	javac	0m 22s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 13s		hadoop-tools/hadoop-aws: The patch generated 0 new + 18 unchanged - 1 fixed = 18 total (was 19)
+1 💚	mvnsite	0m 25s		the patch passed
+1 💚	javadoc	0m 17s		the patch passed
+1 💚	spotbugs	0m 47s		the patch passed
+1 💚	shadedclient	20m 45s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 48s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		88m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6548/2/artifact/out/Dockerfile
GITHUB PR	#6548
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle xmllint
uname	Linux c1dab86f5142 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / feb4a2c
Default Java	Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6548/2/testReport/
Max. process+thread count	551 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6548/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.