Supply gcs-connector options from default Configuration #5549
Conversation
| val config = new Configuration() | ||
| val o = optionsAs[GcsOptions] | ||
|
|
||
| o.setGoogleCloudStorageReadOptions( |
There was a problem hiding this comment.
There's a method to parse all this for us but it's not public: https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v3.0.4/gcs/src/main/java/com/google/cloud/hadoop/fs/gcs/GoogleHadoopFileSystemConfiguration.java#L645-L660
I do have a PR up to make it public so hopefully it can be dropped in in a future release.
There was a problem hiding this comment.
Link the PR in the code somewhere so we can find it later
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #5549 +/- ##
==========================================
+ Coverage 61.31% 61.32% +0.01%
==========================================
Files 314 314
Lines 11249 11236 -13
Branches 770 778 +8
==========================================
- Hits 6897 6891 -6
+ Misses 4352 4345 -7 ☔ View full report in Codecov by Sentry. |
| val config = new Configuration() | ||
| val o = optionsAs[GcsOptions] | ||
|
|
||
| o.setGoogleCloudStorageReadOptions( |
There was a problem hiding this comment.
Link the PR in the code somewhere so we can find it later
apache/beam#32769 is released in Beam 2.62.0; this PR uses the new GcsOption to pass all available gcs-connector options, parsed from the default
Configuration(i.e. core-site.xml).