Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perform HEAD request for HttpStore::head #4837

Merged
merged 5 commits into from
Sep 23, 2023

Conversation

tustvold
Copy link
Contributor

Which issue does this PR close?

Closes #.

Rationale for this change

Issuing a head request instead of a GET request allows HttpStore::head to be used against regular HTTP servers that don't speak WebDav.

What changes are included in this PR?

Are there any user-facing changes?

Technically perhaps, but we already enforced that GET requests return the necessary headers for metadata, and so this is just an evolution of that to HEAD requests.

@github-actions github-actions bot added the object-store Object Store Interface label Sep 18, 2023
@kylebarron
Copy link
Contributor

kylebarron commented Sep 19, 2023

If I'm following the code changes correctly, a possible user-facing change for this is with signed urls, because those are only valid for either GET or HEAD (but not both) at one time, IIRC. So sometimes using a GET request instead of HEAD is nicer. Related to kylebarron/parquet-wasm#272

@tustvold
Copy link
Contributor Author

tustvold commented Sep 19, 2023

TBC previously this issued a WebDav specific PROPFIND request not a GET request, so is extremely unlikely to have worked with signed URLs

@universalmind303
Copy link
Contributor

LGTM, We (GlareDB) are already running a wrapper around objectstore for this exact reason. Using HEAD seems much more flexible than PROPFIND.

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the code carefully and I think this change makes sense to me.

Is there any way to test this change (with the standard admonition "how would we know if we broke the behavior in a future refactoring")?

store: T::STORE,
source: Box::new(e),
let meta =
header_meta(location, response.headers(), T::HEADER_CONFIG).map_err(|e| {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It took me a while to track down that the default for HEADER_CONFIG is the same as HeaderConfig::default and thus this isn't a change for S3Client and other implementations

Comment on lines +31 to +35
/// Configure the [`HeaderConfig`] for this client
const HEADER_CONFIG: HeaderConfig = HeaderConfig {
etag_required: true,
last_modified_required: true,
};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think using Default would make the intent clearer here

Suggested change
/// Configure the [`HeaderConfig`] for this client
const HEADER_CONFIG: HeaderConfig = HeaderConfig {
etag_required: true,
last_modified_required: true,
};
/// Configure the [`HeaderConfig`] used for requests issued by this client
const HEADER_CONFIG: HeaderConfig = HeaderConfig::default();

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sadly this isn't possible as Default is not const

impl GetClient for Client {
const STORE: &'static str = "HTTP";

const HEADER_CONFIG: HeaderConfig = HeaderConfig {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps you can add an explanatory comment about why this client overrides the default HEADER_CONFIG (so as to support http servers that don't implement WEBDAV?)

@tustvold tustvold merged commit 431be3f into apache:master Sep 23, 2023
13 checks passed
ryanaston pushed a commit to segmentio/arrow-rs that referenced this pull request Nov 6, 2023
* Perform HEAD request for HttpStore::head

* Logical merge conflicts

* Review feedback
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
object-store Object Store Interface
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants