Client support for receiving 1xx informational responses

[seanmonstar]

This is to discuss adding support for accessing 1xx informational responses on the client side (see #2426 for server side). Currently, hyper parses all 1xx responses and ignores them, as RFC 7230 says it must at least be able to receive and ignore them.

The Client (and client::conn) interfaces return an impl Future<Output = Result<Response>>, which only allows awaiting it for a single response. We can't really add new methods to allow also awaiting for informational responses, unless we stuck to always return a specialized type that is both a Future and has extra methods. However, using those together wouldn't be that clear to a user. You can't be sure if or how many informational responses would be received before the full response, so you would want to await "both" types.

Instead, we can use a callback format. We could define a type (hyper::client::ext::Informational? On1xx? names...), and the user would store that in the request extensions. We'd then check for that in the h1 proto code and if it exists, call the callback with the 1xx headers.

[nox]

I much prefer the idea of adding methods on ResponseFuture, hooks make bookkeeping harder for the caller, and I don't think it is more confusing to have both those methods and the Future impl than say, using poll_data and poll_trailers on HttpBody.

[bagder]

In the curl project, there's right now 105 test cases disabled when built with hyper. 46 of them use 100-continue in one way or another and we really need to know when we get 1xx responses (or not) to make curl+hyper behave like the built-in HTTP code does. It could allow us to make significant progress in my task to make curl+hyper match the built-in HTTP code.

[seanmonstar]

Right-o, I got distracted with other work. I'll do some thinking and work on this in the next few days.

[seanmonstar]

I've spent some time researching options (prior art of APIs), and considering the existing code, and I think I've settled on staged approach. I agree with you, @nox, we can add a method to ResponseFuture on the Rust side, and make it work right. However, it'd be less easy to access in the exposed C API, and will take more time to get right. So:

1. First, add support for a callback extension, which will be exposed in the C API. This is the least complicated to do on hyper's side, so we can do it quickly. Importantly, though, most of it can be reused or only slightly altered to support the next steps.
2. Because of the current client design, which separates the ResponseFuture from the separate client dispatcher task running in the background, we'd need some to add some sort of channel where the receiver is inside ResponseFuture, and the sender is in a request extension. Instead of storing the sender directly, we can reuse the callback to simply store a closure sending on the sender side of the channel.
3. Finally, the higher level ResponseFuture (returned by hyper::Client, not hyper::client::conn) would need to separate some of its stages into an internal enum, so that polling ResponseFuture::poll_informational() could advance the stages far enough that the task wouldn't be stuck since the dns/connect/write stage hadn't gotten far enough.

---

The first step is my next work priority.

[seanmonstar]

Step 1 PR is available at #2594.

[seanmonstar]

@bagder the piece you'd need is merged to master now. hyper's C API doesn't use timers at the moment, so you'd need to use a timer in the normal curl way, and check in the body poll callback whether a 100 Continue has been received (in your informational callback, you could set a flag and trigger the body waker to start polling again) or the timer has expired.

[bagder]

I would've liked the callback to take a return code so that we can return error easier.

If my code in the callback runs into a fatal error, such as out of memory, I need to return error and abort everything. Instead of returning error, now I need to pass that info back to the parent caller by storing a magic value in a struct field instead. Not your typical erroring out style code.

[bagder]

What's perhaps worse, is that it doesn't seem to work? At least not the way I thought it is supposed to work. Shouldn't the following callback code be able to extract the 100 response and its associated response headers?

When I run this in curl test 158, the header callback doesn't get invoked even once. I suppose I've done something wrong?

https://github.com/curl/curl/blob/931a3e33e11c75feca367b90532d95d6f17985f7/lib/c-hyper.c#L726-L750

[seanmonstar]

I would've liked the callback to take a return code so that we can return error easier.

We could probably adjust it to be more like the foreach functions, returning a HYPERE_ABORTED_BY_CALLBACK if you return a non-zero.

Shouldn't the following callback code be able to extract the 100 response and its associated response headers?

It should... I added it's usage to the upload.c example, and checked it worked there. I'll dig a little into how you're using it.