Use write function callback without copying data #88
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change provides an additional signature variation for CURLOPT_WRITEFUNCTION that invokes the callback without first copying buffer contents. This approximately doubles performance for workloads bound by throughput.
Detail:
In benchmarks I found that for HTTP payloads 512kB and over, nearly the entire cost of using libcurl is copying payloads into the application.
The intended use of the write function callback is (usually) for the application to accumulate chunks of data into an assembled payload. This (usually) requires copying the data on the application side, for example into a larger buffer.
Existing implementations of this callback in ocurl allocate and copy an OCaml string before invoking the callback, which then (likely) has to copy the data again. In this implementation we expose libcurl's buffer to the application so the allocation and copy can happen only once.
The downside of this signature is that it is possible to do unsafe things with the buffer, like access it outside of the callback or write to it.