Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use write function callback without copying data #88

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

lukepalmer
Copy link
Contributor

This change provides an additional signature variation for CURLOPT_WRITEFUNCTION that invokes the callback without first copying buffer contents. This approximately doubles performance for workloads bound by throughput.

Detail:

In benchmarks I found that for HTTP payloads 512kB and over, nearly the entire cost of using libcurl is copying payloads into the application.

The intended use of the write function callback is (usually) for the application to accumulate chunks of data into an assembled payload. This (usually) requires copying the data on the application side, for example into a larger buffer.

Existing implementations of this callback in ocurl allocate and copy an OCaml string before invoking the callback, which then (likely) has to copy the data again. In this implementation we expose libcurl's buffer to the application so the allocation and copy can happen only once.

The downside of this signature is that it is possible to do unsafe things with the buffer, like access it outside of the callback or write to it.

@lukepalmer lukepalmer force-pushed the writefunction-buf branch 2 times, most recently from e93e384 to b93868d Compare February 1, 2025 20:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant