Skip to content

out_azure_blob: added buffering to azure blob #10263

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 14 commits into
base: master
Choose a base branch
from

Conversation

tanmaya-panda1
Copy link
Contributor

@tanmaya-panda1 tanmaya-panda1 commented Apr 28, 2025

This pull request introduces significant enhancements to the Azure Blob plugin, focusing on adding buffering capabilities and improving URI handling.

Buffering Enhancements

  • Added new configuration options and fields to support buffering, such as buffering_enabled, buffer_dir, store_dir_limit_size, and current_buffer_size in the flb_azure_blob struct. These changes enable the plugin to buffer data before uploading to Azure Blob Storage.
  • Introduced validation to ensure buffering is not used with the appendblob blob type, as it is incompatible. An error is logged if this configuration is detected.

URI Handling Improvements

  • Updated the azb_block_blob_uri and azb_block_blob_uri_commit functions to include a new random_str parameter for generating more unique URIs for block blobs. This ensures better handling of concurrent uploads.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Signed-off-by: Tanmaya Panda <[email protected]>
@tanmaya-panda1 tanmaya-panda1 changed the title Feature/added buffering to azure blob out_azure_blob: added buffering to azure blob May 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant