Skip to content

Commit abda423

Browse files
esmerelesmerel
authored
Merge pull request #1892 from fluent/lynettemiles/sc-136226/update-fluent-bit-docs-pipeline-outputs-azure
2 parents 39539d0 + 56c762b commit abda423

File tree

1 file changed

+58
-55
lines changed

1 file changed

+58
-55
lines changed

‎pipeline/outputs/azure_blob.md

Lines changed: 58 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -1,55 +1,53 @@
11
---
2-
description: Official and Microsoft Certified Azure Storage Blob connector
2+
description: Microsoft certified Azure Storage Blob connector
33
---
44

55
# Azure Blob
66

77
The Azure Blob output plugin allows ingesting your records into [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/) service. This connector is designed to use the Append Blob and Block Blob API.
88

9-
Our plugin works with the official Azure Service and also can be configured to be used with a service emulator such as [Azurite](https://github.com/Azure/Azurite).
9+
The Fluent Bit plugin works with the official Azure Service and can be configured to be used with a service emulator such as [Azurite](https://github.com/Azure/Azurite).
1010

11-
## Azure Storage Account
11+
## Azure Storage account
1212

13-
Before getting started, make sure you already have an Azure Storage account. As a reference, the following link explains step-by-step how to set up your account:
13+
Ensure you have an Azure Storage account. [Azure Blob Storage Tutorial \(Video\)](https://www.youtube.com/watch?v=-sCKnOm8G_g) explains how to set up your account.
1414

15-
* [Azure Blob Storage Tutorial \(Video\)](https://www.youtube.com/watch?v=-sCKnOm8G_g)
15+
## Configuration parameters
1616

17-
## Configuration Parameters
17+
Fluent Bit exposes the following configuration properties.
1818

19-
We expose different configuration properties. The following table lists all the options available, and the next section has specific configuration details for the official service or the emulator.
20-
21-
| Key | Description | default |
19+
| Key | Description | Default |
2220
| :--- | :--- | :--- |
23-
| account\_name | Azure Storage account name. This configuration property is mandatory | |
24-
| auth\_type | Specify the type to authenticate against the service. Fluent Bit supports `key` and `sas`. | key |
25-
| shared\_key | Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory when `auth_type` is `key`. | |
26-
| sas\_token | Specify the Azure Storage shared access signatures to authenticate against the service. This configuration property is mandatory when `auth_type` is `sas`. | |
27-
| container\_name | Name of the container that will contain the blobs. This configuration property is mandatory | |
28-
| blob\_type | Specify the desired blob type. Fluent Bit supports `appendblob` and `blockblob`. | appendblob |
29-
| auto\_create\_container | If `container_name` does not exist in the remote service, enabling this option will handle the exception and auto-create the container. | on |
30-
| path | Optional path to store your blobs. If your blob name is `myblob`, you can specify sub-directories where to store it using path, so setting path to `/logs/kubernetes` will store your blob in `/logs/kubernetes/myblob`. | |
31-
| compress | Sets payload compression in network transfer. Only value supported is: `gzip` | |
32-
| compress\_blob | Enables GZIP compression in the final `blockblob` file. This option is not compatible when `blob_type` = `appendblob` | |
33-
| emulator\_mode | If you want to send data to an Azure emulator service like [Azurite](https://github.com/Azure/Azurite), enable this option so the plugin will format the requests to the expected format. | off |
34-
| endpoint | If you are using an emulator, this option allows you to specify the absolute HTTP address of such service. e.g: [http://127.0.0.1:10000](http://127.0.0.1:10000). | |
35-
| tls | Enable or disable TLS encryption. Note that Azure service requires this to be turned on. | off |
36-
| workers | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` |
37-
| buffering\_enabled | Enable buffering into disk before ingesting into Azure Blob. | false |
38-
| buffer\_dir | Specifies the location of directory where the buffered data will be stored. | /tmp/fluent-bit/azure-blob/ |
39-
| upload\_timeout | Optionally specify a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than x minutes and haven't reached upload_file_size limit yet. | 30m |
40-
| upload\_file\_size | Specifies the size of files to be uploaded in MBs. | 200M |
41-
| azure\_blob\_buffer\_key | Set the azure blob buffer key which needs to be specified when using multiple instances of azure blob output plugin and buffering is enabled. | key |
42-
| store\_dir\_limit\_size | Set the max size of the buffer directory. | 8G |
43-
| buffer\_file\_delete\_early | Whether to delete the buffered file early after successful blob creation. | false |
44-
| blob\_uri\_length | Set the length of generated blob uri before ingesting to Azure Kusto. | 64 |
45-
| unify\_tag | Whether to create a single buffer file when buffering mode is enabled. | false |
46-
| scheduler\_max\_retries | Maximum number of retries for the scheduler send blob. | 3 |
47-
| delete\_on\_max\_upload\_error | Whether to delete the buffer file on maximum upload errors. | false |
48-
| io\_timeout | HTTP IO timeout. | 60s |
49-
50-
## Getting Started
51-
52-
As mentioned above, you can either deliver records to the official service or an emulator. Below we have an example for each use case.
21+
| `account_name` | Azure Storage account name. | _none_ |
22+
| `auth_type` | Specify the type to authenticate against the service. Supported values: `key`, `sas`. | `key` |
23+
| `shared_key` | Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory when `auth_type` is `key`. | _none_ |
24+
| `sas_token` | Specify the Azure Storage shared access signatures to authenticate against the service. This configuration property is mandatory when `auth_type` is `sas`. | _none_ |
25+
| `container_name` | Name of the container that will contain the blobs. | _none_ |
26+
| `blob_type` | Specify the desired blob type. Supported values: `appendblob`, `blockblob`. | `appendblob` |
27+
| `auto_create_container` | If `container_name` doesn't exist in the remote service, enabling this option handles the exception and auto-creates the container. | `on` |
28+
| `path` | Optional. The path to store your blobs. If your blob name is `myblob`, specify subdirectories for storage using `path`. For example, setting `path` to `/logs/kubernetes` will store your blob in `/logs/kubernetes/myblob`. | _none_ |
29+
| `compress` | Sets payload compression in network transfer. Supported value: `gzip` | _none_ |
30+
| `compress_blob` | Enables GZIP compression in the final `blockblob` file. This option isn't compatible when `blob_type` = `appendblob`. | _none_ |
31+
| `emulator_mode` | To send data to an Azure emulator service like [Azurite](https://github.com/Azure/Azurite), enable this option to format the requests in the expected format. | `off` |
32+
| `endpoint` | When using an emulator, this option lets you specify the absolute HTTP address of such service. For example, `http://127.0.0.1:10000`. | _none_ |
33+
| `tls` | Enable or disable TLS encryption. Azure service requires this to be set to `on`. | `off` |
34+
| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` |
35+
| `buffering_enabled` | Enable buffering into disk before ingesting into Azure Blob. | `false` |
36+
| `buffer_dir` | Specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-blob/` |
37+
| `upload_timeout` | Optional. Specify a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than `x` minutes and ago haven't reached `upload_file_size` limit yet. | `30m` |
38+
| `upload_file_size` | Specifies the size of files to be uploaded in MB. | `200M` |
39+
| `azure_blob_buffer_key` | Set the Azure Blob buffer key which needs to be specified when using multiple instances of Azure Blob output plugin and buffering is enabled. | `key` |
40+
| `store_dir_limit_size` | Set the max size of the buffer directory. | `8G` |
41+
| `buffer_file_delete_early` | Whether to delete the buffered file early after successful blob creation. | `false` |
42+
| `blob_uri_length` | Set the length of generated blob URI before ingesting to Azure Kusto. | `64` |
43+
| `unify_tag` | Whether to create a single buffer file when buffering mode is enabled. | `false` |
44+
| `scheduler_max_retries` | Maximum number of retries for the scheduler send blob. | `3` |
45+
| `delete_on_max_upload_error` | Whether to delete the buffer file on maximum upload errors. | `false` |
46+
| `io_timeout` | HTTP IO timeout. | `60s` |
47+
48+
## Get started
49+
50+
Fluent Bit can deliver records to the official service or an emulator.
5351

5452
### Configuration for Azure Storage Service
5553

@@ -77,33 +75,38 @@ The following configuration example generates a random message with a custom tag
7775
tls on
7876
```
7977

80-
After you run the configuration file above, you will be able to query the data using the Azure Storage Explorer. The example above will generate the following content in the explorer:
78+
After you run the configuration file, you will be able to query the data using the Azure Storage Explorer. The example generates the following content in the explorer:
8179

82-
![](../../.gitbook/assets/azure_blob.png)
80+
![Azure Blob](../../.gitbook/assets/azure_blob.png)
8381

8482
### Configuring and using Azure Emulator: Azurite
8583

8684
#### Install and run Azurite
8785

88-
The quickest way to get started is to install Azurite using npm:
86+
1. Install Azurite using `npm`:
8987

90-
```bash
91-
npm install -g azurite
92-
```
88+
```bash
89+
npm install -g azurite
90+
```
9391

94-
then run the service:
92+
1. Run the service:
9593

96-
```bash
97-
$ azurite
98-
Azurite Blob service is starting at http://127.0.0.1:10000
99-
Azurite Blob service is successfully listening at http://127.0.0.1:10000
100-
Azurite Queue service is starting at http://127.0.0.1:10001
101-
Azurite Queue service is successfully listening at http://127.0.0.1:10001
102-
```
94+
```bash
95+
azurite
96+
```
97+
98+
The command should return results similar to:
99+
100+
```text
101+
Azurite Blob service is starting at http://127.0.0.1:10000
102+
Azurite Blob service is successfully listening at http://127.0.0.1:10000
103+
Azurite Queue service is starting at http://127.0.0.1:10001
104+
Azurite Queue service is successfully listening at http://127.0.0.1:10001
105+
```
103106

104107
#### Configuring Fluent Bit for Azurite
105108

106-
[Azurite](https://github.com/Azure/Azurite) comes with a default `account_name` and `shared_key`, so make sure to use the specific values provided in the example below \(do an exact copy/paste\):
109+
[Azurite](https://github.com/Azure/Azurite) comes with a default `account_name` and `shared_key`. Instead of the defaults, be sure to use the specific values provided in the following example:
107110

108111
```python
109112
[SERVICE]
@@ -129,7 +132,7 @@ Azurite Queue service is successfully listening at http://127.0.0.1:10001
129132
endpoint http://127.0.0.1:10000
130133
```
131134

132-
after running that Fluent Bit configuration you will see the data flowing into Azurite:
135+
After running the Fluent Bit configuration, you will see the data flowing into Azurite:
133136

134137
```text
135138
$ azurite

0 commit comments

Comments
 (0)