buffer: add description about evacuate chunk feature

buffer/README.md

@@ -77,7 +77,37 @@ Fluentd will abort the attempt to transfer the failing chunks on the following c
 
    \(default: `72h`\)
 
-In these events, **all chunks in the queue are discarded.** If you want to avoid this, you can enable `retry_forever` to make Fluentd retry indefinitely.
+Before v1.19.0 (fluent-package v6), **all chunks in the queue are discarded** in these events.
+If you want to avoid this, you can enable `retry_forever` to make Fluentd retry indefinitely or enable `secondary` output.
+
+Since v1.19.0 (fluent-package v6), the file buffer plugins ([`buf_file`](file.md) and [`buf_file_single`](file_single.md)) provide a safer and more user-friendly feature.
+They evacuate the chunk files in the queue to the following directory before clearing the queue.
+
+```text
+${root_dir}/buffer/${plugin_id}/
+```
+
+`${root_dir}` is determined by the parameter `root_dir` in `<system>`.
+If you do not configure this parameter, it will be `/tmp/fluent/`.
+
+After the issues, such as network failures, are resolved, you can recover these chunks by moving the evacuated chunk files back to the plugin's buffer directory and restarting Fluentd.
+
+{% hint style='info' %}
+If the plugin runs with multi-worker, the original buffer directory should be separated per worker. The evacuated chunk files can be restored to any worker's directory.
+{% endhint %}
+
+{% hint style='info' %}
+You can use [zero-downtime restart](../deployment/zero-downtime-restart.md) to prevent downtime of some input plugins like `in_tcp`.
+{% endhint %}
+
+{% hint style='info' %}
+For third-party buffer plugins, it is necessary to implement a function to support this feature.
+Please see [How to Write Buffer Plugin](../plugin-development/api-plugin-buffer.md) for details.
+{% endhint %}
+
+{% hint style='warning' %}
+Please note that [`buf_memory`](memory.md) does not support this feature.
+{% endhint %}
 
 ### Handling Unrecoverable Errors
 

plugin-development/api-plugin-buffer.md

@@ -2,5 +2,28 @@
 
 TODO: Write
 
+## Methods
+
+### evacuate_chunk
+
+You can override this method to add feature to evacuate chunks before clearing the queue when reaching the retry limit.
+See [Buffer - Handling Successive Failures](../buffer/README.md#handling-successive-failures) for details.
+
+```ruby
+def evacuate_chunk(chunk)
+  unless chunk.is_a?(Fluent::Plugin::Buffer::FileChunk)
+    raise ArgumentError, "The chunk must be FileChunk, but it was #{chunk.class}."
+  end
+
+  backup_dir = File.join(backup_base_dir, 'buffer', safe_owner_id)
+  FileUtils.mkdir_p(backup_dir, mode: system_config.dir_permission || Fluent::DEFAULT_DIR_PERMISSION) unless Dir.exist?(backup_dir)
+
+  FileUtils.copy([chunk.path, chunk.meta_path], backup_dir)
+  log.warn "chunk files are evacuated to #{backup_dir}.", chunk_id: dump_unique_id_hex(chunk.unique_id)
+rescue => e
+  log.error "unexpected error while evacuating chunk files.", error: e
+end
+```
+
 If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License.