MDEV-33763 : Buffered error logging for Galera

[janlindstrom]

This is adaptation of feature by Zsolt Parragi [email protected] for Percona XtraDB Cluster to MariaDB. Original feature is documented at https://www.percona.com/blog/introducing-buffered-error-logging-in-percona-server-for-mysql/.

In this adaptation variable names are changed and integration to server code is different.

Commit adds new variables
wsrep_buffered_error_log_buffer_size ulong default 0 dynamic
Size of buffer used in buffered error logging. Because
wsrep debug setting produces large number of messages
it is recomended to set this variable around 1-100Mb.

wsrep_buffered_error_log_filename string default NULL static
  Name of the file where buffered error logging is done.

wsrep_buffered_error_log_file_size ulong default 0 dynamic
  Maximum size of one buffered error log file. This variable
  should be configured to be 10-100 times bigger than buffer
  size.

wsrep_buffered_error_log_rotations uint default 9 dynamic.
  Number of buffered error log files kept before deleting
  them.

When wsrep_buffered_error_log_buffer_size is specified and non zero, a buffer is
allocated with the specified size, and log messages are
written there. This includes also wsrep debug messages.

When this buffer gets full, the server shuts down, or crashes, the
buffer is written to wsrep_buffered_error_log_filename.

When log files reaches configured size limit log is rotated and
only configured amount of log files are kept.

This is useful in situations where debug logs create lots of output and
slow down normal server operation, but are helpful for figuring out
issues, as writing to a memory buffer has less overhead.

Warning: When this feature is used server will write big number
of messages to error log. Therefore, it is recomended to store
these logs on separate location to avoid filling up your
database disk. Furthermore, it is recomended to set up
logrotate to delete old log files.

Detailed changes:

wsrep.h
Log printing is now controlled by wsrep_debug_mode variable and
controlled by wsrep_debug and wsrep_buffered_error_log_buffer_size
variables.

sql/log.cc
Added sql_print_debug to log wsrep debug messages (WSREP_DEBUG()).

print_buffer_to_file
Actual buffered error logging if enabled will happen in this
function.

sql/mysqld.cc
Buffered error logging is initialized and shutdown here if it is
configured.

sql/signal_handler.cc
If buffered error logging is enabled remaining buffer is written
before continuing signal handling.

sql/sys_vars.cc
New variables are defined here.

sql/wsrep_buffered_error_log.cc
sql/wsrep_buffered_error_log.h
Buffered error log implementation using logger API (mysys/file_logger.c).

sql/wsrep_var.cc
New variables update and check functions.

• The Jira issue number for this PR is: MDEV-______

Description

TODO: fill description here

Release Notes

TODO: What should the release notes say about this change?
Include any changed system variables, status variables or behaviour. Optionally list any https://mariadb.com/kb/ pages that need changing.

How can this PR be tested?

TODO: modify the automated test suite to verify that the PR causes MariaDB to behave as intended.
Consult the documentation on "Writing good test cases".

If the changes are not amenable to automated testing, please explain why not and carefully describe how to test manually.

Basing the PR against the correct MariaDB version

• This is a new feature or a refactoring, and the PR is based against the main branch.
• This is a bug fix, and the PR is based against the earliest maintained branch in which the bug can be reproduced.

PR quality check

• I checked the CODING_STANDARDS.md file and my PR conforms to this where appropriate.
• For any trivial modifications to the PR, I am ok with the reviewer making the changes themselves.