Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bitnami/mysql] Feat mysqldump master database #54038

Merged
merged 15 commits into from
Jan 16, 2024
Merged

[bitnami/mysql] Feat mysqldump master database #54038

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
b679327
add dump to mysql 8
michel-silva Dec 22, 2023
6744063
add dump to mysql 8.2
michel-silva Dec 22, 2023
b8636bc
add exemplo on docker compose
michel-silva Dec 22, 2023
6fd722a
add doc
michel-silva Dec 22, 2023
12bef92
Remove blank spaces
michel-silva Dec 22, 2023
55dd33c
check database is set
michel-silva Dec 22, 2023
8ff11a4
Merge branch 'bitnami:main' into feat-mysqldump-master-database
michel-silva Dec 22, 2023
db3c069
Use DB_DATA_DIR
michel-silva Dec 26, 2023
2f0bafb
use lower case
michel-silva Dec 26, 2023
2d13676
update notes
michel-silva Dec 26, 2023
ccb352a
update comment
michel-silva Dec 26, 2023
8a5f108
fix if statement
michel-silva Dec 26, 2023
06a88af
update info
michel-silva Dec 26, 2023
bc53c91
remove head
michel-silva Dec 27, 2023
6b49f3e
update info error message
michel-silva Dec 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions bitnami/mysql/8.0/debian-11/rootfs/opt/bitnami/scripts/libmysql.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,68 @@ pid_file=${DB_PID_FILE}
EOF
}

########################
# Make a dump on master database and update slave database
# Globals:
# DB_*
# Arguments:
# None
# Returns:
# None
#########################
mysql_exec_initial_dump() {
DATA_DIR="/bitnami/mysql/data"
michel-silva marked this conversation as resolved.
Show resolved Hide resolved
info "MySQL dump master data start..."

mysql -h "$DB_MASTER_HOST" -P "$DB_MASTER_PORT_NUMBER" -u "$DB_MASTER_ROOT_USER" -p"$DB_MASTER_ROOT_PASSWORD" -e 'RESET MASTER;'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have one last question about this. This would execute RESET MASTER for every slave. Is this necessary?
Note that RESET MASTER is already executed in mysql_initialize() . Could you please confirm?

Copy link
Contributor Author

@michel-silva michel-silva Jan 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @dgomezleon.
Sorry about my delay response.. I was on vacation and I returned in the last days.
Your question is very interesting and makes me think about how replication works. So let me explain my point of view.

I have a master setup and a slave was broken because of a crash on the slave server.. So, I make a reinstallation of the new slave server and when I up the MySQL slave server this error occurs and this error is because the Mysql master server is missing some binary files. To fix it, I could make a manual mysql dump on master and update the slave, but to do it I must login in recovery mode on MySQL slave because the mysql.user could not be transferred by binary files and we do not have any valid user to login. Analyzing this scenario, I make this approach to make an auto mysql dump on master and import to a slave database.

Analyzing your question about the mysql_initialize(), this is only exec on the first up of the container. In my scenario, the reset will not be executed and the mysql master binary files are not modified. To resolve this, I could manually get mysql dump on the master server, make the reset master and run the mysql dump in slave.

So, when the MySQL master server doesn't have all the binary files, for each slave that is added to the server the reset master will be necessary for the slave to start replicating at the point of the master server stop.

One important point that your question made me think about is: that is very important that all slave servers have the last data present in the master database, when we restart replication after reset master the slave has the same data and does not miss any data. So, I think is important to add one alert to make sure that all slaves have all data updated, before start the setup the new slave.

I don't know if it explains my point of view. But I'm open to having some discussion about this and making the better to the package.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes sense to me. Thanks for the detailed explanation.


databases=("mysql")

if [ -n "$DB_DATABASE" ]; then
databases+=("$DB_DATABASE")
fi

for DB in "${databases[@]}"; do
info "Start dump process database $DB"

if [[ $DB = "information_schema" ]] || [[ $DB = "performance_schema" ]] || [[ $DB = "sys" ]]; then
michel-silva marked this conversation as resolved.
Show resolved Hide resolved
info "The table $DB not be imported, skiping"
michel-silva marked this conversation as resolved.
Show resolved Hide resolved
continue
fi

FILE_LOCATION="$DATA_DIR/dump_$DB.sql"
michel-silva marked this conversation as resolved.
Show resolved Hide resolved

mysqldump --verbose -h "$DB_MASTER_HOST" -P "$DB_MASTER_PORT_NUMBER" -u "$DB_MASTER_ROOT_USER" -p"$DB_MASTER_ROOT_PASSWORD" $DB > $FILE_LOCATION

info "Finish dump database $DB"

if [ $? -eq 0 ]; then
head -n 15 $FILE_LOCATION

michel-silva marked this conversation as resolved.
Show resolved Hide resolved
info "Ensure database exists $DB"

mysql -u "$DB_MASTER_ROOT_USER" <<EOF
CREATE DATABASE IF NOT EXISTS $DB;
michel-silva marked this conversation as resolved.
Show resolved Hide resolved
EOF

info "Start import dump database $DB"
mysql_execute $DB < $FILE_LOCATION
info "Finish import dump database $DB"

else
info "Fail create dump"
michel-silva marked this conversation as resolved.
Show resolved Hide resolved
fi

info "Remove dump file"
rm -f $FILE_LOCATION

info "Finish dump process database $DB"

done

info "MySQL dump master data finish..."
}

########################
# Migrate old custom configuration files
# Globals:
Expand All @@ -194,6 +256,11 @@ mysql_configure_replication() {
while ! echo "select 1" | mysql_remote_execute "$DB_MASTER_HOST" "$DB_MASTER_PORT_NUMBER" "mysql" "$DB_MASTER_ROOT_USER" "$DB_MASTER_ROOT_PASSWORD"; do
sleep 1
done

if [[ "$DB_REPLICATION_SLAVE_DUMP" = "true" ]]; then
mysql_exec_initial_dump
fi

debug "Replication master ready!"
debug "Setting the master configuration"
mysql_execute "mysql" <<EOF
Expand Down
2 changes: 2 additions & 0 deletions bitnami/mysql/8.0/debian-11/rootfs/opt/bitnami/scripts/mysql-env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,8 @@ export MYSQL_REPLICATION_PASSWORD="${MYSQL_REPLICATION_PASSWORD:-}"
export DB_REPLICATION_PASSWORD="$MYSQL_REPLICATION_PASSWORD" # only used during the first initialization

# Settings
export MYSQL_REPLICATION_SLAVE_DUMP="${MYSQL_REPLICATION_SLAVE_DUMP:-false}"
export DB_REPLICATION_SLAVE_DUMP="$MYSQL_REPLICATION_SLAVE_DUMP"
export MYSQL_PORT_NUMBER="${MYSQL_PORT_NUMBER:-}"
export DB_PORT_NUMBER="$MYSQL_PORT_NUMBER"
export MYSQL_REPLICATION_MODE="${MYSQL_REPLICATION_MODE:-}"
Expand Down
67 changes: 67 additions & 0 deletions bitnami/mysql/8.2/debian-11/rootfs/opt/bitnami/scripts/libmysql.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,68 @@ pid_file=${DB_PID_FILE}
EOF
}

########################
# Make a dump on master database and update slave database
# Globals:
# DB_*
# Arguments:
# None
# Returns:
# None
#########################
mysql_exec_initial_dump() {
DATA_DIR="/bitnami/mysql/data"
info "MySQL dump master data start..."

mysql -h "$DB_MASTER_HOST" -P "$DB_MASTER_PORT_NUMBER" -u "$DB_MASTER_ROOT_USER" -p"$DB_MASTER_ROOT_PASSWORD" -e 'RESET MASTER;'

databases=("mysql")

if [ -n "$DB_DATABASE" ]; then
databases+=("$DB_DATABASE")
fi

for DB in "${databases[@]}"; do
info "Start dump process database $DB"

if [[ $DB = "information_schema" ]] || [[ $DB = "performance_schema" ]] || [[ $DB = "sys" ]]; then
info "The table $DB not be imported, skiping"
continue
fi

FILE_LOCATION="$DATA_DIR/dump_$DB.sql"

mysqldump --verbose -h "$DB_MASTER_HOST" -P "$DB_MASTER_PORT_NUMBER" -u "$DB_MASTER_ROOT_USER" -p"$DB_MASTER_ROOT_PASSWORD" $DB > $FILE_LOCATION

info "Finish dump database $DB"

if [ $? -eq 0 ]; then
head -n 15 $FILE_LOCATION

info "Ensure database exists $DB"

mysql -u "$DB_MASTER_ROOT_USER" <<EOF
CREATE DATABASE IF NOT EXISTS $DB;
EOF

info "Start import dump database $DB"
mysql_execute $DB < $FILE_LOCATION
info "Finish import dump database $DB"

else
info "Fail create dump"
fi

info "Remove dump file"
rm -f $FILE_LOCATION

info "Finish dump process database $DB"

done

info "MySQL dump master data finish..."
}

########################
# Migrate old custom configuration files
# Globals:
Expand All @@ -194,6 +256,11 @@ mysql_configure_replication() {
while ! echo "select 1" | mysql_remote_execute "$DB_MASTER_HOST" "$DB_MASTER_PORT_NUMBER" "mysql" "$DB_MASTER_ROOT_USER" "$DB_MASTER_ROOT_PASSWORD"; do
sleep 1
done

if [[ "$DB_REPLICATION_SLAVE_DUMP" = "true" ]]; then
mysql_exec_initial_dump
fi

debug "Replication master ready!"
debug "Setting the master configuration"
mysql_execute "mysql" <<EOF
Expand Down
2 changes: 2 additions & 0 deletions bitnami/mysql/8.2/debian-11/rootfs/opt/bitnami/scripts/mysql-env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,8 @@ export MYSQL_REPLICATION_PASSWORD="${MYSQL_REPLICATION_PASSWORD:-}"
export DB_REPLICATION_PASSWORD="$MYSQL_REPLICATION_PASSWORD" # only used during the first initialization

# Settings
export MYSQL_REPLICATION_SLAVE_DUMP="${MYSQL_REPLICATION_SLAVE_DUMP:-false}"
export DB_REPLICATION_SLAVE_DUMP="$MYSQL_REPLICATION_SLAVE_DUMP"
export MYSQL_PORT_NUMBER="${MYSQL_PORT_NUMBER:-}"
export DB_PORT_NUMBER="$MYSQL_PORT_NUMBER"
export MYSQL_REPLICATION_MODE="${MYSQL_REPLICATION_MODE:-}"
Expand Down
5 changes: 5 additions & 0 deletions bitnami/mysql/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -389,6 +389,11 @@ The above command scales up the number of slaves to `3`. You can scale down in t

> **Note**: You should not scale up/down the number of master nodes. Always have only one master node running.

If your master database missing some binary files, the replication will broken. It's possible add `MYSQL_REPLICATION_SLAVE_DUMP=true` toe make a dump on master and import it on slave.
michel-silva marked this conversation as resolved.
Show resolved Hide resolved

> **Note**: It's most important that master database is only used by this process until the end to avoid missing data.
michel-silva marked this conversation as resolved.
Show resolved Hide resolved
> **Note**: This process will use "RESET MASTER;" on master database, for security make a backup of binary files.
michel-silva marked this conversation as resolved.
Show resolved Hide resolved

### Configuration file

The image looks for user-defined configurations in `/opt/bitnami/mysql/conf/my_custom.cnf`. Create a file named `my_custom.cnf` and mount it at `/opt/bitnami/mysql/conf/my_custom.cnf`.
Expand Down
2 changes: 2 additions & 0 deletions bitnami/mysql/docker-compose-replication.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ services:
- MYSQL_MASTER_ROOT_PASSWORD=my_root_password
# ALLOW_EMPTY_PASSWORD is recommended only for development.
- ALLOW_EMPTY_PASSWORD=yes
# Only use true if master node missing binary files (The binary files will be reset, recomendate backup files)
michel-silva marked this conversation as resolved.
Show resolved Hide resolved
- MYSQL_REPLICATION_SLAVE_DUMP=false
healthcheck:
test: ['CMD', '/opt/bitnami/scripts/mysql/healthcheck.sh']
interval: 15s
Expand Down
Loading