Skip to content

Add jobs flag to spec on pgupgrade to determine number of pgupgrade jobs #4039

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

glynn-scott-hofbauer
Copy link

Checklist:

  • Have you added an explanation of what your changes do and why you'd like them to be included?
  • Have you updated or added documentation for the change, as applicable?
  • Have you tested your changes on all related environments with successful results, as applicable?
    • Have you added automated tests?

Type of Changes:

  • New feature
  • Bug fix
  • Documentation
  • Testing enhancement
  • Other

What is the current behavior (link to any open issues here)?

Currently, the PGUpgrade job uses the default number of parallel jobs to execute the upgrade process. However, we have encountered issues on NFS filesystems where running multiple jobs simultaneously causes filesystem conflicts, leading to upgrade failures. One such error is shown below:

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.Performing Upgrade
------------------
Setting locale and encoding for new cluster                   ok
Analyzing all rows in the new cluster                         ok
Freezing all rows in the new cluster                          ok
pg_upgrade: warning: could not remove directory "/pgdata/pg16/pg_xact": Directory not empty
Deleting files from new pg_xact                               
could not delete directory "/pgdata/pg16/pg_xact" 

To address this issue, we tested modifying the PGUpgrade job by overriding the default behavior and setting the number of jobs to 1. This adjustment resolved the filesystem conflicts, and the upgrade completed successfully.

What is the new behavior (if this is a feature change)?

  • Breaking change (fix or feature that would cause existing functionality to change)

Other Information:

@glynn-scott-hofbauer glynn-scott-hofbauer changed the title Add jobs flag to spec on pgupgrade to determine number of jobs pgupgrade Add jobs flag to spec on pgupgrade to determine number of pgupgrade jobs Nov 26, 2024
@cbandy
Copy link
Member

cbandy commented Dec 5, 2024

@glynn-scott-hofbauer thank you for this.

I took a different approach in #4043 that keeps the API the same. There, --jobs=1 for PGUpgrades with low or no CPU resources specified. Behind a new feature gate, the operator will choose a larger number of --jobs when sufficient CPU is available.

cbandy added a commit to cbandy/postgres-operator that referenced this pull request Feb 18, 2025
This also graduates the PGUpgradeCPUConcurrency feature to Beta.
Users may set PGUpgrade.spec.jobs to disable that behavior case by case.

Issue: PGO-2022
Resolves: CrunchyData#4039
cbandy added a commit to cbandy/postgres-operator that referenced this pull request Feb 18, 2025
This also graduates the PGUpgradeCPUConcurrency feature to Beta.
Users may set PGUpgrade.spec.jobs to disable that behavior case by case.

Issue: PGO-2022
Resolves: CrunchyData#4039
cbandy added a commit to cbandy/postgres-operator that referenced this pull request Feb 19, 2025
This also enables the PGUpgradeCPUConcurrency feature by default.
Users may set PGUpgrade.spec.jobs to disable that behavior case by case.

Issue: PGO-2022
Resolves: CrunchyData#4039
@cbandy cbandy closed this in #4098 Feb 21, 2025
cbandy added a commit that referenced this pull request Feb 21, 2025
This also enables the PGUpgradeCPUConcurrency feature by default.
Users may set PGUpgrade.spec.jobs to disable that behavior case by case.

Issue: PGO-2022
Resolves: #4039
benjaminjb pushed a commit that referenced this pull request Feb 28, 2025
This also enables the PGUpgradeCPUConcurrency feature by default.
Users may set PGUpgrade.spec.jobs to disable that behavior case by case.

Issue: PGO-2022
Resolves: #4039
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants