Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v24.3.x] storage: always schedule adjacent segment compaction #24956

Open
wants to merge 1 commit into
base: v24.3.x
Choose a base branch
from
Open

[v24.3.x] storage: always schedule adjacent segment compaction #24956

wants to merge 1 commit into from

Conversation

vbotbuildovich
Copy link
Collaborator

Backport of PR #24874

@andrwng @vbotbuildovich
storage: always schedule adjacent segment compaction
48f35d7 
We previously fell back on adjacent segment compaction only if there was
no new data to compact. In some situations, we've seen the rate of
incoming data outpace the compaction interval, causing segments to pile
up without ever being merged.

This change tweaks the logic to always run adjacent segment compaction
after running sliding window compaction.

Along the way, a couple tests needed to be tweaked to handle the fact
that housekeeping now may merge segments.

(cherry picked from commit 08d0433)
@vbotbuildovich vbotbuildovich added this to the v24.3.x-next milestone Jan 28, 2025
@vbotbuildovich vbotbuildovich added the kind/backport PRs targeting a stable branch label Jan 28, 2025
@vbotbuildovich
Copy link
Collaborator Author

CI test results

test results on build#61273
test_id test_kind job_url test_status passed
rptest.tests.archive_retention_test.CloudArchiveRetentionTest.test_delete.cloud_storage_type=CloudStorageType.ABS.retention_type=retention.bytes ducktape https://buildkite.com/redpanda/redpanda/builds/61273#0194ab6b-eef6-4e28-a83a-8724bd481ce4 FLAKY 1/2
rptest.tests.consumer_group_balancing_test.ConsumerGroupBalancingTest.test_coordinator_nodes_balance ducktape https://buildkite.com/redpanda/redpanda/builds/61273#0194ab6b-eef6-4e28-a83a-8724bd481ce4 FLAKY 1/2
rptest.tests.consumer_group_balancing_test.ConsumerGroupBalancingTest.test_coordinator_nodes_balance ducktape https://buildkite.com/redpanda/redpanda/builds/61273#0194ab70-5a68-4034-8c28-bb3fd6635e09 FLAKY 1/2
rptest.transactions.producers_api_test.ProducersAdminAPITest.test_producers_state_api_during_load ducktape https://buildkite.com/redpanda/redpanda/builds/61273#0194ab6b-eef6-4e28-a83a-8724bd481ce4 FLAKY 1/2
storage_e2e_single_thread_rpunit.storage_e2e_single_thread_rpunit unit https://buildkite.com/redpanda/redpanda/builds/61273#0194ab29-bd26-4c20-8cf1-1d821aec3bec FAIL 0/2
storage_e2e_single_thread_rpunit.storage_e2e_single_thread_rpunit unit https://buildkite.com/redpanda/redpanda/builds/61273#0194ab29-bd27-4317-b345-53ef1c52c303 FAIL 0/2
storage_single_thread_rpunit.storage_single_thread_rpunit unit https://buildkite.com/redpanda/redpanda/builds/61273#0194ab29-bd26-4c20-8cf1-1d821aec3bec FLAKY 1/2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrwng andrwng Awaiting requested review from andrwng

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
area/redpanda kind/backport PRs targeting a stable branch
Projects
None yet
Milestone
v24.3.x-next
Development

Successfully merging this pull request may close these issues.
2 participants
@vbotbuildovich @andrwng