Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WT-14050 Store live restore extents in the metadata file. #11550

Merged
merged 42 commits into from
Feb 7, 2025
Merged
Changes from 1 commit
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
30f028c
Initial PoC for storing extents in the metadata
luke-pearson Jan 27, 2025
1bcf626
Use clib functions for string parsing, handle null termination case.
luke-pearson Jan 27, 2025
100af45
Revert "Use clib functions for string parsing, handle null terminatio…
luke-pearson Jan 27, 2025
b98daba
Null terminate string
luke-pearson Jan 27, 2025
df4f2c2
use strtol
luke-pearson Jan 28, 2025
3924fd2
Merge branch 'develop' into wt-14050-import-export
luke-pearson Jan 28, 2025
573fc37
Remove lseek
luke-pearson Jan 28, 2025
e76374d
Extent string compression
luke-pearson Jan 28, 2025
25394a9
Fix s_aff
luke-pearson Jan 28, 2025
b6f9707
Use __wt_strndup
luke-pearson Jan 28, 2025
ff375b4
Free the exten str
luke-pearson Jan 28, 2025
d103d71
Add windows support
luke-pearson Jan 28, 2025
132de02
Revert "Add windows support"
luke-pearson Jan 28, 2025
bbd3ab7
Fix windows compile
luke-pearson Jan 28, 2025
1a6269f
Fix windows again?
luke-pearson Jan 28, 2025
9fdd35d
Size_fmt and clang analyzer
luke-pearson Jan 28, 2025
a93546b
Merge branch 'develop' into wt-14050-import-export
luke-pearson Jan 28, 2025
b347b54
Bring back reopen
luke-pearson Jan 28, 2025
dc9f75f
Fix crash when opening non file: uris
luke-pearson Jan 29, 2025
a582467
Various fixes, adding a new test
luke-pearson Jan 29, 2025
631fefc
Additional fixes
luke-pearson Jan 29, 2025
8d7e8eb
Update the unit tests for this functionality shift
luke-pearson Jan 30, 2025
06af77f
S_string -r and removing an accidentally added assert
luke-pearson Jan 30, 2025
f7e0617
Fix extent stringify algo
luke-pearson Jan 30, 2025
0c02dc2
Remove assertion and fix order of arguments
luke-pearson Jan 30, 2025
2126db3
Reword some comments about correctness
luke-pearson Jan 31, 2025
49b8368
Fix weird mechanics around directories causing all sorts of strangeness
luke-pearson Jan 31, 2025
4b58531
Initial review fixes and making the string parsing logic more paranoid
luke-pearson Jan 31, 2025
33dcfc5
Close out the file handles
luke-pearson Jan 31, 2025
e651cfb
Rephrase comment and run s_string -r
luke-pearson Jan 31, 2025
992970f
Easier review fixes
luke-pearson Feb 3, 2025
6a3910f
Review fixes and cast
luke-pearson Feb 3, 2025
b279c89
use set equality to have a more complete check
luke-pearson Feb 3, 2025
8bc1a12
Making andrew a happy camper
luke-pearson Feb 3, 2025
37e8e56
Turtle set can exist if the test runs fast
luke-pearson Feb 4, 2025
6396b53
Add live restore config to the metadata
luke-pearson Feb 4, 2025
a8b5519
Merge branch 'develop' into wt-14050-import-export
luke-pearson Feb 4, 2025
b604639
Revert "Add live restore config to the metadata"
luke-pearson Feb 4, 2025
9433d4d
Merge branch 'develop' into wt-14050-import-export
luke-pearson Feb 4, 2025
41b2183
Skip test_live_restore04
luke-pearson Feb 5, 2025
0ab342b
adding a fixme
luke-pearson Feb 5, 2025
af744dd
Merge branch 'develop' into wt-14050-import-export
luke-pearson Feb 5, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
use strtol
luke-pearson committed Jan 28, 2025
commit df4f2c2d347b028252be1ec438c6c94ded490c75
46 changes: 12 additions & 34 deletions src/live_restore/live_restore_fs.c
Original file line number Diff line number Diff line change
@@ -157,7 +157,7 @@ __wt_live_restore_fh_extent_to_metadata_string(
WT_RET(
__wt_buf_catfmt(session, extent_string, "%" PRId64 "-%" PRIu64, head->off, head->len));
if (head->next != NULL)
WT_RET(__wt_buf_catfmt(session, extent_string, "_"));
WT_RET(__wt_buf_catfmt(session, extent_string, ";"));
head = head->next;
}
__wt_verbose_info(session, WT_VERB_LIVE_RESTORE,
@@ -1239,43 +1239,21 @@ __wt_live_restore_import_extents_from_string(
WTI_LIVE_RESTORE_FILE_HANDLE *lr_fh = (WTI_LIVE_RESTORE_FILE_HANDLE *)fh;
__wt_verbose_info(session, WT_VERB_LIVE_RESTORE,
"Got live restore extents from metadata!! %s", ckpt_string);
/* The extents are separated by _. And have the shape %d-%u. */
char *offset = ckpt_string;
char *int_start = offset;
/* The extents are separated by ;. And have the shape %d-%u. */
wt_off_t off;
size_t len;
WTI_LIVE_RESTORE_HOLE_NODE **current = &lr_fh->destination.hole_list_head;
while (true) {
if (*offset >= '0' && *offset <= '9') {
offset++;
continue;
}
if (*offset == '-') {
/*
* Okay I don't like mutating the ckpt string but this is lazy because atoi doesn't
* take a length. Lets just ignore the fact that atoi returns an _int_.
*/
*offset = '\0';
off = (wt_off_t)atoi(int_start);
offset++;
int_start = offset;
continue;
}
if (*offset == '_' || *offset == '\0') {
*current = NULL;
*offset = '\0';
len = (size_t)atoi(int_start);
offset++;
int_start = offset;
__wt_verbose_info(session, WT_VERB_LIVE_RESTORE,
char *str_ptr = ckpt_string;
do {
off = (wt_off_t)strtol(str_ptr, &str_ptr, 10);
str_ptr++;
len = (size_t)strtol(str_ptr, &str_ptr, 10);
str_ptr++;
__wt_verbose_info(session, WT_VERB_LIVE_RESTORE,
"Adding an extent: %" PRId64 "-%" PRIu64, off, len);
WT_ERR(__live_restore_alloc_extent(session, off, len, NULL, current));
current = &((*current)->next);
if (*offset == '\0')
break;
continue;
}
}
WT_ERR(__live_restore_alloc_extent(session, off, len, NULL, current));
current = &((*current)->next);
} while (*str_ptr != '\0');
WT_ERR(__live_restore_handle_verify_hole_list(
session, (WTI_LIVE_RESTORE_FS *)S2C(session)->file_system, lr_fh, fh->name));
}