Skip to content

Try to fix the non-persisted rocksdb #3403

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 16, 2025

Conversation

Kailai-Wang
Copy link
Collaborator

@Kailai-Wang Kailai-Wang commented Apr 16, 2025

Context

See the linked issue for full context.

Normally we just write once in rpc handler, so set_sync is used.

Maybe there're some improvement room, like only flush it once when necessary? (e.g. program exit? or any hook from gramine)

Track of I/O

before:

2025-04-16T10:34:42.657347Z [INFO][tokio-runtime-worker][rpc_server::methods::pumpx::get_next_intent_id]: intent_id is 0
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- return from write(...) = 0x79
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- clock_gettime(0, 0x52c9230) = 0x0
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- write(2, 0x3491a10, 0x64) ...
2025-04-16T10:34:42.657384Z [INFO][tokio-runtime-worker][intent_core]: Before updating intent_id: 5
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- return from write(...) = 0x64
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- write(13, 0x5380000, 0x5a) ...
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- return from write(...) = 0x5a
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- time(0) = 0x67ff87c2
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- clock_gettime(0, 0x52c9230) = 0x0
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- write(2, 0x3491a10, 0x63) ...
2025-04-16T10:34:42.657505Z [INFO][tokio-runtime-worker][intent_core]: After updating intent_id: 5
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- return from write(...) = 0x63

with this workaround:

2025-04-16T12:57:01.178136Z [DEBUG][tokio-runtime-worker][executor_storage]: before inserting key
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- return from write(...) = 0x62
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- write(13, 0x9cda000, 0x5a) ...
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- return from write(...) = 0x5a
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- fdatasync(13) = 0x0
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- fsync(10) = 0x0
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- time(0) = 0x67ffa91d
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- clock_gettime(0, 0x9c231f0) = 0x0
(libos_parser.c:1701:buf_write_all) [P1:T2:omni-executor] trace: ---- write(2, 0x7de6210, 0x61) ...
2025-04-16T12:57:01.199020Z [DEBUG][tokio-runtime-worker][executor_storage]: after inserting key

@Kailai-Wang Kailai-Wang requested review from kziemianek, silva-fj and a team April 16, 2025 12:35
@Kailai-Wang Kailai-Wang self-assigned this Apr 16, 2025
Copy link

linear bot commented Apr 16, 2025

Copy link
Collaborator

@BillyWooo BillyWooo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So opts.set_sync(true); does the trick 👍

@Kailai-Wang
Copy link
Collaborator Author

Hold on - still need a bit fix

@Kailai-Wang Kailai-Wang enabled auto-merge (squash) April 16, 2025 13:31
@Kailai-Wang Kailai-Wang merged commit 3a0a53b into dev Apr 16, 2025
21 checks passed
@Kailai-Wang Kailai-Wang deleted the p-1469-rocksdb-corruptednot-persisted-on-oe-staging branch April 16, 2025 14:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants