Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test an FST bytes store that re-reverses (reads bytes forward) on-the-fly #12992

Open
mikemccand opened this issue Jan 4, 2024 · 0 comments
Open

Comments

@mikemccand
Copy link
Member

Description

At read-time the FST apis must read bytes in reverse, which is perverse and unnatural for all stacks in modern CPUs / IO devices that do read-ahead optimizations for forward reading.

It's quite complex to change the underlying FST format to become fundamentally forward only. It'd require rewriting node addresses, which may then take different numbers of vInt bytes, causing more renumbering, etc.

A simpler first step might be, at FST freeze() time (when the FST is done being compiled), reverse all bytes in the underlying storage (on disk or on heap), and at read time, pretend to the caller that they are still reading backwards, yet actually read forwards. We could even do this separately for each store, e.g. start by testing on-heap read-time double reversal, or maybe start with on-disk where the OS's readahead optimizations may matter more.

It should be relatively trivial to implement yet hard to think about, and we could see if it helps performance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant