Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Arrow casting errors during/after KNN search when vectors are float16, if a vector index is created with gpu #3478

Open
ascillitoe opened this issue Feb 26, 2025 · 3 comments
Open

Arrow casting errors during/after KNN search when vectors are float16, if a vector index is created with gpu #3478

ascillitoe opened this issue Feb 26, 2025 · 3 comments
Assignees
@BubbleCal

Comments

@ascillitoe
Copy link
Contributor

Problem

The docs suggest that vectors can be written as float16 to save disk space. However, doing this causes an error in arrow-array when I perform a KNN search with to_table(nearest=...). Strangely this only occurs if a vector index is built on gpu (see MWE below).

Is this a bug or am I doing something wrong?

Relevant dependency verisons

pylance==0.23.2 
pyarrow==17 to 19 tried
arrow-array==53.3.0 (rust crate)

MWE

Also see colab gist.

import lance
import pyarrow as pa
import numpy as np
import shutil

import lance
import pyarrow as pa
import numpy as np
import shutil

N = 10_000
INDEX = True
ACCELERATOR = None # 'cuda'

table = pa.table({
   "idx": pa.array(np.arange(N)),
   "embedding": pa.FixedShapeTensorArray.from_numpy_ndarray(
       np.random.rand(N, 128).astype(np.float16))
})
shutil.rmtree("embeddings.lance", ignore_errors=True)
dataset = lance.write_dataset(table, "embeddings.lance")
print(dataset.schema)

if INDEX:
    # Build vector index
    dataset.create_index(
        "embedding",
        #metric='dot',
        index_type="IVF_PQ",
        num_partitions=256,  # IVF
        num_sub_vectors=16,
        accelerator=ACCELERATOR,
        replace=True
    )

# Run ANN search
query = np.random.rand(128)#.astype(np.float16)
result = dataset.to_table(
    nearest={
        "column": "embedding",
        "q": query,
        "k": 10,
        "nprobes": 10,
        "refine_factor": 5,
        #"metric": "dot",
    },
    columns=["idx"],
    with_row_id=True,
    prefilter=True
    ).to_pandas()

print(result)

Result if INDEX=False or INDEX=True and ACCELERATOR = None:

    idx  _distance  _rowid
0  8926  14.312071    8926
1  5869  15.094225    5869
2  2262  15.247785    2262
3  3685  15.499866    3685
4  9258  15.622038    9258
5  8482  15.640593    8482
6  7616  15.673938    7616
7  3368  15.706622    3368
8   632  15.747130     632
9  3106  15.764575    3106

Result if INDEX=True and ACCELERATOR = 'cuda':

thread 'lance-cpu' panicked at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-array-53.3.0/src/cast.rs:758:33:
primitive array

The same problem occurs if the FixedShapeTensorArray is created as float32 and then casted to float16 as done in the docs:

table = pa.table({
   "idx": pa.array(np.arange(N)),
   "embedding": pa.FixedShapeTensorArray.from_numpy_ndarray(
       np.random.rand(N, 128).astype(np.float32))
})
dataset = lance.write_dataset(table, "embeddings.lance")
dataset.alter_columns({"path": "embedding",
                       "data_type": pa.list_(pa.float16(), 128)})
@wjones127
Copy link
Contributor

FYI @BubbleCal . I wonder if we are assuming the vectors are f32 somewhere here? Would be good to rerun this with RUST_BACKTRACE=1 so we can get the full context of where this is happening. Might need to do that on a debug build.

@ascillitoe
Copy link
Contributor Author

Sorry I should have provided that myself. The full traceback is below. It looks like the relevant bit might be this bit?

   4: lance_index::vector::pq::ProductQuantizer::dot_distances
   5: lance_index::vector::pq::ProductQuantizer::compute_distances
             at /home/ashley/Software/lance/rust/lance-index/src/vector/pq.rs:211:34
   6: <lance::index::vector::pq::PQIndex as lance_index::vector::VectorIndex>::search::{{closure}}::{{closure}}::{{closure}}
             at /home/ashley/Software/lance/rust/lance/src/index/vector/pq.rs:227:29
   7: lance_core::utils::tokio::spawn_cpu::{{closure}}
Full traceback 
thread 'lance-cpu' panicked at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-array-53.3.0/src/cast.rs:758:33:
primitive array
stack backtrace:
   0: rust_begin_unwind
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5
   1: core::panicking::panic_fmt
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14
   2: core::panicking::panic_display
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:261:5
   3: core::option::expect_failed
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/option.rs:2024:5
   4: lance_index::vector::pq::ProductQuantizer::dot_distances
   5: lance_index::vector::pq::ProductQuantizer::compute_distances
             at /home/ashley/Software/lance/rust/lance-index/src/vector/pq.rs:211:34
   6: <lance::index::vector::pq::PQIndex as lance_index::vector::VectorIndex>::search::{{closure}}::{{closure}}::{{closure}}
             at /home/ashley/Software/lance/rust/lance/src/index/vector/pq.rs:227:29
   7: lance_core::utils::tokio::spawn_cpu::{{closure}}
             at /home/ashley/Software/lance/rust/lance-core/src/utils/tokio.rs:62:22
   8: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/blocking/task.rs:42:21
   9: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/core.rs:331:17
  10: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/loom/std/unsafe_cell.rs:16:9
  11: tokio::runtime::task::core::Core<T,S>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/core.rs:320:30
  12: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:499:19
  13: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panic/unwind_safe.rs:272:9
  14: std::panicking::try::do_call
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:584:40
  15: std::panicking::try
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:547:19
  16: std::panic::catch_unwind
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panic.rs:358:14
  17: tokio::runtime::task::harness::poll_future
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:487:18
  18: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:209:27
  19: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:154:15
  20: tokio::runtime::task::raw::RawTask::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/raw.rs:201:18
  21: tokio::runtime::task::UnownedTask<S>::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/mod.rs:472:9
  22: tokio::runtime::blocking::pool::Task::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/blocking/pool.rs:161:9
  23: tokio::runtime::blocking::pool::Inner::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/blocking/pool.rs:511:17
  24: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/blocking/pool.rs:469:13
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

thread 'lance-cpu' panicked at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-array-53.3.0/src/cast.rs:758:33:
primitive array
stack backtrace:
   0: rust_begin_unwind
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5
   1: core::panicking::panic_fmt
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14
   2: core::panicking::panic_display
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:261:5
   3: core::option::expect_failed
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/option.rs:2024:5
   4: lance_index::vector::pq::ProductQuantizer::dot_distances
   5: lance_index::vector::pq::ProductQuantizer::compute_distances
             at /home/ashley/Software/lance/rust/lance-index/src/vector/pq.rs:211:34
   6: <lance::index::vector::pq::PQIndex as lance_index::vector::VectorIndex>::search::{{closure}}::{{closure}}::{{closure}}
             at /home/ashley/Software/lance/rust/lance/src/index/vector/pq.rs:227:29
   7: lance_core::utils::tokio::spawn_cpu::{{closure}}
             at /home/ashley/Software/lance/rust/lance-core/src/utils/tokio.rs:62:22
   8: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/blocking/task.rs:42:21
   9: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/core.rs:331:17
  10: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/loom/std/unsafe_cell.rs:16:9
  11: tokio::runtime::task::core::Core<T,S>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/core.rs:320:30
  12: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:499:19
  13: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panic/unwind_safe.rs:272:9
  14: std::panicking::try::do_call
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:584:40
  15: std::panicking::try
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:547:19
  16: std::panic::catch_unwind
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panic.rs:358:14
  17: tokio::runtime::task::harness::poll_future
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:487:18
  18: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:209:27
  19: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:154:15
  20: tokio::runtime::task::raw::RawTask::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/raw.rs:201:18
  21: tokio::runtime::task::UnownedTask<S>::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/mod.rs:472:9
  22: tokio::runtime::blocking::pool::Task::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/blocking/pool.rs:161:9
  23: tokio::runtime::blocking::pool::Inner::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/blocking/pool.rs:511:17
  24: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/blocking/pool.rs:469:13
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

thread 'lance_background_thread' panicked at /home/ashley/Software/lance/rust/lance-core/src/utils/tokio.rs:65:24:
called `Result::unwrap()` on an `Err` value: RecvError(())
stack backtrace:
   0: rust_begin_unwind
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5
   1: core::panicking::panic_fmt
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14
   2: core::result::unwrap_failed
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/result.rs:1704:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/result.rs:1109:23
   4: lance_core::utils::tokio::spawn_cpu::{{closure}}
             at /home/ashley/Software/lance/rust/lance-core/src/utils/tokio.rs:65:20
   5: <T as futures_util::fns::FnOnce1<A>>::call_once
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/fns.rs:15:9
   6: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/future/map.rs:57:73
   7: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86:13
   8: <lance::index::vector::pq::PQIndex as lance_index::vector::VectorIndex>::search::{{closure}}::{{closure}}
             at /home/ashley/Software/lance/rust/lance/src/index/vector/pq.rs:269:10
   9: <lance::index::vector::pq::PQIndex as lance_index::vector::VectorIndex>::search::{{closure}}
             at /home/ashley/Software/lance/rust/lance/src/index/vector/pq.rs:203:5
  10: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/future/future.rs:124:9
  11: <lance::index::vector::ivf::IVFIndex as lance_index::vector::VectorIndex>::search_in_partition::{{closure}}
             at /home/ashley/Software/lance/rust/lance/src/index/vector/ivf.rs:886:59
  12: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/future/future.rs:124:9
  13: <F as futures_core::future::TryFuture>::try_poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/future.rs:92:9
  14: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/try_future/into_future.rs:34:9
  15: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/future/map.rs:55:37
  16: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86:13
  17: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86:13
  18: <lance::io::exec::knn::ANNIvfSubIndexExec as datafusion_physical_plan::execution_plan::ExecutionPlan>::execute::{{closure}}::{{closure}}
             at /home/ashley/Software/lance/rust/lance/src/io/exec/knn.rs:652:30
  19: <futures_util::stream::futures_ordered::OrderWrapper<T> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/futures_ordered.rs:56:9
  20: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/futures_unordered/mod.rs:528:17
  21: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/mod.rs:1638:9
  22: <futures_util::stream::futures_ordered::FuturesOrdered<Fut> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/futures_ordered.rs:195:49
  23: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/mod.rs:1638:9
  24: <futures_util::stream::stream::buffered::Buffered<St> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/buffered.rs:73:42
  25: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:130:9
  26: <datafusion_physical_plan::stream::RecordBatchStreamAdapter<S> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-physical-plan-44.0.0/src/stream.rs:394:9
  27: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:130:9
  28: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/mod.rs:1638:9
  29: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/next.rs:32:9
  30: <datafusion_physical_plan::sorts::sort::SortExec as datafusion_physical_plan::execution_plan::ExecutionPlan>::execute::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-physical-plan-44.0.0/src/sorts/sort.rs:968:62
  31: <futures_util::stream::once::Once<Fut> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/once.rs:46:33
  32: <S as futures_core::stream::TryStream>::try_poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:206:9
  33: <futures_util::stream::try_stream::try_flatten::TryFlatten<St> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/try_stream/try_flatten.rs:66:44
  34: <datafusion_physical_plan::stream::RecordBatchStreamAdapter<S> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-physical-plan-44.0.0/src/stream.rs:394:9
  35: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:130:9
  36: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/mod.rs:1638:9
  37: datafusion_physical_plan::coalesce_batches::CoalesceBatchesStream::poll_next_inner
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-physical-plan-44.0.0/src/coalesce_batches.rs:294:57
  38: <datafusion_physical_plan::coalesce_batches::CoalesceBatchesStream as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-physical-plan-44.0.0/src/coalesce_batches.rs:230:20
  39: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:130:9
  40: <futures_util::stream::stream::fuse::Fuse<S> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/fuse.rs:53:27
  41: <futures_util::stream::stream::zip::Zip<St1,St2> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/zip.rs:82:19
  42: <futures_util::stream::stream::map::Map<St,F> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/map.rs:58:26
  43: <futures_util::stream::stream::fuse::Fuse<S> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/fuse.rs:53:27
  44: <futures_util::stream::stream::buffered::Buffered<St> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/buffered.rs:66:19
  45: <futures_util::stream::stream::map::Map<St,F> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/map.rs:58:26
  46: <S as futures_core::stream::TryStream>::try_poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:206:9
  47: <futures_util::stream::try_stream::try_for_each::TryForEach<St,Fut,F> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/try_stream/try_for_each.rs:60:30
  48: lance::io::exec::take::Take::new::{{closure}}
             at /home/ashley/Software/lance/rust/lance/src/io/exec/take.rs:81:22
  49: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-0.1.41/src/instrument.rs:321:9
  50: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/core.rs:331:17
  51: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/loom/std/unsafe_cell.rs:16:9
  52: tokio::runtime::task::core::Core<T,S>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/core.rs:320:30
  53: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:499:19
  54: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panic/unwind_safe.rs:272:9
  55: std::panicking::try::do_call
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:584:40
  56: std::panicking::try
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:547:19
  57: std::panic::catch_unwind
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panic.rs:358:14
  58: tokio::runtime::task::harness::poll_future
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:487:18
  59: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:209:27
  60: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:154:15
  61: tokio::runtime::task::raw::RawTask::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/raw.rs:201:18
  62: tokio::runtime::task::LocalNotified<S>::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/mod.rs:435:9
  63: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/scheduler/multi_thread/worker.rs:596:18
  64: tokio::runtime::coop::with_budget
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/coop.rs:107:5
  65: tokio::runtime::coop::budget
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/coop.rs:73:5
  66: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/scheduler/multi_thread/worker.rs:595:9
  67: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/scheduler/multi_thread/worker.rs:546:24
  68: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/scheduler/multi_thread/worker.rs:511:21
  69: tokio::runtime::context::scoped::Scoped<T>::set
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/context/scoped.rs:40:9
  70: tokio::runtime::context::set_scheduler::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/context.rs:180:26
  71: std::thread::local::LocalKey<T>::try_with
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/local.rs:308:12
  72: std::thread::local::LocalKey<T>::with
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/local.rs:272:9
  73: tokio::runtime::context::set_scheduler
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/context.rs:180:17
  74: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/scheduler/multi_thread/worker.rs:506:9
  75: tokio::runtime::context::runtime::enter_runtime
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/context/runtime.rs:65:16
  76: tokio::runtime::scheduler::multi_thread::worker::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/scheduler/multi_thread/worker.rs:498:5
  77: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/scheduler/multi_thread/worker.rs:464:45
  78: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/blocking/task.rs:42:21
  79: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/core.rs:331:17
  80: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/loom/std/unsafe_cell.rs:16:9
  81: tokio::runtime::task::core::Core<T,S>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/core.rs:320:30
  82: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:499:19
  83: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panic/unwind_safe.rs:272:9
  84: std::panicking::try::do_call
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:584:40
  85: std::panicking::try
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:547:19
  86: std::panic::catch_unwind
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panic.rs:358:14
  87: tokio::runtime::task::harness::poll_future
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:487:18
  88: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:209:27
  89: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:154:15
  90: tokio::runtime::task::raw::RawTask::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/raw.rs:201:18
  91: tokio::runtime::task::UnownedTask<S>::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/mod.rs:472:9
  92: tokio::runtime::blocking::pool::Task::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/blocking/pool.rs:161:9
  93: tokio::runtime::blocking::pool::Inner::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/blocking/pool.rs:511:17
  94: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/blocking/pool.rs:469:13
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

thread 'lance_background_thread' panicked at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/core.rs:378:22:
JoinHandle polled after completion
stack backtrace:
   0: rust_begin_unwind
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5
   1: core::panicking::panic_fmt
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14
   2: tokio::runtime::task::core::Core<T,S>::take_output::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/core.rs:378:22
   3: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/loom/std/unsafe_cell.rs:16:9
   4: tokio::runtime::task::core::Core<T,S>::take_output
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/core.rs:374:26
   5: tokio::runtime::task::harness::Harness<T,S>::try_read_output
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:282:44
   6: tokio::runtime::task::raw::RawTask::try_read_output
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/raw.rs:220:9
   7: <tokio::runtime::task::join::JoinHandle<T> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/join.rs:339:13
   8: futures_util::future::future::FutureExt::poll_unpin
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/future/mod.rs:558:9
   9: <lance::io::exec::take::Take as futures_core::stream::Stream>::poll_next
             at /home/ashley/Software/lance/rust/lance/src/io/exec/take.rs:143:29
  10: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:130:9
  11: <S as futures_core::stream::TryStream>::try_poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:206:9
  12: <futures_util::stream::try_stream::try_filter::TryFilter<St,Fut,F> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/try_stream/try_filter.rs:83:47
  13: <futures_util::stream::stream::map::Map<St,F> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/map.rs:58:26
  14: <futures_util::stream::stream::fuse::Fuse<S> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/fuse.rs:53:27
  15: <futures_util::stream::stream::buffer_unordered::BufferUnordered<St> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/buffer_unordered.rs:68:19
  16: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:130:9
  17: <datafusion_physical_plan::stream::RecordBatchStreamAdapter<S> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-physical-plan-44.0.0/src/stream.rs:394:9
  18: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:130:9
  19: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/mod.rs:1638:9
  20: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/next.rs:32:9
  21: <datafusion_physical_plan::sorts::sort::SortExec as datafusion_physical_plan::execution_plan::ExecutionPlan>::execute::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-physical-plan-44.0.0/src/sorts/sort.rs:968:62
  22: <futures_util::stream::once::Once<Fut> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/once.rs:46:33
  23: <S as futures_core::stream::TryStream>::try_poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:206:9
  24: <futures_util::stream::try_stream::try_flatten::TryFlatten<St> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/try_stream/try_flatten.rs:66:44
  25: <datafusion_physical_plan::stream::RecordBatchStreamAdapter<S> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-physical-plan-44.0.0/src/stream.rs:394:9
  26: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:130:9
  27: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/mod.rs:1638:9
  28: <datafusion_physical_plan::filter::FilterExecStream as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-physical-plan-44.0.0/src/filter.rs:490:37
  29: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:130:9
  30: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/mod.rs:1638:9
  31: datafusion_physical_plan::coalesce_batches::CoalesceBatchesStream::poll_next_inner
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-physical-plan-44.0.0/src/coalesce_batches.rs:294:57
  32: <datafusion_physical_plan::coalesce_batches::CoalesceBatchesStream as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-physical-plan-44.0.0/src/coalesce_batches.rs:230:20
  33: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:130:9
  34: <futures_util::stream::stream::fuse::Fuse<S> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/fuse.rs:53:27
  35: <futures_util::stream::stream::zip::Zip<St1,St2> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/zip.rs:82:19
  36: <futures_util::stream::stream::map::Map<St,F> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/map.rs:58:26
  37: <futures_util::stream::stream::fuse::Fuse<S> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/fuse.rs:53:27
  38: <futures_util::stream::stream::buffered::Buffered<St> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/buffered.rs:66:19
  39: <futures_util::stream::stream::map::Map<St,F> as futures_core::stream::Stream>::poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/map.rs:58:26
  40: <S as futures_core::stream::TryStream>::try_poll_next
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:206:9
  41: <futures_util::stream::try_stream::try_for_each::TryForEach<St,Fut,F> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/try_stream/try_for_each.rs:60:30
  42: lance::io::exec::take::Take::new::{{closure}}
             at /home/ashley/Software/lance/rust/lance/src/io/exec/take.rs:81:22
  43: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-0.1.41/src/instrument.rs:321:9
  44: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/core.rs:331:17
  45: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/loom/std/unsafe_cell.rs:16:9
  46: tokio::runtime::task::core::Core<T,S>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/core.rs:320:30
  47: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:499:19
  48: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panic/unwind_safe.rs:272:9
  49: std::panicking::try::do_call
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:584:40
  50: std::panicking::try
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:547:19
  51: std::panic::catch_unwind
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panic.rs:358:14
  52: tokio::runtime::task::harness::poll_future
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:487:18
  53: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:209:27
  54: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:154:15
  55: tokio::runtime::task::raw::RawTask::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/raw.rs:201:18
  56: tokio::runtime::task::LocalNotified<S>::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/mod.rs:435:9
  57: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/scheduler/multi_thread/worker.rs:659:22
  58: tokio::runtime::coop::with_budget
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/coop.rs:107:5
  59: tokio::runtime::coop::budget
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/coop.rs:73:5
  60: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/scheduler/multi_thread/worker.rs:595:9
  61: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/scheduler/multi_thread/worker.rs:546:24
  62: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/scheduler/multi_thread/worker.rs:511:21
  63: tokio::runtime::context::scoped::Scoped<T>::set
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/context/scoped.rs:40:9
  64: tokio::runtime::context::set_scheduler::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/context.rs:180:26
  65: std::thread::local::LocalKey<T>::try_with
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/local.rs:308:12
  66: std::thread::local::LocalKey<T>::with
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/local.rs:272:9
  67: tokio::runtime::context::set_scheduler
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/context.rs:180:17
  68: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/scheduler/multi_thread/worker.rs:506:9
  69: tokio::runtime::context::runtime::enter_runtime
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/context/runtime.rs:65:16
  70: tokio::runtime::scheduler::multi_thread::worker::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/scheduler/multi_thread/worker.rs:498:5
  71: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/scheduler/multi_thread/worker.rs:464:45
  72: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/blocking/task.rs:42:21
  73: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/core.rs:331:17
  74: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/loom/std/unsafe_cell.rs:16:9
  75: tokio::runtime::task::core::Core<T,S>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/core.rs:320:30
  76: tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:499:19
  77: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panic/unwind_safe.rs:272:9
  78: std::panicking::try::do_call
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:584:40
  79: std::panicking::try
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:547:19
  80: std::panic::catch_unwind
             at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panic.rs:358:14
  81: tokio::runtime::task::harness::poll_future
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:487:18
  82: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:209:27
  83: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/harness.rs:154:15
  84: tokio::runtime::task::raw::RawTask::poll
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/raw.rs:201:18
  85: tokio::runtime::task::UnownedTask<S>::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/task/mod.rs:472:9
  86: tokio::runtime::blocking::pool::Task::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/blocking/pool.rs:161:9
  87: tokio::runtime::blocking::pool::Inner::run
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/blocking/pool.rs:511:17
  88: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /home/ashley/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.42.0/src/runtime/blocking/pool.rs:469:13
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Traceback (most recent call last):
  File "/home/ashley/Software/lance/temp_test/basic_selection_id_test.py", line 54, in <module>
    result = dataset.to_table(
  File "/home/ashley/Software/lance/python/python/lance/dataset.py", line 648, in to_table
    ).to_table()
  File "/home/ashley/Software/lance/python/python/lance/dataset.py", line 3327, in to_table
    return self.to_reader().read_all()
  File "pyarrow/ipc.pxi", line 762, in pyarrow.lib.RecordBatchReader.read_all
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: Io error: Execution error: ExecNode(Take): thread panicked: task 567 panicked with message "JoinHandle polled after completion"

@wjones127
Copy link
Contributor

Thanks! that traceback helps narrow it down.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@BubbleCal BubbleCal

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@wjones127 @ascillitoe @BubbleCal