Code

Add foreign table providers#921

Code

Merged

timsaucer merged 28 commits intoapache:mainapache/datafusion-python:mainfrom timsaucer:feature/ffitimsaucer/datafusion-python:feature/ffiCopy head branch name to clipboardNov 15, 2024

+4,080-91

Commits

Commits on Nov 11, 2024

• testing FFI for table provider

  timsaucer

  committed
  5cd598a

  Copy full SHA for 5cd598a

• Was able to get round trip schema from datafusion -> delta table -> datafusion

  timsaucer

  committed
  7a476d2

  Copy full SHA for 7a476d2

• Expand file structure

  timsaucer

  committed
  f830058

  Copy full SHA for f830058

• WIP on execution plan

  timsaucer

  committed
  eec590f

  Copy full SHA for eec590f

• Working through execution plan FFI

  timsaucer

  committed
  f7e7aca

  Copy full SHA for f7e7aca

• Using datafusion-proto for execution plan properties

  timsaucer

  committed
  803e6ff

  Copy full SHA for 803e6ff

• Adding plan properties parsing from ffi

  timsaucer

  committed
  9df2f86

  Copy full SHA for 9df2f86

• Standardize naming for FFI structs

  timsaucer

  committed
  73a1899

  Copy full SHA for 73a1899

• Intermediate testing and troubleshooting

  timsaucer

  committed
  0d23d55

  Copy full SHA for 0d23d55

• Adding record batch stream ffi representation

  timsaucer

  committed
  7d42f73

  Copy full SHA for 7d42f73

• Mimimum viable product demonstrating foreign table provider

  timsaucer

  committed
  405a89f

  Copy full SHA for 405a89f

• Move ffi module to datafusion core

  timsaucer

  committed
  d122393

  Copy full SHA for d122393

• Modifications need to compile against latest DF

  timsaucer

  committed
  88afcc7

  Copy full SHA for 88afcc7

• Set DF to 42.0.0

  timsaucer

  committed
  5e386e6

  Copy full SHA for 5e386e6

• Rebasing and pulling in a few changes for DF43.0

  timsaucer

  committed
  2764bca

  Copy full SHA for 2764bca

• Add wrapper for register table provider

  timsaucer

  committed
  fbf54f1

  Copy full SHA for fbf54f1

• Suppress deprecation warning

  timsaucer

  committed
  01a15f1

  Copy full SHA for 01a15f1

• Add example for FFI table provider

  timsaucer

  committed
  7f58117

  Copy full SHA for 7f58117

• Add pytest for FFI module to CI

  timsaucer

  committed
  cee25b8

  Copy full SHA for cee25b8

• Add license text

  timsaucer

  committed
  a284554

  Copy full SHA for a284554

• Change the name of the FFI table provider test so it doesn't try to run during the first pass of pytest when the module hasn't been built

  timsaucer

  committed
  5551ef9

  Copy full SHA for 5551ef9

• Build example in build stage to be used during test stage

  timsaucer

  committed
  50b4bf2

  Copy full SHA for 50b4bf2

• Combine pytests into one stage

  timsaucer

  committed
  468ef5b

  Copy full SHA for 468ef5b

• Fix path for unit test

  timsaucer

  committed
  a0a39fd

  Copy full SHA for a0a39fd

• Installing maturin for ffi test in test script

  timsaucer

  committed
  9529bea

  Copy full SHA for 9529bea

• Need to install the wheel for unit test

  timsaucer

  committed
  15458c7

  Copy full SHA for 15458c7

Commits on Nov 12, 2024

• Add online documentation about using custom table providers

  timsaucer

  committed
  64dcbd3

  Copy full SHA for 64dcbd3

• Raise an error if method is not implemented when it is expected

  timsaucer

  committed
  e31e478

  Copy full SHA for e31e478