Delta Sharing 0.6.11

We’d like to announce the release of Delta Sharing 0.6.11, which introduces the following improvements.

Improvements:

• Improve error handling and logging in DeltaSharingSource. (#434)
• Introduce Timestamp and UUID as suffix to table path in RemoteDeltaLog (#441)

Credits: Lin Zhou

Delta Sharing 1.0.2

We’d like to announce the release of Delta Sharing 1.0.2, which introduces the following improvements.
Improvements:

• DeltaSharingClient able to set responseFormat as a list of format it could cover, and handle the response in any format in the list. (#422, #427)
• RemoteDeltaLog to be able to accept table metadata as an input parameter when being initialized. (#423)

Credits: Lin Zhou

Delta Sharing 1.0.1

We’d like to announce the release of Delta Sharing 1.0.1, which introduces the following improvements.
Improvements:

• Improve error message on unrecognized file from server, and add queryTableVersionIntervalSeconds (#402)
• Migrate parquet sharing related classes from spark/ to client/ (#405)
• Fix Tests with a copy of DeltaMetadata (#408)
• Move limit pushdown config to ConfUtils (#411)
• Fix type of files used for DeltaTableFiles (#418)

Credits: Charlene Lyu, Lin Zhou

Delta Sharing 0.7.6

We’d like to announce the release of Delta Sharing 0.7.6, which introduces the following improvements.
Improvements:

• Support refresh token to refresh the pre-signed urls correctly for queries on the latest snapshot. (#391)
• Improve error message on unrecognized file from server, and add queryTableVersionIntervalSeconds (#402)
• Fix type used for DeltaTableFiles (#417)

Credits: Charlene Lyu, Lin Zhou

Delta Sharing 0.6.10

We’d like to announce the release of Delta Sharing 0.6.10, which introduces the following improvements.
Improvements:

• Support refresh token to refresh the pre-signed urls correctly for queries on the latest snapshot. (#390)
• Improve error message on unrecognized file from server, and add queryTableVersionIntervalSeconds (#401)

Credits: Charlene Lyu, Lin Zhou

Delta Sharing 1.0.0

We are super excited to announce the release of Delta Sharing 1.0.0, which includes several exciting new features and artifacts.

Delta Format Sharing

In order to support Delta advanced features such as DeletionVectors and ColumnMapping in the Delta Sharing OSS Spark connector, “Delta format sharing” is introduced. The core Delta Sharing functionalities are grouped into the delta-sharing-client library and will be released separately. The response of Delta format sharing will not be processed by delta-sharing-client; instead,advanced Delta Spark library will be leveraged to read the data. We also upgraded the Delta Sharing server to support this. (#334, #335, #337, #338, #370, #374, #375, #376, #377, #378, #388, #389, #392, #393, #395)

JsonPredicate Enhancements (JsonPredicatesV2)

We enhanced JsonPredicates to support predicate-based filtering on all columns and not just partition columns. We also added support for the following schema types: Float, Double, Timestamp. (#349, #352, #368, #394)
We added JsonPredicate support in the pandas connector (#396)

Pagination on Delta Sharing Queries

Pagination on Delta Sharing data access APIs enables control over the number of files returned per Query Table request. This improves the scalability of the Delta Sharing service when working with tables with large file volumes, preventing timeouts and resource limit exceedances. (#352, #353, #354, #356, #360, #362, #366)

Other improvements and fixes

Support the release of Maven artifacts for both Scala 2.12 and Scala 2.13. #333
Support refresh token to refresh the pre-signed urls correctly for queries on the latest snapshot. (#385, #386, #387)
Fix bug where to_pandas doesn't properly handle the case when column names between Delta and parquet metadata are different in casing (#384)
A couple build and test fixes. (#330, #364, #369)
A couple fixes on the PROTOCOL description. #340, #365
Fix an issue in DeltaSharingSource: to double check sortedFetchedFiles contain the correct files and set previousOffset correctly. #328, #344

Credits

Abhijit Chakankar, Charlene Lyu, Kr eis, Lin Zhou, William Chau, andyl-db

Delta Sharing 0.7.5

We’d like to announce the release of Delta Sharing 0.7.5, which introduces the following improvements.
Improvements:

• Fix build errors: use a static Cast init function in all places under spark/ (#371)

Credits: Lin Zhou

Delta Sharing 0.7.4

We’d like to announce the release of Delta Sharing 0.7.4, which introduces the following improvements.
Improvements:

• Fix two build errors: use a static Cast init function and isinstance for type comparison in python (#361)

Credits: Lin Zhou

Delta Sharing 0.7.3

We’d like to announce the release of Delta Sharing 0.7.3, which introduces the following improvements.

Improvements:

• Use JsonRawValue in fasterxml.jackson instead of codehaus.jackson (#358)

Credits: Lin Zhou

Delta Sharing 0.7.2

We’d like to announce the release of Delta Sharing 0.7.2, which introduces the following improvements.

Improvements:

• Upgrade to scala 2.13 and spark 3.3.2 in branch-0.7(#355)

Credits: Lin Zhou