Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove decimal32/64 to decimal128 conversion in Parquet writer #17869

Merged
merged 12 commits into from
Feb 12, 2025
Merged

Remove decimal32/64 to decimal128 conversion in Parquet writer #17869

merged 12 commits into from
Feb 12, 2025

Conversation

mhaseeb123
Copy link
Member

@mhaseeb123 mhaseeb123 commented Jan 30, 2025 •
edited
Loading

Description

Fixes #17080. Related to #17422

This PR removes the decimal32/64 to decimal128 conversion in Parquet writer as it's no longer needed with Arrow v19.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented Jan 30, 2025

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions bot added the libcudf Affects libcudf (C++/CUDA) code. label Jan 30, 2025
@mhaseeb123 mhaseeb123 added 2 - In Progress Currently a work in progress cuIO cuIO issue improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Jan 30, 2025
@mhaseeb123 mhaseeb123 linked an issue Jan 30, 2025 that may be closed by this pull request
[FEA] Do not convert decimal32/64 cols to decimal128 in to_arrow API and PQ writer when arrow schema is in use #17080
Closed
@mhaseeb123 mhaseeb123 changed the title 🚧 Remove decimal32/64 to decimal128 conversion in Parquet writer Remove decimal32/64 to decimal128 conversion in Parquet writer Feb 1, 2025
mhaseeb123
mhaseeb123 commented Feb 1, 2025
View reviewed changes
cpp/src/interop/decimal_conversion_utilities.cu
@@ -1,70 +0,0 @@
/*
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed

@github-actions github-actions bot added the CMake CMake build issue label Feb 4, 2025
@mhaseeb123 mhaseeb123 marked this pull request as ready for review February 4, 2025 02:16
@mhaseeb123 mhaseeb123 requested review from a team as code owners February 4, 2025 02:16
@mhaseeb123 mhaseeb123 added 3 - Ready for Review Ready for review by team and removed 2 - In Progress Currently a work in progress labels Feb 4, 2025
@mhaseeb123 mhaseeb123 requested a review from vuule February 6, 2025 23:25
@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented Feb 6, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@mhaseeb123
Copy link
Member Author

/ok to test

@mhaseeb123 mhaseeb123 added 4 - Needs Review Waiting for reviewer to review or respond and removed 3 - Ready for Review Ready for review by team labels Feb 6, 2025
@nvdbaranec
Copy link
Contributor

To confirm, this only affects the writer, changes things so that:

  • previously 32 and 64 bit decimal were being promoted to 128 bit decimal when writing to arrow.
  • now they just write to 32 or 64 bit decimal since arrow supports it.

Is that correct?

vyasr
vyasr approved these changes Feb 12, 2025
View reviewed changes
@mhaseeb123
Copy link
Member Author

/merge

@rapids-bot rapids-bot bot merged commit aab7edb into rapidsai:branch-25.04 Feb 12, 2025
110 checks passed
@mhaseeb123 mhaseeb123 deleted the fea/remove-decimal-conversion-pq-writer branch February 12, 2025 22:10
@mhaseeb123
Copy link
Member Author

mhaseeb123 commented Feb 13, 2025 •
edited
Loading

To confirm, this only affects the writer, changes things so that:

  • previously 32 and 64 bit decimal were being promoted to 128 bit decimal when writing to arrow.
  • now they just write to 32 or 64 bit decimal since arrow supports it.

Is that correct?

Yes exactly 🎯 . Just to elaborate, the promotion to decimal128 was done when we also wanted to write the arrow_schema as well for better interop with arrow which is no longer needed as arrow now support 32/64 bit decimals

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vyasr vyasr vyasr approved these changes

@vuule vuule vuule approved these changes

@nvdbaranec nvdbaranec Awaiting requested review from nvdbaranec nvdbaranec is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

Assignees

@mhaseeb123 mhaseeb123

Labels
4 - Needs Review Waiting for reviewer to review or respond CMake CMake build issue cuIO cuIO issue improvement Improvement / enhancement to an existing function libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA] Do not convert decimal32/64 cols to decimal128 in to_arrow API and PQ writer when arrow schema is in use
4 participants
@mhaseeb123 @nvdbaranec @vyasr @vuule