Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RELEASE] cudf v25.02 #17893

Merged
merged 338 commits into from
Feb 13, 2025
Merged

[RELEASE] cudf v25.02 #17893

merged 338 commits into from
Feb 13, 2025

Conversation

AyodeAwe
Copy link
Contributor

❄️ Code freeze for branch-25.02 and v25.02 release

What does this mean?

Only critical/hotfix level issues should be merged into branch-25.02 until release (merging of this PR).

What is the purpose of this PR?

  • Update documentation
  • Allow testing for the new release
  • Enable a means to merge branch-25.02 into main for the release

Matt711 and others added 30 commits December 6, 2024 12:38
@Matt711
Plumb pylibcudf.io.parquet options classes through cudf python (#17506)
169a45a 
Apart of #15162

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Matthew Roeschke (https://github.com/mroeschke)

URL: #17506
@davidwendt
Improve strings contains/find performance for smaller strings (#17330)
38261f8 
Replaces usage of `cudf::string_view::find()` with loop and call to `cudf::string_view::compare()` where possible.
This showed significant performance improvement. 
This was also slightly faster than a KMP prototype implementation.
Also updates the find/contains benchmarks to remove the 2GB limit and include column versions of the find APIs.

Authors:
  - David Wendt (https://github.com/davidwendt)

Approvers:
  - Basit Ayantunde (https://github.com/lamarrr)
  - Bradley Dice (https://github.com/bdice)

URL: #17330
@mroeschke
Remove cudf._lib.text in favor of inlining pylibcudf (#17408)
c791f80 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Matthew Murray (https://github.com/Matt711)

URL: #17408
@Matt711
Replaces uses of cudf._lib.Column.from_unique_ptr with `pylibcudf.C…
467cf7a 
…olumn.from_libcudf` (#17517)

Apart of #15162. In a follow-up PR we'll deprecate the cudf python column APIs and others that are used outside cudf.

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Matthew Roeschke (https://github.com/mroeschke)

URL: #17517
@mroeschke
Remove cudf._lib.round in favor of inlining pylibcudf (#17430)
1a62b46 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Matthew Murray (https://github.com/Matt711)

URL: #17430
@mroeschke
Remove cudf._lib.orc in favor of inlining pylibcudf (#17466)
b6f7e6e 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)

URL: #17466
@Matt711
Add Parquet Reader options classes to pylibcudf (#17464)
cbeefd8 
Follow up of #17263, this PR adds the parquet reader options classes to pylibcudf and plumbs the changes through cudf python.

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Matthew Roeschke (https://github.com/mroeschke)
  - Yunsong Wang (https://github.com/PointKernel)
  - Nghia Truong (https://github.com/ttnghia)
  - MithunR (https://github.com/mythrocks)

URL: #17464
@karthikeyann
Fix all null list column with missing child column in JSON reader (#1…
14b4321 
…7348)

Authors:
  - Karthikeyan (https://github.com/karthikeyann)

Approvers:
  - Nghia Truong (https://github.com/ttnghia)
  - Basit Ayantunde (https://github.com/lamarrr)
  - David Wendt (https://github.com/davidwendt)

URL: #17348
@bdice
Update cuda-python lower bounds to 12.6.2 / 11.8.5 (#17547)
80fc629 
We require a newer cuda-python lower bound for new features and to use the new layout.
This will fix a number of errors observed when the runtime version of cuda-python is older than the version used to build packages using Cython features from cuda-python.

See rapidsai/build-planning#117 (comment) for details.

Authors:
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - James Lamb (https://github.com/jameslamb)

URL: #17547
@davidwendt
Use cooperative-groups instead of cub warp-reduce for strings contains (
a0fc6a8 
#17540)

Replaces the `cub::WarpReduce` usage in `cudf::strings::contains` with cooperative-groups `any()`.
The change is only for the `contains_warp_parallel` kernel which is used for wider strings.
Using cooperative-groups generates more efficient code for the same results and gives an additional 11-14% performance improvement.

Authors:
  - David Wendt (https://github.com/davidwendt)

Approvers:
  - Yunsong Wang (https://github.com/PointKernel)
  - Nghia Truong (https://github.com/ttnghia)
  - Shruti Shivakumar (https://github.com/shrshi)

URL: #17540
@Matt711
Remove unused IO utilities from cudf python (#17374)
0f5d4b9 
Removes unused IO utilities from cuDF Python. Depends on #17163 #16042 #17252 #17263

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #17374
@vuule
Fix nvcc-imposed UB in constexpr functions (#17534)
ba3ed57 
nvcc does not support `constexpr` functions that are not well-defined to call from the device. This is UB even when the function is not called from the device.

Throwing an exception is one such operation. This PR cleans up error handling for functions that are called from device, and removes `constexpr` from the ones that are not actually used from the device, or in the constexpr context.

Authors:
  - Vukasin Milovanovic (https://github.com/vuule)
  - MithunR (https://github.com/mythrocks)

Approvers:
  - Karthikeyan (https://github.com/karthikeyann)
  - MithunR (https://github.com/mythrocks)
  - David Wendt (https://github.com/davidwendt)
  - Vyas Ramasubramani (https://github.com/vyasr)
  - Bradley Dice (https://github.com/bdice)
  - Mike Wilson (https://github.com/hyperbolic2346)
  - Yunsong Wang (https://github.com/PointKernel)
  - Muhammad Haseeb (https://github.com/mhaseeb123)

URL: #17534
@davidwendt
Document undefined behavior in div_rounding_up_safe (#17542)
ed2892c 
Adds more description to the `div_rounding_up_safe` utility identifying undefined behavior.
Closes #17539

Authors:
  - David Wendt (https://github.com/davidwendt)

Approvers:
  - Paul Mattione (https://github.com/pmattione-nvidia)
  - Lawrence Mitchell (https://github.com/wence-)
  - Nghia Truong (https://github.com/ttnghia)

URL: #17542
@abellina
[JNI] Enables fabric handles for CUDA async memory pools (#17526)
a79077c 
This PR adds a `CUDA_ASYNC_FABRIC` allocation mode in `RmmAllocationMode` and pipes in the options to RMM's `cuda_async_memory_resource` of a `fabric` for the handle type, and `read_write` as the memory protection mode (as that's the only mode supported by the pools, and is required for IPC).

If `CUDA_ASYNC` is used, fabric handles are not requested, and the memory protection is `none`.

Authors:
  - Alessandro Bellina (https://github.com/abellina)

Approvers:
  - Nghia Truong (https://github.com/ttnghia)
  - Jason Lowe (https://github.com/jlowe)

URL: #17526
@mroeschke
Remove cudf._lib.string.convert/split in favor of inlining pylibcudf (#…
f595592 
…17496)

Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Matthew Murray (https://github.com/Matt711)

URL: #17496
@abellina
[JNI] remove rmm argument to set rw access for fabric handles (#17553)
5b412dc 
This is a follow up from #17526, where fabric handles can be enabled from RMM. That PR also sets the memory access protection flag (`cudaMemPoolSetAccess`), but I have learned that this second flag is not needed from the owner device. In fact, it causes confusion because the owning device fails to call this function with some of the flags (access none).  `cudaMemPoolSetAccess` is meant to only be called from peer processes that have imported the pool's handle. In our case, UCX handles this from the peer's side and it does not need to be anywhere in RMM or cuDF.

Sorry for the noise. I'd like to get this fix in, and then I am going to fix RMM by removing that API.

Authors:
  - Alessandro Bellina (https://github.com/abellina)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - Nghia Truong (https://github.com/ttnghia)
  - Jason Lowe (https://github.com/jlowe)

URL: #17553
@mroeschke
Remove cudf._lib.transform in favor of inlining pylibcudf (#17505)
9df95d1 
Contributes to #17317

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Matthew Murray (https://github.com/Matt711)

URL: #17505
@Matt711
Remove unused BufferArrayFromVector (#17549)
ebad043 
Follow up to #17506. This PR removes an unused buffer class.

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Matthew Roeschke (https://github.com/mroeschke)

URL: #17549
@a-hirota
Enable rounding for Decimal32 and Decimal64 in cuDF (#17332)
4764395 
Authors:
  - Hirota Akio (https://github.com/a-hirota)
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Matthew Roeschke (https://github.com/mroeschke)

URL: #17332
@mroeschke
Fix CMake format in cudf/_lib/CMakeLists.txt (#17559)
c53ace8 
Due to 2 of my cudf._lib refactoring PRs going in which then impacted formatting of `cudf/_lib/CMakeLists.txt`

Authors:
  - Matthew Roeschke (https://github.com/mroeschke)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #17559
@bdice
Remove Thrust patch in favor of CMake definition for Thrust 32-bit of…
e16b3a3 
…fset types. (#17527)

Follow-up for #17523 to use `target_compile_definitions` and drop the Thrust patch.

Authors:
  - Bradley Dice (https://github.com/bdice)
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17527
@msarahan
gate telemetry dispatch calls on TELEMETRY_ENABLED env var (#17551)
13e983e 
Because of the switch away from certificates/mTLS, we are having to rework a few things. In the meantime, telemetry jobs are failing. This PR adds a switch to turn all of the telemetry stuff off - to skip it instead. It is meant to be controlled by an org-wide environment variable, which can be applied to individual repos by ops. At the time of submitting this PR, the environment variable is 'false' and no telemetry is being reported.

Authors:
  - Mike Sarahan (https://github.com/msarahan)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #17551
@davidwendt
Replace cudf::detail::copy_if logic with thrust::copy_if and gather (#…
3468e92 
…17520)

Replaces the custom kernels for `cudf::detail::copy_if` with a call to `thrust::copy_if` to build indices to call `cudf::detail::gather`.
This is easier to maintain and faster for some cases but slower in others.

Authors:
  - David Wendt (https://github.com/davidwendt)

Approvers:
  - Yunsong Wang (https://github.com/PointKernel)
  - Nghia Truong (https://github.com/ttnghia)

URL: #17520
@vyasr
Use rapids-logger to generate the cudf logger (#17307)
5306eca 
This PR replaces cudf's logger implementation with one generated using https://github.com/rapidsai/rapids-logger. This approach allows us to centralize the logger definition across different RAPIDS projects while allowing each project to vendor its own copy with a suitable set of macros and default logger objects. The common logger also takes care of handling the more complex packaging problems around ensuring that we fully isolate our spdlog dependency and do not leak any of its symbols, allowing our libraries to be safely installed in a much broader set of environments.

Contributes to rapidsai/build-planning#104.

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Nghia Truong (https://github.com/ttnghia)
  - James Lamb (https://github.com/jameslamb)
  - Bradley Dice (https://github.com/bdice)

URL: #17307
@PointKernel
Fix typos, rename types, and add null_probability benchmark axis for …
657f50b 
…distinct (#17546)

This PR addresses several minor issues discovered while working on #17467:

- Corrected a typo where `RowHasher` should have been `RowEqual`
- Renamed `hash_set_type` to `distinct_set_t`
- Added a `null_probability` benchmark axis for the distinct benchmark, similar to other stream compaction benchmarks

Authors:
  - Yunsong Wang (https://github.com/PointKernel)

Approvers:
  - Muhammad Haseeb (https://github.com/mhaseeb123)
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17546
@Matt711
Merge branch-24.12 into branch-25.02
f904a7f
@AyodeAwe
Update version references in workflow (#17568)
be62ea6 
Update version references in breaking-change trigger workflow
@rjzamora
Fix Dask-cuDF clip APIs (#17509)
1e95864 
Closes #17502

**Background Info**: The cudf and pandas `axis` defaults are different, and the upstream dask-expr `clip` APIs are consistent with the behavior of Pandas (not cudf).

Authors:
  - Richard (Rick) Zamora (https://github.com/rjzamora)
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

URL: #17509
@davidwendt
Rework minhash APIs for deprecation cycle (#17421)
0c5bd66 
Renames `minhash_permuted()` to `minhash()` and deprecates `minhash_permuted`
Also removes the `word_minhash` APIs deprecated in 24.12.

Authors:
  - David Wendt (https://github.com/davidwendt)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - Matthew Murray (https://github.com/Matt711)
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17421
@jlowe @ttnghia
@vyasr @bdice
Specify a version for rapids_logger dependency (#17573)
cd3a79b 
## Description
#17307 broke builds that use the rapids-cmake pinned dependencies
feature since no version was specified for the rapids_logger dependency.
This adds a version string equal to the git tag so the dependency has a
stated version.

## Checklist
- [X] I am familiar with the [Contributing
Guidelines](https://github.com/rapidsai/cudf/blob/HEAD/CONTRIBUTING.md).
- [ ] New or existing tests cover these changes.
- [X] The documentation is up to date with these changes.

---------

Co-authored-by: Nghia Truong <[email protected]>
Co-authored-by: Vyas Ramasubramani <[email protected]>
Co-authored-by: Bradley Dice <[email protected]>
@AyodeAwe AyodeAwe requested a review from msarahan January 31, 2025 21:39
@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented Jan 31, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@review-notebook-app Review Notebook App
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@github-actions github-actions bot added libcudf Affects libcudf (C++/CUDA) code. Python Affects Python cuDF API. CMake CMake build issue Java Affects Java cuDF API. cudf.pandas Issues specific to cudf.pandas cudf.polars Issues specific to cudf.polars pylibcudf Issues specific to the pylibcudf package labels Jan 31, 2025
@galipremsagar
Fix third-party cudf.pandas tests (#17900)
f1c2f2a 
## Description
This PR fixes cudf ci nightly test failures:
https://github.com/rapidsai/cudf/actions/runs/13097249137/job/36541039646

## Checklist
- [x] I am familiar with the [Contributing
Guidelines](https://github.com/rapidsai/cudf/blob/HEAD/CONTRIBUTING.md).
- [x] New or existing tests cover these changes.
- [x] The documentation is up to date with these changes.
galipremsagar and others added 5 commits February 3, 2025 21:36
@galipremsagar
Fix numpy data access by making attribute private (#17890)
d4bda07 
`data` attribute of numpy should be marked private as it actually points to the underlying memory and it will be distinct for a cupy array.

Authors:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17890
@shrshi
Require batches to be non-empty in multi-batch JSON reader (#17837)
8b89ea0 
Fixes #17836

Authors:
  - Shruti Shivakumar (https://github.com/shrshi)

Approvers:
  - Vukasin Milovanovic (https://github.com/vuule)
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17837
@Matt711
Fix incorrect example in pylibcudf docs (#17912)
2bada0d 
Fixes incorrect pylibcudf/libcudf example created in #17803.

Authors:
  - Matthew Murray (https://github.com/Matt711)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17912
@vyasr
Add missing pin (#17915)
0136e24 
Currently pylibcudf does not export a dependency on libcudf at all, which is incorrect.

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)
  - Bradley Dice (https://github.com/bdice)
  - James Lamb (https://github.com/jameslamb)

URL: #17915
@Matt711
Remove pandas backend from cudf.pandas - ibis integration tests (#1…
94ac29e 
…7945)

ibis has removed their pandas backend in version 10.0.0. In their
release notes,

> pandas: The pandas backend is removed. Note that pandas DataFrames are
STILL VALID INPUTS AND OUTPUTS and will remain so for the foreseeable
future. Please use one of the other local backends like DuckDB, Polars,
or DataFusion to perform operations directly on pandas DataFrames.

This PR removes the pandas backend from the integration tests. And
asserts that the inputs and outputs to ibis APIs are proxy objects.
Matt711 and others added 6 commits February 10, 2025 16:49
@Matt711
Pin ibis version in the cudf.pandas integration tests <10.0.0 (#17975)
d1a5558 
Follows up #17972. This PR is intended to get 25.02 nightly CI passing,
which has been failing for few days.
@vuule
Fix the index type in the indexing operator of the span types (#17971)
9ae04a7 
Closes #17949
Closes #17960

Derived span classes use `size_type` for the index type in their
`operator[]` implementations. The intent was to use `base::size_type`,
but the type actually resolves to `cudf::size_type`, which is `int32_t`,
and does not allow access past `int32_t::max`.

This PR fixes the type used by explicitly using `typename
base::size_type`. Also added static_asserts to make sure the type has
the right size for element indexing.
@PointKernel
Fix race check failures in shared memory groupby (#17985)
4b2ce98 
Replace #17976

This fixes the race check failures in shared memory groupby and resolves
NVIDIA/spark-rapids#11835.
@AyodeAwe
Update Changelog [skip ci]
c86ff6e
@raydouglass
Merge branch-24.12 into branch-25.02 [skip ci]
a4e1df8
@AyodeAwe
Merge pull request #18002 from raydouglass/branch-25.02-merge-24.12
12176f3 
Fix forward merge 24.12->25.02
@AyodeAwe AyodeAwe merged commit 1dff899 into main Feb 13, 2025
29 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ttnghia ttnghia ttnghia approved these changes

@msarahan msarahan Awaiting requested review from msarahan msarahan is a code owner automatically assigned from rapidsai/packaging-codeowners

@mroeschke mroeschke Awaiting requested review from mroeschke mroeschke is a code owner automatically assigned from rapidsai/cudf-python-codeowners

@galipremsagar galipremsagar Awaiting requested review from galipremsagar galipremsagar is a code owner automatically assigned from rapidsai/cudf-python-codeowners

@kingcrimsontianyu kingcrimsontianyu Awaiting requested review from kingcrimsontianyu kingcrimsontianyu is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

@davidwendt davidwendt Awaiting requested review from davidwendt davidwendt is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

Assignees

@raydouglass raydouglass

@AyodeAwe AyodeAwe

Labels
CMake CMake build issue cudf.pandas Issues specific to cudf.pandas cudf.polars Issues specific to cudf.polars Java Affects Java cuDF API. libcudf Affects libcudf (C++/CUDA) code. pylibcudf Issues specific to the pylibcudf package Python Affects Python cuDF API.
Projects
cuDF Python
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.