[BUG]The cudf to_csv interface cannot read files larger than 2GB and displays a negative size error.

[Ploverain]

Describe the bug
The cudf to_csv interface cannot write files larger than 2GB and displays a negative size error.

Steps/Code to reproduce bug
import cudf

df = cudf.read_csv("3G.csv")
df.to_csv("result.csv")

Expected behavior
i hope df.to_csv() create a 3G size csv

Environment overview (please complete the following information)

• Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)]
• Method of cuDF install: [conda, Docker, or from source]
  • If method of install is [Docker], provide docker pull & docker run commands used

Environment details
Please run and paste the output of the cudf/print_env.sh script here, to gather any other relevant environment details

Additional context
Add any other context about the problem here.

[GregoryKimball]

Thank you @Ploverain for sharing this request. It sounds like the CSV writer is limited by our strings column character limit of 2.1B characters (also see #13733). At a minimum we should provide a better error message that recommends partitioning the dataframe.

[beckernick]

You can use the chunksize parameter to get around this issue. E.g.,

df = cudf.read_csv("3G.csv")
df.to_csv("result.csv", chunksize=5e6) # assuming five million rows will work -- you may want to try a higher or lower value

We could also explore handling this under the hood in the Python layer (via some kind of data introspection or otherwise) (cc @wence- , as this came up in a recent discussion)

[davidwendt]

If you have enough GPU memory, this should work now in 24.08. Fixed in #16148
Regardless, I would still recommend using chunksize parameter as mentioned in Nick's comment above.