-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix compatibility with Pandas 2 #63
base: main
Are you sure you want to change the base?
Conversation
The `line_terminator` parameter of `read_csv` was changed to `lineterminator` in Pandas 1.5, and the old name was dropped entirely in Pandas 2.
@@ -69,7 +71,8 @@ def write(self, path): | |||
|
|||
with open(path, 'w') as f: | |||
f.write(f'"sep={self.delimiter}"\n') | |||
write_df.to_csv(f, index=False, sep=self.delimiter, line_terminator=f"{self.delimiter}\n") | |||
write_df.to_csv(f, index=False, sep=self.delimiter, | |||
**{_to_csv_line_terminator: f"{self.delimiter}\n"}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm. This raises a ValueError with Python 3.13.
________________________ ShimmerPlusTestCase.test_write ________________________
self = <test_shimmer.ShimmerPlusTestCase testMethod=test_write>
def test_write(self):
> self.reader.write(self.WRITE_PATH)
tests/test_shimmer.py:82:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../BUILDROOT/usr/lib/python3.13/site-packages/devicely/shimmer_plus.py:74: in write
write_df.to_csv(f, index=False, sep=self.delimiter,
/usr/lib64/python3.13/site-packages/pandas/util/_decorators.py:333: in wrapper
return func(*args, **kwargs)
/usr/lib64/python3.13/site-packages/pandas/core/generic.py:3964: in to_csv
return DataFrameRenderer(formatter).to_csv(
/usr/lib64/python3.13/site-packages/pandas/io/formats/format.py:1014: in to_csv
csv_formatter.save()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <pandas.io.formats.csvs.CSVFormatter object at 0x7f2269a70a50>
def save(self) -> None:
"""
Create the writer & save.
"""
# apply compression and byte/text conversion
with get_handle(
self.filepath_or_buffer,
self.mode,
encoding=self.encoding,
errors=self.errors,
compression=self.compression,
storage_options=self.storage_options,
) as handles:
# Note: self.encoding is irrelevant here
> self.writer = csvlib.writer(
handles.handle,
lineterminator=self.lineterminator,
delimiter=self.sep,
quoting=self.quoting,
doublequote=self.doublequote,
escapechar=self.escapechar,
quotechar=self.quotechar,
)
E ValueError: bad delimiter or lineterminator value
/usr/lib64/python3.13/site-packages/pandas/io/formats/csvs.py:260: ValueError
So far, I haven't got a clue as to why?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That seems unrelated to this change. The existing call (slightly) abuses lineterminator
to always insert a trailing separator. This must have been made more strict in Python 3.13, and I see a few changelog entries that could be related, most likely "gh-113796: Add more validation checks in the csv.Dialect constructor. ValueError is now raised if the same character is used in different roles."
The
line_terminator
parameter ofread_csv
was changed tolineterminator
in Pandas 1.5, and the old name was dropped entirely in Pandas 2.This adds compatibility shims across both, though if you prefer to bump minimum Pandas version to 1.5, then the new name could be used everywhere instead.