Null characters, should we sanitize? #60
Comments
visch
changed the title
|
running into this as well. think it would be nice to at least have the option too 😅 |
github-merge-queue bot
pushed a commit
that referenced
this issue
Dec 10, 2024
Null characters are currently passed as-is to Postgres despite being unsupported. If it is encountered, it causes the sink to fail as noted here: #60 with an error like `ValueError: A string literal cannot contain NUL (0x00) characters.` This PR introduces a new option called `sanitize_null_text_characters` which enables sanitization of these characters. --------- Co-authored-by: Edgar Ramírez Mondragón <[email protected]> Co-authored-by: Edgar Ramírez-Mondragón <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Right now you'll get something like
ValueError: A string literal cannot contain NUL (0x00) charactersorsqlalchemy.exc.DataError: (psycopg2.errors.UntranslatableCharacter) unsupported Unicode escape sequenceas postgres doesn't allow NUL characters see https://www.postgresql.org/docs/current/functions-string.html#:~:text=chr(0)%20is%20disallowed%20because%20text%20data%20types%20cannot%20store%20that%20character.Should we sanitize the data ie something like
data.replace("\u0000","")or leave the offending record?The text was updated successfully, but these errors were encountered: