You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, if lag is applied to incremental, non-UTC datatime will be transformed to UTC datatime. If the initial_value is not a UTC datetime, lagged_last_value may be wrong in the string comparison (max ormin).
More importantly, if the cursor field value is a non-UTC datatime, it will compare to a UTC lagged_last_value, which is start_value and last_value in IncrementalTransform.
Cause lag may modify the last_value in IncrementalTransform, it is not safe to write back state if rows are empty.
datatime.strftime(below Python 3.12.0) can not preserve the colon in the timezone when converting datetime objects back to strings. This may also cause an unexpected behavior:
printf(max("2024-10-20T15:30:00-00:00", "2024-10-20T15:30:00-0030"))
# 2024-10-20T15:30:00-00:00 will be printed
I have added tests that can reproduce all the above cases, Let me know if there is anything to improve in the code.
Steps to reproduce
The tests can cover all the cases.
Operating system
macOS
Runtime environment
Local
Python version
3.11
The text was updated successfully, but these errors were encountered:
dlt version
1.5.0
Describe the problem
Currently, if
lag
is applied toincremental
, non-UTC datatime will be transformed to UTC datatime. If theinitial_value
is not a UTC datetime,lagged_last_value
may be wrong in the string comparison (max
ormin
).More importantly, if the cursor field value is a non-UTC datatime, it will compare to a UTC
lagged_last_value
, which isstart_value
andlast_value
inIncrementalTransform
.Cause lag may modify the last_value in
IncrementalTransform
, it is not safe to write back state if rows are empty.datatime.strftime
(below Python 3.12.0) can not preserve the colon in the timezone when converting datetime objects back to strings. This may also cause an unexpected behavior:I have added tests that can reproduce all the above cases, Let me know if there is anything to improve in the code.
Steps to reproduce
The tests can cover all the cases.
Operating system
macOS
Runtime environment
Local
Python version
3.11
The text was updated successfully, but these errors were encountered: