Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: CustomStreamMap always convert datetime field to date #2739

Closed
1 task done
lngvietthang opened this issue Nov 4, 2024 · 1 comment · Fixed by #2741
Closed
1 task done

bug: CustomStreamMap always convert datetime field to date #2739

lngvietthang opened this issue Nov 4, 2024 · 1 comment · Fixed by #2741
Assignees
Labels
kind/Bug Something isn't working Stream Maps
Milestone

Comments

@lngvietthang
Copy link

Singer SDK Version

0.41.0

Is this a regression?

  • Yes

Python Version

3.9

Bug scope

Mapping (stream maps, flattening, etc.)

Operating System

Linux

Description

In my Meltano configuration, I use meltano-map-transformer (latest version with [email protected]) with config as below:

stream_maps:
  xxx:
    creation: datetime.datetime.strptime(creation, '%Y-%m-%dT%H:%M:%S.%f%z')
...

With new update from singer-sdk v0.41, this field (creation) is mapped with date type in schema message.

class CustomStreamMap(StreamMap):
    def _eval_type(  # noqa: PLR0911
        self,
        expr: str,
        default: th.JSONTypeHelper | None = None,
    ) -> th.JSONTypeHelper:
        ...
        if expr.startswith("datetime.date") or expr.endswith(".date()"):
            return th.DateType()

        if expr.startswith("datetime.datetime"):
            return th.DateTimeType()
        ....

I think we need to swap the position of 2 if clause to check the datetime data type firstly.

Related PR:
#2665

Code

class CustomStreamMap(StreamMap):
def _eval_type( # noqa: PLR0911
self,
expr: str,
default: th.JSONTypeHelper | None = None,
) -> th.JSONTypeHelper:
...
if expr.startswith("datetime.date") or expr.endswith(".date()"):
return th.DateType()

    if expr.startswith("datetime.datetime"):
        return th.DateTimeType()
    ....

Link to Slack/Linen

No response

@edgarrmondragon
Copy link
Collaborator

I think we need to swap the position of 2 if clause to check the datetime data type firstly.

That makes sense @lngvietthang. PRs welcome!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/Bug Something isn't working Stream Maps
Projects
None yet
2 participants