-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Filter collections #36
Changes from 3 commits
6877925
6095152
238461c
78c6f98
f4f0e0e
d847006
d7f633d
b7e0040
f6031fb
4cde476
ea60814
62d2fc2
ae5c785
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -96,6 +96,18 @@ class TapMongoDB(Tap): | |
default="", | ||
description="An optional prefix which will be added to each stream name.", | ||
), | ||
th.Property( | ||
"filter_collections", | ||
th.OneOf( | ||
th.StringType, | ||
th.ArrayType(th.StringType), | ||
), | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since Meltano doesn't yet support setting union types and that is arguably how most of the time this tap will be used, wdyt of only accepting an array of strings for this? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't feel that strongly about the union type - this just started out as a POC for @melgazar9 who wanted to filter one collection at a time, so I thought it made sense to support that pattern as a string also. I think the Meltano setting definition needs to be There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we should remove it. It adds little benefit and complicates things for automatic There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
required=True, | ||
default=[], | ||
description=( | ||
"Collections to discover (default: all). Useful for improving catalog discovery performance." | ||
), | ||
), | ||
th.Property( | ||
"start_date", | ||
th.DateTimeType, | ||
|
@@ -210,6 +222,7 @@ def connector(self) -> MongoDBConnector: | |
self.config.get("database"), | ||
self.config.get("datetime_conversion"), | ||
prefix=self.config.get("prefix", None), | ||
collections=self.config["filter_collections"], | ||
) | ||
|
||
@property | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This would be ideal, but I get the following error:
Can't see any indication from docs that this isn't supported: