forked from raystack/compass
-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: create delete assets API #77
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
luthfifahlevi
requested review from
haveiss,
mabdh,
utsav14nov,
irainia,
anjali9791,
batrov,
sumslim and
Mayurjag
August 8, 2024 08:15
luthfifahlevi
changed the title
feat: create delete assets api
feat: create delete assets API
Aug 8, 2024
Pull Request Test Coverage Report for Build 10503449624Details
💛 - Coveralls |
mabdh
reviewed
Aug 8, 2024
internal/store/postgres/migrations/000019_alter_asset_table_refreshed_at.down.sql
Outdated
Show resolved
Hide resolved
…y and in situ worker
haveiss
reviewed
Aug 21, 2024
haveiss
reviewed
Aug 21, 2024
irainia
reviewed
Aug 21, 2024
haveiss
reviewed
Aug 21, 2024
irainia
requested changes
Aug 21, 2024
irainia
requested changes
Aug 22, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
small changes needed
irainia
approved these changes
Aug 22, 2024
haveiss
reviewed
Aug 22, 2024
haveiss
approved these changes
Aug 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background
Currently, there are two services for data discovery: Meteor and Compass. Meteor extracts assets from various sources (e.g., BigQuery tables, Optimus, Firehose, Dagger jobs) and syncs the metadata to Compass as the catalog service. However, currently, Compass has a lot of assets that no longer present and not synced. We need a housekeeping mechanism in Compass to automatically remove stale assets from the database.
Goals in this Repo:
Create delete assets API in Compass
Idea:
DeleteAssets
as DELETE/v1/assets/
: an async API that will return how many rows that will affect based on filter criteria, and delete those rows from PostgreSQL (assets and lineage) and Elasticsearch in asynchronous.Request:
query_expr
represent query expr for filter criteria: https://github.com/expr-lang/expr.dry_run
represent whether it only return how much affected rows (if set to true), or also perform deletion process in the background (default: false).Response:
affected_rows
represent how many rows that will affect based on filter criteria.Current Approach: