Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(airflow): fixed issue with _is_memory_dataset #1011

Merged
merged 25 commits into from
Feb 21, 2025
Merged

fix(airflow): fixed issue with _is_memory_dataset #1011

merged 25 commits into from
Feb 21, 2025

Conversation

CF-FHB-X
Copy link
Contributor

Description

Fixed the issue described in #998

Development notes

Tweaked the code in grouping/_is_memory_dataset() to also check for datasets that are set as MemoryDataset in the Data Catalog.

Checklist

  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Updated jsonschema/kedro-catalog-X.XX.json if necessary
  • Added a description of this change in the relevant RELEASE.md file
  • Added tests to cover my changes
  • Received approvals from at least half of the TSC (required for adding a new, non-experimental dataset)

@CF-FHB-X CF-FHB-X force-pushed the fix/998-is-memory-dataset branch from 96b5075 to 9bb91ed Compare February 17, 2025 13:44
@CF-FHB-X CF-FHB-X force-pushed the fix/998-is-memory-dataset branch from 98874b0 to e68f09c Compare February 18, 2025 12:54
ravi-kumar-pilla and others added 23 commits February 18, 2025 10:15
@ravi-kumar-pilla @CF-FHB-X
build(datasets): Release 6.0.0 (kedro-org#968)
32fdf69 
release draft

Signed-off-by: Richard Asselin <[email protected]>
@ravi-kumar-pilla @CF-FHB-X
chore(datasets): Remove tracking datasets which are used in Kedro Viz…
ef05d23 
… Experiment Tracking (kedro-org#969)

* remove et related kedro datasets

* update release note and static json schema

* temporary doc fix

Signed-off-by: Richard Asselin <[email protected]>
@ankatiyar @CF-FHB-X
docs(datasets): Move to linkcode extension (kedro-org#985)
778811c 
Move to linkcode extension

Signed-off-by: Ankita Katiyar <[email protected]>
Signed-off-by: Richard Asselin <[email protected]>
@ravi-kumar-pilla @CF-FHB-X
fix(datasets): Fix polars.CSVDataset save on Windows (kedro-org#979)
5a6b100 
* test csv win

Signed-off-by: ravi_kumar_pilla <[email protected]>

* change ci yaml for testing

Signed-off-by: ravi_kumar_pilla <[email protected]>

* change ci yaml for testing

Signed-off-by: ravi_kumar_pilla <[email protected]>

* add default encoding when opening file

* revert workflow tests

Signed-off-by: ravi_kumar_pilla <[email protected]>

* fix lint

Signed-off-by: ravi_kumar_pilla <[email protected]>

* update release note

* update release note

---------

Signed-off-by: ravi_kumar_pilla <[email protected]>
Signed-off-by: Richard Asselin <[email protected]>
@ElenaKhaustova @CF-FHB-X
feat(all): Replace trufflehog with detect-secrets (kedro-org#983)
377cf97 
* Removed trufflehog

Signed-off-by: Elena Khaustova <[email protected]>

* Updated github actions per plugin

Signed-off-by: Elena Khaustova <[email protected]>

* Updated release notes

Signed-off-by: Elena Khaustova <[email protected]>

* Updated validate-pr check scopes

Signed-off-by: Elena Khaustova <[email protected]>

* Updated lint command

Signed-off-by: Elena Khaustova <[email protected]>

* Added key to trigger check

Signed-off-by: Elena Khaustova <[email protected]>

* Updated GH action to track per plugin

Signed-off-by: Elena Khaustova <[email protected]>

* Removed secret

Signed-off-by: Elena Khaustova <[email protected]>

* Updated GH for kedro-datasets

Signed-off-by: Elena Khaustova <[email protected]>

* Updated secrets baseline

Signed-off-by: Elena Khaustova <[email protected]>

---------

Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Richard Asselin <[email protected]>
@deepyaman @astrojuanlu @CF-FHB-X
build(datasets): use intersphinx over type_targets (kedro-org#801)
7ed49f9 
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>
Signed-off-by: Richard Asselin <[email protected]>
@ElenaKhaustova @CF-FHB-X
fix(datasets): Add parameter to enable/disable lazy saving for `Parti…
fb9a099 
…tionedDataset` (kedro-org#978)

* Replaced callable check

Signed-off-by: Elena Khaustova <[email protected]>

* Updateds lazy_save test

Signed-off-by: Elena Khaustova <[email protected]>

* Added test_callable_save

Signed-off-by: Elena Khaustova <[email protected]>

* Fixed lint

Signed-off-by: Elena Khaustova <[email protected]>

* Fixed docs links

Signed-off-by: Elena Khaustova <[email protected]>

* Fixed all docs links

Signed-off-by: Elena Khaustova <[email protected]>

* Updated release notes

Signed-off-by: Elena Khaustova <[email protected]>

* Fixed all docs links

Signed-off-by: Elena Khaustova <[email protected]>

* Fixed typo

Signed-off-by: Elena Khaustova <[email protected]>

* Added argument to disable lazy saving

Signed-off-by: Elena Khaustova <[email protected]>

* Removed save function argument

Signed-off-by: Elena Khaustova <[email protected]>

* Updated unit test

Signed-off-by: Elena Khaustova <[email protected]>

* Fixed lint

Signed-off-by: Elena Khaustova <[email protected]>

* Updated related docs

Signed-off-by: Elena Khaustova <[email protected]>

* Revert test changes

Signed-off-by: Elena Khaustova <[email protected]>

* Updated baseline

Signed-off-by: Elena Khaustova <[email protected]>

* Updated release notes

Signed-off-by: Elena Khaustova <[email protected]>

* Updated release notes

Signed-off-by: Elena Khaustova <[email protected]>

* Updated docstrings

Signed-off-by: Elena Khaustova <[email protected]>

---------

Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Richard Asselin <[email protected]>
@deepyaman @CF-FHB-X
fix(datasets): use kwarg for Ibis read_* methods (kedro-org#1005)
4a15b80 
* fix(datasets): use kwarg for Ibis `read_*` methods

Signed-off-by: Deepyaman Datta <[email protected]>

* Update RELEASE.md

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Richard Asselin <[email protected]>
@deepyaman @merelcht @CF-FHB-X
build(datasets): pin PyArrow until 19.0.1 is out (kedro-org#1006)
e6b22c3 
* build(datasets): pin PyArrow until `19.0.1` is out

Signed-off-by: Deepyaman Datta <[email protected]>

* chore(datasets): exclude `19.0.0` instead of bound

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>
Co-authored-by: Merel Theisen <[email protected]>
Signed-off-by: Richard Asselin <[email protected]>
@deepyaman @CF-FHB-X
build(datasets): update list of extras for Ibis 10 (kedro-org#1003)
0fed73c 
* build(datasets): update list of extras for Ibis 10

Signed-off-by: Deepyaman Datta <[email protected]>

* Update RELEASE.md

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Richard Asselin <[email protected]>
@deepyaman @merelcht @CF-FHB-X
chore: remove internal devtools from release notes (kedro-org#1004)
a652129 
* chore: remove internal devtools from release notes

Signed-off-by: Deepyaman Datta <[email protected]>

* chore: remove internal devtools from release notes

Signed-off-by: Deepyaman Datta <[email protected]>

* chore: remove internal devtools from release notes

Signed-off-by: Deepyaman Datta <[email protected]>

* chore: remove internal devtools from release notes

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>
Co-authored-by: Merel Theisen <[email protected]>
Signed-off-by: Richard Asselin <[email protected]>
@CF-FHB-X
998: Fixed case where MemoryDatasets in catalog wouldn't trigger `_is…
c384f0c 
…_memory_dataset`

Signed-off-by: Richard Asselin <[email protected]>
@CF-FHB-X
998: Tests to ensure that MemoryDatasets are passed in mocked data ca…
b9005e6 
…talog

Signed-off-by: Richard Asselin <[email protected]>
@CF-FHB-X
998: Changelog
9fe874a 
Signed-off-by: Richard Asselin <[email protected]>
@CF-FHB-X
998: Linting fixes
4dd6619 
Signed-off-by: Richard Asselin <[email protected]>
@deepyaman @CF-FHB-X
build(datasets): update list of extras for Ibis 10 (kedro-org#1003)
15a0e3c 
* build(datasets): update list of extras for Ibis 10

Signed-off-by: Deepyaman Datta <[email protected]>

* Update RELEASE.md

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Richard <[email protected]>
Signed-off-by: Richard Asselin <[email protected]>
@deepyaman @merelcht @CF-FHB-X
chore: remove internal devtools from release notes (kedro-org#1004)
6060e67 
* chore: remove internal devtools from release notes

Signed-off-by: Deepyaman Datta <[email protected]>

* chore: remove internal devtools from release notes

Signed-off-by: Deepyaman Datta <[email protected]>

* chore: remove internal devtools from release notes

Signed-off-by: Deepyaman Datta <[email protected]>

* chore: remove internal devtools from release notes

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>
Co-authored-by: Merel Theisen <[email protected]>
Signed-off-by: Richard <[email protected]>
Signed-off-by: Richard Asselin <[email protected]>
@CF-FHB-X
998: Tests to ensure that MemoryDatasets are passed in mocked data ca…
e49f25d 
…talog

Signed-off-by: Richard <[email protected]>
Signed-off-by: Richard Asselin <[email protected]>
@CF-FHB-X
998: Linting fixes
cffbaa2 
Signed-off-by: Richard <[email protected]>
Signed-off-by: Richard Asselin <[email protected]>
@CF-FHB-X
998: Changed function according to PR comments
b07b8cb 
Signed-off-by: Richard <[email protected]>
Signed-off-by: Richard Asselin <[email protected]>
@CF-FHB-X
998: Tests to ensure that MemoryDatasets are passed in mocked data ca…
2ec88b0 
…talog

Signed-off-by: Richard Asselin <[email protected]>
@CF-FHB-X
998: Linting fixes
69aad57 
Signed-off-by: Richard Asselin <[email protected]>
Signed-off-by: Richard <[email protected]>
Signed-off-by: Richard Asselin <[email protected]>
@CF-FHB-X
998: Tweaked release
fbb8081 
Signed-off-by: Richard Asselin <[email protected]>
@CF-FHB-X CF-FHB-X force-pushed the fix/998-is-memory-dataset branch from 9136a92 to fbb8081 Compare February 18, 2025 15:22
ElenaKhaustova
ElenaKhaustova approved these changes Feb 21, 2025
View reviewed changes
Copy link
Contributor

@ElenaKhaustova ElenaKhaustova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a nit comment, the rest looks good to me, thank you!

@CF-FHB-X @ElenaKhaustova
Update RELEASE.md
8eb9beb 
Co-authored-by: ElenaKhaustova <[email protected]>
Signed-off-by: Richard <[email protected]>
ankatiyar
ankatiyar approved these changes Feb 21, 2025
View reviewed changes
Copy link
Contributor

@ankatiyar ankatiyar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks @CF-FHB-X

@ElenaKhaustova ElenaKhaustova merged commit 4458d2c into kedro-org:main Feb 21, 2025
18 checks passed
@CF-FHB-X CF-FHB-X deleted the fix/998-is-memory-dataset branch February 21, 2025 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ankatiyar ankatiyar ankatiyar approved these changes

@ElenaKhaustova ElenaKhaustova ElenaKhaustova approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@CF-FHB-X @ankatiyar @ElenaKhaustova @ravi-kumar-pilla @deepyaman