feat: support yaml dataset definitions #914

SiQube · 2025-01-01T20:34:31Z

💥 Breaking Changes

DatasetLibrary.get() now returns an instantiated DatasetDefinition object instead of a DatasetDefinition class.

Description

Supports specifying a DatasetDefinition as a yaml file.

Example file:

name: YourDataset

has_files:
  gaze: true
  precomputed_events: false
  precomputed_reading_measures: false

mirrors:
  gaze:
    - https://osf.io/download/

resources:
  gaze:
    - resource: your_resource/
      filename: target_filename.zip

for more information, see https://pre-commit.ci

codecov · 2025-01-01T20:35:41Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (a84d433) to head (52073b1).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #914   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           78        79    +1     
  Lines         3564      3588   +24     
  Branches       629       645   +16     
=========================================
+ Hits          3564      3588   +24

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

SiQube · 2025-01-08T09:36:05Z

pymovements.datasets.BSC => pymovements.DatasetDefinition('BSC')

Co-authored-by: SiQube <reich.davidr@gmail.com> Co-authored-by: Daniel G. Krakowczyk <daniel.krakowczyk@uni-potsdam.de>

…10 (#913) Co-authored-by: Daniel G. Krakowczyk <daniel.krakowczyk@uni-potsdam.de>

updates: - github.com/asottile/pyupgrade: v3.19.0 → v3.19.1 - github.com/jsh9/pydoclint: 0.5.10 → 0.6.0 - github.com/hhatto/autopep8: v2.3.1 → v2.3.2 - github.com/PyCQA/pylint: v3.3.2 → v3.3.3 Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel G. Krakowczyk <daniel.krakowczyk@uni-potsdam.de>

Co-authored-by: senisioi <sergiu.nisioi@gmail.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ng dataclasses

…s-loader' into yaml-public-datasets-loader

…_datasets_files`

…_test.py

for more information, see https://pre-commit.ci

tests/unit/dataset/dataset_library_test.py

for more information, see https://pre-commit.ci

…s of both

dkrako

This PR is a major breakthrough for pymovements!

YAML dataset definitions will simplify the dataset contribution process by a margin.

Thank you for all the hard work and keeping up with all my change requests!

As a side note: this PR set up a new record for the most comments in a PR, with a whopping count of 155 comments.

You now occupy the top 3 in the list. I may have played a part in this 😇

SiQube and others added 19 commits January 1, 2025 18:40

construct yaml loader and new dataset registry and library

f21bc07

move bsc.py to bsc.yaml

b763659

move sbsat.py to sbsat.yaml

ff17dc0

move codecomprehension.py to codecomprehension.yaml

27244b0

move copco.py to copco.yaml

d9da57a

move didec.py to didec.yaml

269dc6a

move emtec.py to emtec.yaml

e2c9d79

move fakenews.py to fakenews.yaml

6cc30c7

move gazebase*.py to gazebase*.yaml

16f2c89

move gaze_graph.py to gaze_graph.yaml

17bf864

move gaze_on_faces.py to gaze_on_faces.yaml

6311b89

move hbn.py to hbn.yaml

806a826

move judo1000.py to judo1000.yaml

c26392d

add special case of eyetracker within config

808dd1a

move interead.py to interead.yaml

91aca73

move potec.py to potec.yaml

188f1de

move toy_dataset*.py to toy_dataset*.yaml

1465ca4

adjust tests

9965874

[pre-commit.ci] auto fixes from pre-commit.com hooks

d75d153

for more information, see https://pre-commit.ci

SiQube mentioned this pull request Jan 1, 2025

remove ignore override from pyproject.toml #912

Open

adjust raise

a9deb54

SiQube and others added 7 commits February 12, 2025 10:30

update licenses (#915)

2ce121f

feat: split TextStimulus by column values (#879)

afcb9bb

Co-authored-by: SiQube <reich.davidr@gmail.com> Co-authored-by: Daniel G. Krakowczyk <daniel.krakowczyk@uni-potsdam.de>

build: update nbsphinx requirement from <0.9.6,>=0.8.8 to >=0.8.8,<0.…

79a3f0c

…10 (#913) Co-authored-by: Daniel G. Krakowczyk <daniel.krakowczyk@uni-potsdam.de>

fix: support missing dummy row in asc files (#918)

d4298c6

Co-authored-by: senisioi <sergiu.nisioi@gmail.com>

fix!: correctly load trial_columns to GazeDataFrame (#928)

b035025

build: update pyopenssl requirement (#923)

6e45eae

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

SiQube added 11 commits March 14, 2025 14:23

update and

90d62d0

remove from dataset

cd2c6b4

rename to

b39d2cb

incorporate new features due to dataclasses screen and eyetracker bei…

996b8b1

…ng dataclasses

tests definitions

1506691

remove write datasets tests

b7801b6

update datasets tests due to #919 being merged and renamed argument

8f54711

rename tests

5307e3b

Merge remote-tracking branch 'refs/remotes/origin/yaml-public-dataset…

ddfbf0b

…s-loader' into yaml-public-datasets-loader

mistakes with builtin types

14c7205

add test to load dataset from yaml file

2ca35bb

dkrako mentioned this pull request Mar 15, 2025

guide dataset contributors through contribution process by improving dataset library tests #1047

Open

dkrako and others added 5 commits March 15, 2025 15:06

Update assertion message of `test_dataset_library_contains_all_public…

a1471ee

…_datasets_files`

rename variables from dataset to definition in dataset_definition…

f269cbb

…_test.py

remove workaround for #919

3ba43a7

updated wording definition instead of dataset in datasets_test.py

0335522

[pre-commit.ci] auto fixes from pre-commit.com hooks

ddbe2ad

for more information, see https://pre-commit.ci

dkrako reviewed Mar 15, 2025

View reviewed changes

tests/unit/dataset/dataset_library_test.py Outdated Show resolved Hide resolved

tests/unit/dataset/dataset_library_test.py Show resolved Hide resolved

dkrako and others added 4 commits March 15, 2025 15:24

seems that #919 didn't resolve the issue with the __dict__ in one test

7da2d18

whoopsie

bcddbe5

Merge branch 'main' into yaml-public-datasets-loader

3cbaa43

[pre-commit.ci] auto fixes from pre-commit.com hooks

324e025

for more information, see https://pre-commit.ci

dkrako enabled auto-merge (squash) March 15, 2025 14:53

remove one of the names() implementations and use best docstring part…

52073b1

…s of both

dkrako approved these changes Mar 15, 2025

View reviewed changes

dkrako merged commit 42f33a2 into main Mar 15, 2025
25 checks passed

dkrako deleted the yaml-public-datasets-loader branch March 15, 2025 15:27

SiQube mentioned this pull request Mar 19, 2025

dataset: add potsdam binge pvt datasets #1044

Merged

11 tasks

This was referenced Mar 19, 2025

depr: deprecate utils.archives in favor of Dataset.extract() #1028

Merged

remove datasets.yaml #1053

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support yaml dataset definitions #914

feat: support yaml dataset definitions #914

SiQube commented Jan 1, 2025 •

edited by dkrako

Loading

codecov bot commented Jan 1, 2025 •

edited

Loading

SiQube commented Jan 8, 2025

dkrako left a comment

feat: support yaml dataset definitions #914

feat: support yaml dataset definitions #914

Conversation

SiQube commented Jan 1, 2025 • edited by dkrako Loading

💥 Breaking Changes

Description

codecov bot commented Jan 1, 2025 • edited Loading

Codecov Report

SiQube commented Jan 8, 2025

dkrako left a comment

Choose a reason for hiding this comment

SiQube commented Jan 1, 2025 •

edited by dkrako

Loading

codecov bot commented Jan 1, 2025 •

edited

Loading