Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding MapDataset and MapDataloader to Caked #35

Closed
wants to merge 27 commits into from
Closed

Adding MapDataset and MapDataloader to Caked #35

wants to merge 27 commits into from

Conversation

hllelli2
Copy link
Collaborator

@hllelli2 hllelli2 commented Oct 1, 2024

  • MapDataset, ArrayDataset and MapDataloader to handle inputs of .mrc or .map data.
  • Transformations and Augmentations build upon CCPEM_Utils.
  • Hdf5 intermediate data store
  • Tests

hllelli2 added 27 commits June 19, 2024 16:08
@hllelli2
Updates to gitignore
c64fbff
@hllelli2
basic setup config for pip install
2f4d58f
@hllelli2
Added MapDataset and MapDataLoader
6afd99e
@hllelli2
Added custom augments and transfroms as well as tiling
ef73dd2
@hllelli2
Added tests for new code
2ab4260
@hllelli2
feat: Add HDF5DataStore class for handling hdf5 files
8b9d06d
@hllelli2
Add pytest configuration file and conftest.py for test fixtures
d8729a1
@hllelli2
Added a dataset config class to have all the defaults in one place
1a6ed24
@hllelli2
Create a class for lazy-loading a hdf5 file
c8486b8
@hllelli2
Moved multi-processing to utils, Added an array dataset which can loa…
3779ee4 
…d arrays into the hdf5 without a file.
@hllelli2
Moved Multi-processing code here, code here to duplciate arrays from …
b9a91cb 
…the dataloader, augment them and saves them to array. For a specific usecase in ml-toolkit
@hllelli2
Changed augment to take arrays instead of map-objects to make it comp…
ab2ce2f 
…atible with ArrayDataset
@hllelli2
Removed mapObject transform base
d6eda3a
@hllelli2
chore: Refactor transforms module and update transform classes
a86c910
@hllelli2
first attempt at reducing if label/weight... is not none repeated code
41d40b3
@hllelli2
chore: Add test fixtures and update conftest.py for test setup
6f73aae
@hllelli2
set_gpu is not in the current ccpem-utils so check added JIC
07e2318
@hllelli2
chore: Refactor HDF5DataStore class and add support for temporary dir…
a6cc9b1 
…ectory storage
@hllelli2
Removed "_" id logic in MapDataLoader, might reimplement later but no…
05f792b 
…t sure if needed for multi-class porblems
@hllelli2
Refactor process_datasets function and remove unnecessary code
5a770f7
@hllelli2
Refactor transforms module and update transform classes
0a99fd7
@hllelli2
re-added __del__ method to HDF5Store
8772fa6
@hllelli2
Refactor MapDataset and ArrayDataset classes to handle weight tensors
93c8424
@hllelli2
Refactor
9128e9a
@hllelli2
Refactor DecomposeToSlices, MapObjectMaskCrop, and MapObjectPadding c…
b2a49c1 
…lasses
@hllelli2
Refactor test_map_io.py to handle weight tensors and update dataset l…
ce47030 
…engths
@hllelli2
Merge branch 'main' into CCPEM-AddingMapDataset
6ff4461
@hllelli2 hllelli2 requested a review from aj26git October 1, 2024 12:59
@hllelli2 hllelli2 self-assigned this Oct 1, 2024
@hllelli2 hllelli2 closed this Oct 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aj26git aj26git Awaiting requested review from aj26git

Assignees

@hllelli2 hllelli2

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@hllelli2