Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch concatenate2 to concatenate3 #529

Open
dcherian opened this issue Jul 31, 2024 · 1 comment
Open

Switch concatenate2 to concatenate3 #529

dcherian opened this issue Jul 31, 2024 · 1 comment

Comments

@dcherian
Copy link

I noticed you're using concatenate2 from dask. I've found this to be quite wasteful, it's cool in its recursive ability but sucks in that it repeatedly allocates new memory.

Cubed should be using concatenate3 which allocates the final output array up top, and then assigns regions of that output, similar to np.block.

With dask at least, I've noticed that _concatenate2 takes half the time it takes to execute a reduction haha.

PS: it feels like the long-term impact of dask is to give a small number of us a common language for these internal utilities. I see your lol_product!

@TomNicholas
Copy link
Member

it feels like the long-term impact of dask is to give a small number of us a common language for these internal utilities

💯 💯 💯

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants