New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Model Parallelism and Distributed Training #1233

Open

3 tasks done

avik-pal opened this issue Feb 5, 2025 · 0 comments

Open

3 tasks done

Model Parallelism and Distributed Training #1233

avik-pal opened this issue Feb 5, 2025 · 0 comments

Labels

Member

avik-pal commented Feb 5, 2025 •

edited

Loading

Batch Parallelism already works with Lux. Users can checkout https://lux.csail.mit.edu/stable/api/Lux/distributed_utils. This issue is to plan out how to implement fully sharded data parallelism using Reactant and XLA

Upstream Features Needed

Sharding
Single Node Parallelism
Distributed Parallelism -- IFRT

ML Package Docs

TODO -- How do other packages like Jax and Pytorch expose this to users?

avik-pal added the reactant label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment