-
-
Notifications
You must be signed in to change notification settings - Fork 611
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consider making the :expand
option the default in @layer
#2531
Comments
No, we had this automatic detection, and it didn't work. |
what is exactly that is not working? I'm testing now with making everything |
Every layer whose implementation contains other layers, or things which could plausibly be layers, but whose interface does not. |
Ok, I see that we want: julia> LayerNorm(10)
LayerNorm(10) # 20 parameters
julia> MultiHeadAttention(64 => 1024 => 1024, nheads = 8)
MultiHeadAttention(64 => 1024 => 1024; nheads=8) # 1_245_184 parameters Maybe we should change the default then? I feel that 90% cases one would do with expand, since it cover both the layer with plain arrays and the container type. |
IDK, maybe the other default would have been better, but I'm against unnecessary churn. My guess was that most people making layers are wrapping some parameter arrays, not building container layers. But I have no data. The present default means that if you don't ever think about |
:expand
option the default in @layer
I think most people don't make layers, they make blocks and models. |
Yea like I said, no data on which is more common. Certainly Flux's source has mostly compact ones. Having another option (but and not breaking existing |
It doesn't seem to be really needed, we could automatically detect if the layer contains only leaf types or is a nested container
The text was updated successfully, but these errors were encountered: