Transformer Gaps Summary

[artyom-beilis]

At this point it looks like the gaps are:

1. Implement norm_layer - it probably can be implemented by slight modifications of BN layer
2. Fix Linear layer to support cases when the input shape isn't of size 2 (should be rather simple) but it causes failure in various ops related to transformers
3. GELU

To Be continued

[trholding]

I was busy this weekend, I'll test in 8 hours.

Some reference that may be useful:

https://github.com/karpathy/micrograd/blob/c911406e5ace8742e5841a7e0df113ecb5d54685/micrograd/nn.py
https://github.com/geohot/tinygrad/blob/df31dde174ce7c64421896f26116eb3a4e24d51f/tinygrad/nn/__init__.py

https://github.com/geohot/tinygrad/blob/df31dde174ce7c64421896f26116eb3a4e24d51f/tinygrad/tensor.py#L277
https://github.com/pytorch/pytorch/blob/2ddefbdc3c4fab70b4c2898a0a25e403610741fc/torch/nn/modules/activation.py#L649

[trholding]

Okay I built it again. Status: Same results. Will wait for your enhancements. :)

[artyom-beilis]

I fixed the 2nd still need to implement 1st and 3rd

[trholding]

I can't wait to try this. I am sorry that could not help you much. I am not that very competent in this mind blowing stuff. I'm still trying to read about neural networks and how they work, I want to know it in the context of C/C++ or even Rust implementation - didn't find good resources yet. While python is easy it will be expensive on constrained / embedded systems. The real reason is that I am trying to build a very low bit rate voice codec that can work over very low bit rate mesh networks.