Implementing basic RNN #162

castelao · 2023-10-28T22:04:44Z

A work in progress. I'm mostly interested in loading a TF model from HDF and applying predict(), but I'll do my best in doing a complete implementation coherent with the rest of the library.

milancurcic · 2023-10-28T22:34:18Z

Amazing, thanks and great to see you, @castelao, after so many years. 🙂

Will review this coming week.

castelao · 2023-10-28T22:49:15Z

@milancurcic , yes, it's great to connect again. Thank you and the other developers for your time in this library! It is great.

I'm not fluent in modern Fortran, so if you see anything that doesn't make sense, please let me know. And be aware, it is still a WIP.

src/nf/nf_rnn_layer_submodule.f90

milancurcic · 2023-12-06T22:24:14Z

I added the support for rnn_layer to network % predict so that we can run the simple_rnn example. However I can't get the simple example to converge and I'm not very familiar with recurrent networks. @castelao is there a small toy example like the one in simple.f90 that's known to converge and fast? We can then use it for testing as well. With a working example and a small test suite (I can add this), this PR is almost good to go.

castelao · 2024-06-25T03:51:27Z

@milancurcic , I have to check how you updated the library since the last time I worked on this and see if what I did is still consistent. If it looks fine, I intend to work on the following:

The toy example
Load functionality from an HDF exported by TensorFlow. I want to avoid recompiling every time I change my coefficients.

Are there any other requirements before I can submit this PR for review? It will be slow progress, but I'm back to this.

jvdp1

As I am currently working with neural-fortran, I did a quick review of this PR and left a few minor comments. Overall LGTM. Thank you.

src/nf/nf_layer_submodule.f90

jvdp1 · 2024-06-25T05:34:13Z

src/nf/nf_rnn_layer.f90

+    procedure :: get_params
+    procedure :: init
+    procedure :: set_params
+    ! procedure :: set_state


Suggested change

! procedure :: set_state

jvdp1 · 2024-06-25T05:34:50Z

src/nf/nf_rnn_layer.f90

+    !module subroutine set_state(self, state)
+    !  type(rnn_layer), intent(inout) :: self
+    !  real, intent(in), optional :: state(:)
+    !end subroutine set_state
+


Suggested change

!module subroutine set_state(self, state)

! type(rnn_layer), intent(inout) :: self

! real, intent(in), optional :: state(:)

!end subroutine set_state

jvdp1 · 2024-06-25T05:41:25Z

src/nf/nf_rnn_layer_submodule.f90

+    db = gradient * self % activation % eval_prime(self % z)
+    dw = matmul(reshape(input, [size(input), 1]), reshape(db, [1, size(db)]))
+    self % gradient = matmul(self % weights, db)
+    self % dw = self % dw + dw


I recently modified these lines in nf_dense_layer_submodule.f90 for better performances. The same logic could be done here IMO.

jvdp1 · 2024-06-25T05:43:14Z

src/nf/nf_rnn_layer_submodule.f90

+    class(rnn_layer), intent(in) :: self
+    real, allocatable :: params(:)
+
+    params = [ &


The pack can be avoided here, by using pointers or because it is not needed. See here for some changes.

Same comment for the subroutines get_gradients and set_params below.

milancurcic · 2024-06-25T17:38:27Z

@castelao Thanks for all the additions. Sounds good.

It may be helpful to merge main into here, which would reflect the few recent changes in other layers that @jvdp1 mentioned.
Regarding loading recurrent layers from Keras HDF5 files; unless you need it, I'd say it's not a priority. If you need it, let's add it in a later PR, as there would be considerable additions needed to make it work (e.g. a Keras script example in neural-fortran/keras-snippets). That format is also no longer the latest Keras saved model format (they change too often!). Related but not part of this PR, I'd like to find an easy way to make this part (and HDF5 dependency) optional at build time. Probably via preprocessor flags in the code, but I'd need to read up how to do this in fpm.toml.
But a standalone example program in examples/ is important.

castelao · 2024-06-26T03:38:33Z

@jvdp1 , thanks for your suggestions. I'll work on that.

@milancurcic , yes, I have already rebased it with main. I'm interested in the loading from Keras output, but I see your point. I'll leave that loading capability for another PR, but I'll work on the example for this one. Thanks!

The dimensions don't match, but let's start with something that compile.

Note a hardcoded 'simple_rnn_cell_23' that must be resolved later.

I'll try with 1D with a state memory and the option to reset state for processing a new time series.

Each neuron is affected by all states. With this change the forward procedure is working correctly. I verified a couple of test cases.

Instead of reset on network level.

Previously `quadratic_derivative`.

Co-authored-by: Jeremie Vandenplas <[email protected]>

castelao commented Nov 1, 2023

View reviewed changes

src/nf/nf_rnn_layer_submodule.f90 Show resolved Hide resolved

milancurcic added the enhancement New feature or request label Nov 1, 2023

castelao force-pushed the RNN branch 2 times, most recently from a9111b3 to 3fa5281 Compare November 14, 2023 16:24

castelao force-pushed the RNN branch from 3396edb to 745c2d6 Compare June 23, 2024 18:49

castelao self-assigned this Jun 23, 2024

jvdp1 reviewed Jun 25, 2024

View reviewed changes

castelao added 17 commits September 14, 2024 18:33

Prototyping RNN layer based on Dense

64e1b69

The dimensions don't match, but let's start with something that compile.

Extenting uses

8c11911

Reading coefficients from h5f model

adef7d7

Note a hardcoded 'simple_rnn_cell_23' that must be resolved later.

feat: get_params()

b51d66f

feat: set_params()

a797502

feat: get_num_params()

ff1c392

Initializing recurrent kernel and states

f686950

feat: forward()

acf1afd

More informative error messages

69fed32

Minor adjustments on rnn_layer

fd24e16

Constructor for RNN

7415081

Loading rnn constructor in the root

6f56863

Back to 1D concept

ad598a8

I'll try with 1D with a state memory and the option to reset state for processing a new time series.

fix: Recurrent is actually a square matrix

0ae7af1

Each neuron is affected by all states. With this change the forward procedure is working correctly. I verified a couple of test cases.

Apply loss function if RNN is the output layer

c164924

fix: Getting biases

55ad96d

Allowing backward 1D from dense to RNN

b345865

castelao and others added 13 commits September 14, 2024 18:36

Allowing backward 1D from RNN

91b85e0

Allowing forward from dense to RNN

5e197f0

Allowing forward from RNN

7f671c8

Getting output from RNN

c27f59c

feat: Implementing reset state for RNN

524d2c4

refactor: set_state() on layer level

598f9e7

Instead of reset on network level.

wip: A simple RNN example

b7bead6

feat: layer getting gradient from RNN

088e4f3

feat: layer setting params for RNN

4d0a4fd

Might not use set_state at rnn_layer level

ee516a8

fix: New access point to 'loss % derivative'

07f7587

Previously `quadratic_derivative`.

Define set_state as pure

9b22826

Co-authored-by: Jeremie Vandenplas <[email protected]>

fix: pure interface for set_state

5bc9bc5

castelao force-pushed the RNN branch from a2c67df to 5bc9bc5 Compare October 21, 2024 05:47

fix: Conciliating with latest main state

4c7c0b9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementing basic RNN #162

Implementing basic RNN #162

castelao commented Oct 28, 2023

milancurcic commented Oct 28, 2023 •

edited

Loading

castelao commented Oct 28, 2023

milancurcic commented Dec 6, 2023

castelao commented Jun 25, 2024

jvdp1 left a comment

jvdp1 Jun 25, 2024

jvdp1 Jun 25, 2024

jvdp1 Jun 25, 2024

jvdp1 Jun 25, 2024

jvdp1 Jun 25, 2024

milancurcic commented Jun 25, 2024

castelao commented Jun 26, 2024

Implementing basic RNN #162

Are you sure you want to change the base?

Implementing basic RNN #162

Conversation

castelao commented Oct 28, 2023

milancurcic commented Oct 28, 2023 • edited Loading

castelao commented Oct 28, 2023

milancurcic commented Dec 6, 2023

castelao commented Jun 25, 2024

jvdp1 left a comment

Choose a reason for hiding this comment

jvdp1 Jun 25, 2024

Choose a reason for hiding this comment

jvdp1 Jun 25, 2024

Choose a reason for hiding this comment

jvdp1 Jun 25, 2024

Choose a reason for hiding this comment

jvdp1 Jun 25, 2024

Choose a reason for hiding this comment

jvdp1 Jun 25, 2024

Choose a reason for hiding this comment

milancurcic commented Jun 25, 2024

castelao commented Jun 26, 2024

milancurcic commented Oct 28, 2023 •

edited

Loading