Haste RNN support

[thegodone]

Do you see a way to replace the keras GRU by the Haste_tf GRU version from here https://github.com/lmnt-com/haste/blob/master/docs/tf/haste_tf.md ?

there are lot of advantage in this package for regularization at cuda level etc...

unfortunately I don't see a way to plug the haste_tf at keras level lmnt-com/haste#33 ?

your help would be appreciate

[thegodone]

I was able to do this :

import numpy as np
import tensorflow as tf
import haste_tf as haste

train_x = np.random.rand(500,40,20)
train_y = np.random.rand(500,40,1)

inputs = tf.keras.Input(shape=(train_x.shape[1], train_x.shape[2]))
haste1 = haste.LayerNormGRU(20, direction='unidirectional', zoneout=0.1, dropout=0.1)
fc1 = tf.keras.layers.Dense(60, activation='relu', kernel_initializer='he_uniform')
dr1 = tf.keras.layers.Dropout(0.2)
fc2 = tf.keras.layers.Dense(1)

x, state = haste1(inputs, training=True)
x = fc1(inputs)
x = dr1(x)
outputs = fc2(x)

model = tf.keras.Model(inputs=inputs, outputs=outputs)

print(model.summary())

opt = tf.keras.optimizers.Adam(learning_rate=0.01)
model.compile(loss='categorical_crossentropy', optimizer=opt)

model_hist = model.fit(train_x, train_y, epochs=10, batch_size=32, verbose=1)

is it possible to add this layer in place of pure "GRU" ?

[PatReis]

I will look into it. However, since kgcnn is fully based on keras, I am not fully sure if we sould add more external dependencies apart from keras. But maybe as a layer module is should be fine.

[thegodone]

I really want to test this special layer if it's possible. Maybe you can an exception for this one.

[thegodone]

there is a typo here

"dopout" => "dropout"

[thegodone]

another issue than

[PatReis]

Thanks, I am still trying to get haste installed. But I think this error is not from us. We must post this error to haste, since I think there must be a
tf.compat.v1.get_variable("gamma",shape=1,initializer=tf.compat.v1.keras.initializers.ones())
since
tf.compat.v1.get_variable("gamma",initializer=tf.compat.v1.keras.initializers.ones())
will always throw an error regardless of the input.

I am not sure that there is a great advantage using haste Cells. I think there is an advantage using the final haste GRU. But because the recurrent step requires some attention pooling over the nodes (for graph networks) we are stuck with the GRUCells. However, I will think about this, maybe I did make a mistake in my logic/implementation.

[thegodone]

if you want to open an issue in Haste repo. Go ahead

[PatReis]

Okay, thanks, I opened and issue in haste.