[WIP] Support target features

[anderleich]

This PR intends to add target features support to OpenNMT-py v3.0. All the code has been adapted for this new version.

Both source and target features support has been refactored for a more simplified handling of features. The way features are passed to the system has been changed and now features are appended to the actual textual data instead of providing a separate file. This also simplifies the way features are passed during inference and to the server. It uses the special character ￨ as a feature separator, as in the previous versions of the OpenNMT framework. For instance:

 I￨1￨3 love￨0￨1 eating￨0￨1 pizza￨0￨1

I've also added a way to provide default values for features. This can be really useful when mixing task specific data (with features) with general data which has not been annotated. Additionally, the filterfeats transform is no longer required and features are checked in the corpus loading process.

A YAML configuration file would look like this:

data:
    train:
        path_src: src_with_features.txt  #  I￨1￨3 love￨0￨1 eating￨0￨1 pizza￨0￨1
        path_tgt: tgt_with_features.txt  #  Me￨1 gusta￨0 comer￨0 pizza￨0
        transforms: [onmt_tokenize, inferfeats, filtertoolong]
    valid:
        path_src: src_with_features.txt
        path_tgt: tgt_with_features.txt
        transforms: [onmt_tokenize, inferfeats]

save_data: ./data
n_sample: -1

# # Vocab opts
src_vocab: data.vocab.src
tgt_vocab: data.vocab.tgt
n_src_feats: 2
n_tgt_feats: 1
src_feats_defaults: "0￨1"
tgt_feats_defaults: "1"
feat_merge: "sum"

For now, I've made the necessary changes in the code for vocabulary generation. That is, to make onmt_build_vocab work.

@vince62s , do you think this a good starting point?

[vince62s]

Hello @anderleich yes good starting point even though would be good to fix the Lint & Tests and maybe add a test to show everything works fine.

[anderleich]

Yes, totally agree. I just wanted to make sure I was on the right track. For now, I've just made the necessary changes in the code for vocabulary generation. That is, to make onmt_build_vocab work. Once I get the training part done, I will add the necessary tests to ensure everything works as expected.

[anderleich]

Closing this PR as v3.0 branch was merged into master. I keep working on target features in a new one: #2289