File tree 2 files changed +8
-2
lines changed
2 files changed +8
-2
lines changed Original file line number Diff line number Diff line change @@ -21,13 +21,19 @@ The project follows [semantic versioning 2.0.0](https://semver.org/). The API co
21
21
22
22
### New features
23
23
24
+ ### Fixes and improvements
25
+
26
+ ## [ v1.21.0] ( https://github.com/OpenNMT/Tokenizer/releases/tag/v1.21.0 ) (2020-10-22)
27
+
28
+ ### New features
29
+
24
30
* Accept vocabularies with tab-separated frequencies (format produced by SentencePiece)
25
31
26
32
### Fixes and improvements
27
33
28
34
* Fix BPE vocabulary restriction when words have a leading or trailing joiner
29
35
* Raise an error when using a multi-character joiner and ` support_prior_joiner `
30
- * [ Python] Implement ` __hash__ ` method of ` pyonmttok.Token ` objects to be consistent with the ` __eq__ ` implementation
36
+ * [ Python] Implement ` __hash__ ` method of ` pyonmttok.Token ` objects to be consistent with the ` __eq__ ` implementation
31
37
* [ Python] Declare ` pyonmttok.Tokenizer ` arguments (except ` mode ` ) as keyword-only
32
38
* [ Python] Improve compatibility with Python 3.9
33
39
Original file line number Diff line number Diff line change @@ -40,7 +40,7 @@ def _maybe_add_library_root(lib_name, header_only=False):
40
40
41
41
setup (
42
42
name = "pyonmttok" ,
43
- version = "1.20 .0" ,
43
+ version = "1.21 .0" ,
44
44
license = "MIT" ,
45
45
description = "OpenNMT tokenization library" ,
46
46
long_description = _get_long_description (),
You can’t perform that action at this time.
0 commit comments