File tree 2 files changed +19
-1
lines changed
2 files changed +19
-1
lines changed Original file line number Diff line number Diff line change @@ -13,6 +13,7 @@ The project follows [semantic versioning 2.0.0](https://semver.org/). The API co
13
13
* Python
14
14
* ` pyonmttok.BPELearner `
15
15
* ` pyonmttok.SentencePieceLearner `
16
+ * ` pyonmttok.SentencePieceTokenizer `
16
17
* ` pyonmttok.Tokenizer `
17
18
18
19
---
@@ -23,6 +24,23 @@ The project follows [semantic versioning 2.0.0](https://semver.org/). The API co
23
24
24
25
### Fixes and improvements
25
26
27
+ ## [ v1.24.0] ( https://github.com/OpenNMT/Tokenizer/releases/tag/v1.24.0 ) (2021-02-16)
28
+
29
+ ### New features
30
+
31
+ * Add ` verbose ` flag in file tokenization APIs to log progress every 100,000 lines
32
+ * [ Python] Add ` options ` property to ` Tokenizer ` instances
33
+ * [ Python] Add class ` pyonmttok.SentencePieceTokenizer ` to help creating a tokenizer compatible with SentencePiece
34
+
35
+ ### Fixes and improvements
36
+
37
+ * Fix deserialization into ` Token ` objects that was sometimes incorrect
38
+ * Fix Windows compilation
39
+ * Fix Google Test integration that was sometimes installed as part of ` make install `
40
+ * [ Python] Update pybind11 to 2.6.2
41
+ * [ Python] Update ICU to 66.1
42
+ * [ Python] Compile ICU with optimization flags
43
+
26
44
## [ v1.23.0] ( https://github.com/OpenNMT/Tokenizer/releases/tag/v1.23.0 ) (2020-12-30)
27
45
28
46
### Changes
Original file line number Diff line number Diff line change @@ -45,7 +45,7 @@ def _maybe_add_library_root(lib_name, header_only=False):
45
45
46
46
setup (
47
47
name = "pyonmttok" ,
48
- version = "1.23 .0" ,
48
+ version = "1.24 .0" ,
49
49
license = "MIT" ,
50
50
description = "OpenNMT tokenization library" ,
51
51
long_description = _get_long_description (),
You can’t perform that action at this time.
0 commit comments