Skip to content

Commit ddfb7ce

Browse files
committed
Bump version to 1.24.0
1 parent b287706 commit ddfb7ce

File tree

2 files changed

+19
-1
lines changed

2 files changed

+19
-1
lines changed

CHANGELOG.md

+18
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ The project follows [semantic versioning 2.0.0](https://semver.org/). The API co
1313
* Python
1414
* `pyonmttok.BPELearner`
1515
* `pyonmttok.SentencePieceLearner`
16+
* `pyonmttok.SentencePieceTokenizer`
1617
* `pyonmttok.Tokenizer`
1718

1819
---
@@ -23,6 +24,23 @@ The project follows [semantic versioning 2.0.0](https://semver.org/). The API co
2324

2425
### Fixes and improvements
2526

27+
## [v1.24.0](https://github.com/OpenNMT/Tokenizer/releases/tag/v1.24.0) (2021-02-16)
28+
29+
### New features
30+
31+
* Add `verbose` flag in file tokenization APIs to log progress every 100,000 lines
32+
* [Python] Add `options` property to `Tokenizer` instances
33+
* [Python] Add class `pyonmttok.SentencePieceTokenizer` to help creating a tokenizer compatible with SentencePiece
34+
35+
### Fixes and improvements
36+
37+
* Fix deserialization into `Token` objects that was sometimes incorrect
38+
* Fix Windows compilation
39+
* Fix Google Test integration that was sometimes installed as part of `make install`
40+
* [Python] Update pybind11 to 2.6.2
41+
* [Python] Update ICU to 66.1
42+
* [Python] Compile ICU with optimization flags
43+
2644
## [v1.23.0](https://github.com/OpenNMT/Tokenizer/releases/tag/v1.23.0) (2020-12-30)
2745

2846
### Changes

bindings/python/setup.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ def _maybe_add_library_root(lib_name, header_only=False):
4545

4646
setup(
4747
name="pyonmttok",
48-
version="1.23.0",
48+
version="1.24.0",
4949
license="MIT",
5050
description="OpenNMT tokenization library",
5151
long_description=_get_long_description(),

0 commit comments

Comments
 (0)