neluca / tinybpe Public

Notifications You must be signed in to change notification settings
Fork 0
Star 1

This is an ultra-fast and clean code implementation of the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

1 star 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 12 Commits
examples		examples
scripts		scripts
src		src
tests		tests
tinybpe		tinybpe
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
README_zh.md		README_zh.md
setup.py		setup.py

Repository files navigation

tinybpe

This is a meticulously designed library of ultra fast and clean BPE algorithms.

python setup.py build_ext

WIP

About

This is an ultra-fast and clean code implementation of the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

tokenizer bpe llm bpe-tokenizer

Report repository

Releases

No releases published

Packages

No packages published

Languages