mmcarpi / bpe Public

Notifications You must be signed in to change notification settings
Fork 0
Star 0

Byte Pair Encoding Tokenizer

0 stars 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
tests		tests
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
__init__.py		__init__.py
heap.py		heap.py
llst.py		llst.py
tokenizer.py		tokenizer.py
train.py		train.py

Repository files navigation

BPE Tokenizer

A fast Byte Pair Encoding Tokenizer written in Python. Inspired by minBPE.

Requirements:

regex

About

Byte Pair Encoding Tokenizer

Report repository

Releases

No releases published

Packages

No packages published

Languages

Python 100.0%