Skip to content

Commit f860491

Browse files
Update tokenizers requirement from 0.21 to 0.22
Updates the requirements on [tokenizers](https://github.com/huggingface/tokenizers) to permit the latest version. - [Release notes](https://github.com/huggingface/tokenizers/releases) - [Changelog](https://github.com/huggingface/tokenizers/blob/main/RELEASE.md) - [Commits](huggingface/tokenizers@v0.21.0...v0.22.0) --- updated-dependencies: - dependency-name: tokenizers dependency-version: 0.22.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
1 parent 5400b0f commit f860491

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

crates/bpe/benchmarks/Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,4 +23,4 @@ bpe-openai = { path = "../../bpe-openai" }
2323
criterion = "0.7"
2424
rand = "0.9"
2525
tiktoken-rs = "0.7"
26-
tokenizers = { version = "0.21", features = ["http"] }
26+
tokenizers = { version = "0.22", features = ["http"] }

0 commit comments

Comments
 (0)