tokenizers¶
Fast State-of-the-Art Tokenizers optimized for Research and Production
homepage: https://github.com/huggingface/tokenizers
Available installations¶
| tokenizers version | Supported CPU targets | Supported GPU targets | EESSI version | Module |
|---|---|---|---|---|
| 0.15.2 | generic: aarch64, x86_64Arm: a64fx, neoverse_n1, neoverse_v1, nvidia/graceAMD: zen2, zen3, zen4Intel: haswell, skylake_avx512, sapphirerapids, icelake, cascadelake |
(none) | 2023.06 | tokenizers/0.15.2-GCCcore-12.3.0 |
Extensions¶
Overview of extensions included in tokenizers installations
huggingface-hub¶
huggingface-hub version |
tokenizers modules that include it |
|---|---|
| 0.21.4 | tokenizers/0.15.2-GCCcore-12.3.0 |
tokenizers¶
tokenizers version |
tokenizers modules that include it |
|---|---|
| 0.15.2 | tokenizers/0.15.2-GCCcore-12.3.0 |