tiktoken¶
tiktoken is a fast BPE tokeniser for use with OpenAI's models
homepage: https://github.com/openai/tiktoken
Available installations¶
| tiktoken version | Supported CPU targets | Supported GPU targets | EESSI version | Module |
|---|---|---|---|---|
| 0.9.0 | generic: aarch64, x86_64Arm: a64fx, neoverse_n1, neoverse_v1, nvidia/graceAMD: zen2, zen3, zen4, zen5Intel: haswell, skylake_avx512, sapphirerapids, icelake, cascadelake |
(none) | 2025.06 | tiktoken/0.9.0-GCCcore-13.3.0 |