tiktoken-go and tiktoken-cli

These are ecosystem siblings: the Go binding provides language-specific access to OpenAI's tokenizer, while the CLI wrapper offers a language-agnostic interface to the same underlying tokenization library for use across different programming environments.

tiktoken-go
48
Emerging
tiktoken-cli
27
Experimental
Maintenance 2/25
Adoption 10/25
Maturity 16/25
Community 20/25
Maintenance 0/25
Adoption 4/25
Maturity 9/25
Community 14/25
Stars: 895
Forks: 102
Downloads: —
Commits (30d): 0
Language: Go
License: MIT
Stars: 7
Forks: 3
Downloads: —
Commits (30d): 0
Language: Python
License: —
Stale 6m No Package No Dependents
Stale 6m No Package No Dependents

About tiktoken-go

pkoukk/tiktoken-go

go version of tiktoken

Implements BPE tokenization for OpenAI models with pluggable dictionary loaders—supporting runtime downloads, cached dictionaries via `TIKTOKEN_CACHE_DIR`, or offline embedded encodings. Provides both encoding-level (cl100k_base, p50k_base) and model-specific APIs (gpt-4, gpt-3.5-turbo), with built-in helpers for counting tokens in chat completion messages compatible with the go-openai SDK.

About tiktoken-cli

oelmekki/tiktoken-cli

Simple wrapper around tiktoken to use it in your favorite language.

Scores updated daily from GitHub, PyPI, and npm data. How scores work