openscilab/tocount

Provides multiple estimation strategies including rule-based approaches and linear regression models trained on tiktoken encodings (R50K, CL100K, O200K) plus emerging model tokenizers like Deepseek R1 and Llama 3.1. Uses a unified `TextEstimator` interface with pre-trained models benchmarked against real-world chat datasets, offering accuracy trade-offs from R² 0.62–0.97 depending on model selection and language specificity. Targets token budgeting for LLM applications with language-specific variants optimized for English text.

Available on PyPI.