unitythemaker/tokdu
tokdu (Token Disk Usage) is a terminal-based utility that helps you analyze and visualize token usage in your codebase. Similar to the classic du (disk usage) command, tokdu shows you how many tokens your files and directories consume, which is essential when working with Large Language Models (LLMs) that have token limits.
No commits in the last 6 months. Available on PyPI.
Stars
4
Forks
—
Language
Python
License
MIT
Category
Last pushed
Apr 07, 2025
Monthly downloads
36
Commits (30d)
0
Dependencies
3
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/unitythemaker/tokdu"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
lenML/tokenizers
a lightweight no-dependency fork from transformers.js (only tokenizers)
aiqinxuancai/TiktokenSharp
Token calculation for OpenAI models, using `o200k_base` `cl100k_base` `p50k_base` encoding.
pkoukk/tiktoken-go
go version of tiktoken
tryAGI/Tiktoken
This project implements token calculation for OpenAI's gpt-4 and gpt-3.5-turbo model,...
dqbd/tiktokenizer
Online playground for OpenAPI tokenizers