neuron-core/tokenizer
High-Performance Tokenizer implementation in PHP.
It provides a Byte-Pair Encoding (BPE) tokenizer, loading pre-trained vocabulary and merge files (e.g., from Hugging Face's GPT-2 datasets). The implementation emphasizes performance through a cache strategy and priority queue for efficient tokenization, targeting PHP applications requiring NLP capabilities.
Stars
6
Forks
1
Language
PHP
License
MIT
Category
Last pushed
Oct 21, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/agents/neuron-core/tokenizer"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
driftbase-labs/driftbase-python
Local-first behavioral drift monitoring for AI agents. One decorator, SQLite, no cloud.
aptible/unpage
Unpage is the open source framework for building SRE agents with infrastructure context and...
Harshit-J004/toolguard
Pytest-style reliability testing for AI agent tool chains. Catches hallucinated payloads, schema...
2001Haru/TokenWaster
The Most Useless EVER agent assistant in the Human History. Always trying to read everything in...
valmi-io/value
⚡ "Value" - https://value.valmi.io . Valmi Value is Outcome-based billing and payments...