sayakpaul/count-tokens-hf-datasets

This project shows how to derive the total number of training tokens from a large text dataset from 🤗 datasets with Apache Beam and Dataflow.

/ 100

Experimental

No commits in the last 6 months.

No License Stale 6m No Package No Dependents

Maintenance 0 / 25

Adoption 7 / 25

Maturity 1 / 25

Community 4 / 25

Stars

Forks

Language

Python

License

—

Category

Last pushed

Oct 20, 2022

Commits (30d)

Get this data via API

curl "https://pt-edge.onrender.com/api/v1/quality/transformers/sayakpaul/count-tokens-hf-datasets"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

Higher-rated alternatives

huggingface/course

The Hugging Face course on Transformers

NielsRogge/Transformers-Tutorials

This repository contains demos I made with the Transformers library by HuggingFace.

azukds/tubular

Python package implementing ML feature engineering and pre-processing for polars or pandas dataframes.

rickiepark/nlp-with-transformers

<트랜스포머를 활용한 자연어 처리> 예제 코드를 위한 저장소입니다.

huggingface/audio-transformers-course

The Hugging Face Course on Transformers for Audio