charent/Phi2-mini-Chinese

Implements a complete training pipeline from tokenizer creation (byte-level BPE) through CLM pretraining and SFT instruction tuning to DPO preference optimization, with Flash Attention 2 acceleration support. Uses BELLE datasets for both pretraining and fine-tuning, with specialized data cleaning and formatting (BOS/EOS tokens, document boundaries) optimized for Chinese text. Integrates with LangChain for RAG applications, enabling retrieval-augmented generation with local knowledge bases through the standard HuggingFace transformers interface.

586 stars. No commits in the last 6 months.