mscheong01/speculative_decoding.c

Implements speculative decoding by pairing a lightweight draft model with a larger base model to accelerate inference—the draft model generates token candidates which the base model validates and accepts or rejects, with visual output distinguishing accepted vs. rejected tokens. Extends llama2.c with multi-token forwarding support and works with both TinyLlama and Meta's Llama 2 model weights. Trade-off: generation length is limited by the draft model's maximum sequence length.

No commits in the last 6 months.