Shalev-Lifshitz/STEVE-1

Builds on pretrained Video PreTraining (VPT) and MineCLIP models using an unCLIP-inspired two-stage approach: first adapting VPT to follow commands in MineCLIP's latent space, then training a conditional VAE prior to map text to latent codes. Achieves instruction-following on low-level keyboard/mouse controls with raw pixel inputs through self-supervised behavioral cloning and hindsight relabeling, requiring only $60 in compute and no large instruction-annotated datasets.

204 stars. No commits in the last 6 months.