kyegomez/RT-2

Combines a vision encoder with a PaLM-E language model backbone to embed images and language into a unified multimodal space, enabling end-to-end training on both web-scale and robotics datasets. The model outputs action tokens directly, allowing robot camera observations paired with natural language instructions to be translated into executable control commands. Includes PyTorch implementation with straightforward API for integration into robotics pipelines and fine-tuning on custom robot demonstration data.

554 stars. No commits in the last 6 months.