eric-ai-lab/R2H

Introduces two task formulations—Respond to Dialog History (RDH) for single-turn response generation and Respond during Interaction (RdI) for real-time cooperative navigation—converting three existing vision-and-dialog datasets (CVDN, AVDN, DialFRED) across photo-realistic and synthetic environments. Proposes SeeRee, a multimodal response generation model combining dialog history and language inquiries with visual observations plus oracle trajectory imagery, deployable both offline for RDH evaluation and as a live API in Matterport3D simulators for RdI interactions. Provides baseline comparisons with zero-shot multimodal LLM approaches and includes human evaluation across distinct environment types.

No commits in the last 6 months.