| Filename | Latest commit message | Latest commit date |
|---|---|---|
| assets | ||
| README.md | ||
EmbRACE-3K
Embodied Reasoning and Action in Complex Environments
📌 Overview
EmbRACE-3K is a large-scale dataset for embodied vision-language reasoning. It includes over 3,000 language-guided tasks and 26,000 decision steps across diverse, photorealistic environments built with Unreal Engine.
Recent vision-language models (VLMs) perform well on offline image and video tasks but struggle in interactive and embodied settings. In closed-loop environments, agents act from a first-person view where each decision alters future observations. Even advanced models like GPT-4o, Claude 3.5 Sonnet, and Gemini 2.5 Pro show clear limitations in spatial reasoning and long-horizon planning.
EmbRACE-3K provides a systematic benchmark to evaluate these limitations, and training on this dataset helps mitigate key challenges in step-wise embodied reasoning.
Each task provides:
- 👁️ Egocentric visual observations
- 📝 High-level task instructions
- 🕹 Grounded action sequences
- 💬 Step-level natural language rationales
We evaluate models on three core reasoning skills:
- 🧭 Exploration
- 🧠 Dynamic Spatial-Semantic Reasoning
- 🎯 Multi-stage Goal Execution
In zero-shot evaluations, all models perform under 20% success rate, revealing significant gaps. Fine-tuning Qwen2.5-VL-7B with supervised learning and reinforcement learning yields notable improvements across all task types.
🔁 Step-by-Step Closed-Loop Reasoning
This demo shows how a fine-tuned Qwen2.5-VL-7B agent reasons and acts step by step in a closed-loop setting, guided by egocentric perception and language-based reasoning.
📊 Dataset Overview
An overview of EmbRACE-3K, consisting of 3.1k tasks and 26k decision steps across diverse environments. Tasks involve multi-step grounded reasoning and perceptual decision making.
🛠 Dataset Construction Pipeline
The dataset is constructed in four stages:
- Sampling diverse 6-DoF egocentric views
- Generating task instructions using Gemini
- Collecting high-quality human demonstrations
- Annotating each action with step-level rationales
📂 Dataset and Code
Coming soon!
📖 Citation
If you find our work helpful, please cite:
@article{lin2025embrace3k,
title={EmbRACE-3K: Embodied Reasoning and Action in Complex Environments},
author={Lin, Mingxian and Huang, Wei and Li, Yitang and Jiang, Chengjie and Wu, Kui and Zhong, Fangwei and Qian, Shengju and Wang, Xin and Qi, Xiaojuan},
journal={arXiv preprint arXiv:2507.10548},
year={2025}
}