- Python 99.6%
- Shell 0.4%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| assets | ||
| configs/eval | ||
| docs | ||
| eval | ||
| scripts | ||
| segvggt | ||
| .gitignore | ||
| LICENSE.txt | ||
| README.md | ||
| requirements.txt | ||
SegVGGT: Joint 3D Reconstruction and Instance Segmentation from Multi-View Images
Authors: Jinyuan Qu, Hongyang Li, and Lei Zhang.
Overview
SegVGGT is a unified feed-forward framework for joint 3D reconstruction and 3D instance segmentation from unposed multi-view RGB images. It integrates object queries into a geometry-grounded transformer and introduces the FADA module to guide instance-aware attention, enabling accurate reconstruction and segmentation in a single forward pass.
Installation
First, clone this repository to your local machine, and install the dependencies.
# Clone SegVGGT
git clone https://github.com/IDEA-Research/SegVGGT
cd SegVGGT
# Create environment
conda create -n segvggt python=3.10.19 -y
conda activate segvggt
# Install pytorch 2.3.1 with cuda 12.1
pip install torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu121
# Install other requirements
pip install -r requirements.txt
Data Preparation
For ScanNet and ScanNet200 evaluation, please prepare the processed 3D annotations and extracted 2D RGB-D frames under data_processing/.
Detailed instructions are provided in docs/data_preparation.md.
The directory structure after data preparation should be as below:
data_processing/
├── scannet/ # or scannet200/
│ ├── meta_data/
│ │ └── scannetv2_val.txt
│ ├── points/
│ │ └── sceneXXXX_XX.bin
│ ├── semantic_mask/
│ │ └── sceneXXXX_XX.bin
│ ├── instance_mask/
│ │ └── sceneXXXX_XX.bin
│ ├── super_points/
│ │ └── sceneXXXX_XX.bin
│ └── scenes_2d/
│ └── sceneXXXX_XX/
│ ├── color/
│ │ ├── 0.jpg
│ │ └── ...
│ ├── depth/
│ │ ├── 0.png
│ │ └── ...
│ ├── pose/
│ │ ├── 0.txt
│ │ └── ...
│ └── intrinsic/
│ └── intrinsic_depth.txt
Evaluation
First, download our provided checkpoints, and put them at "./checkpoint".
# Evaluate on ScanNet200 (default)
bash scripts/eval.sh
# Evaluate on ScanNet
DATASET=scannet bash scripts/eval.sh
# Example: choose a specific GPU
GPU_ID=1 DATASET=scannet200 bash scripts/eval.sh
Models
We provide the configuration files and checkpoints for the ScanNet and ScanNet200 benchmarks (validation set).
| Dataset | mAP | mAP50 | mAP25 | Download |
|---|---|---|---|---|
| ScanNet (val) | 50.4 | 71.7 | 87.0 | model | config |
| ScanNet200 (val) | 31.9 | 45.7 | 53.7 | model | config |
Citation
If you find this work helpful for your research, please cite:
@article{qu2026segvggt,
title={SegVGGT: Joint 3D Reconstruction and Instance Segmentation from Multi-View Images},
author={Qu, Jinyuan and Li, Hongyang and Zhang, Lei},
journal={arXiv preprint arXiv:2603.19926},
year={2026}
}
Acknowledgement
SegVGGT is built on VGGT. We thank the VGGT contributors for their open research.
License
See the LICENSE file for details about the license under which this code is made available.