🌌 SpaceEra / SpaceEra++

Conference Paper

Spatial Understanding from Videos: Structured Prompts Meet Simulation Data
NeurIPS 2025 Spotlight

Haoyu Zhang^1,2, Meng Liu^3,4, Zaijing Li^1,2, Haokun Wen¹,
Weili Guan¹, Yaowei Wang^1,2, Liqiang Nie¹

_{¹ Harbin Institute of Technology (Shenzhen) · ² Pengcheng Laboratory
³ Shandong Jianzhu University · ⁴ Zhongguancun Academy}

Journal Paper

SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Video
Accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2026 · Early Access

Weili Guan^1,2, Haoyu Zhang^1,3, Meng Liu^4,5,
Qianlong Xiang^1,6,2, Yaowei Wang^1,4, Liqiang Nie¹

_{¹ Harbin Institute of Technology (Shenzhen) · ² Shenzhen Loop Area Institute
³ Pengcheng Laboratory · ⁴ Shandong University
⁵ Zhongguancun Academy · ⁶ City University of Hong Kong}

SpatialMind · ScanForgeQA · ScenePick · SpaceAlign

Overview · Installation · Training · Inference · Methods · Citation

---

🔭 Overview

This repository contains the code for both versions of SpaceEra:

Version	Venue	Main components	Purpose
SpaceEra	NeurIPS 2025 Spotlight	SpatialMind, ScanForgeQA	Structured inference and synthetic spatial QA data
SpaceEra++	IEEE TPAMI 2026, accepted	ScenePick, SpaceAlign	Informative frame selection and spatially constrained GRPO

Unified SpaceEra++ framework for data construction, frame sampling, training, and inference.

The integrated workflow is:

Training
ScanForgeQA ──► ScenePick ──► Vision-Language Model ──► SpaceAlign GRPO

Inference
Video sample ──► ScenePick ──► SpatialMind ──► Vision-Language Model ──► Answer

Note

The commands below use Qwen/Qwen2.5-VL-7B-Instruct as a reference example. It can be replaced through --model with another Transformers/TRL-compatible vision-language model. Depending on the architecture, adjust the processor/media loader and --lora-target-modules accordingly.

✨ What is included

Component	Stage	Implementation
🧠 SpatialMind	Inference	Scene decomposition and question decomposition prompts
🏗️ ScanForgeQA	Data	Scene construction, scan creation, and spatial QA generation
🎞️ ScenePick	Input	VGGT reconstruction, semantic voxel weighting, and greedy frame selection
🎯 SpaceAlign	Training	Format, task, absolute-coordinate, and relative-relation rewards
🔗 Integrated workflow	End to end	ScanForgeQA adapters, cached preprocessing, GRPO training, and VLM inference

🗂️ Repository structure

.
├── asset/                              # Paper figures
├── examples/
│   ├── scene0277_02.mp4                # Example scanning video
│   ├── scene_spec.json                 # Scene construction example
│   ├── grounding_boxes.json            # ScenePick semantic boxes
│   ├── scanforgeqa_training_sample.jsonl
│   └── spacealign_reward.json
├── scripts/
│   ├── prepare_scanforgeqa.py           # ScanForgeQA -> ScenePick cache
│   ├── train_spacealign.py              # SpaceAlign GRPO training
│   └── infer_spaceera.py                # ScenePick + SpatialMind inference
├── src/spaceera/
│   ├── scanforgeqa/                     # Conference: dataset construction
│   ├── spatialmind/                     # Conference: prompting
│   ├── scenepick/                       # Journal: frame selection
│   ├── spacealign/                      # Journal: spatial rewards
│   ├── workflows/                       # Integrated data and prompt pipeline
│   ├── schemas.py
│   └── cli.py
├── tests/
├── max_coverage_cal_batch.py            # Legacy ScenePick entry point
├── max_coverage_cal.py                  # Legacy alias
└── pyproject.toml

⚙️ Installation

Basic SpaceEra components

python3 -m pip install -e .

Complete training and inference stack

python3 -m pip install -e '.[scenepick,training]'

Python 3.10 or newer is recommended. The complete stack includes PyTorch, Transformers, TRL, PEFT, Datasets, and Qwen vision utilities. It pins trl==0.21.0; PyTorch SDPA is used by default, while Flash Attention remains optional.

ScenePick additionally expects the official VGGT package to be importable as vggt:

git clone https://github.com/facebookresearch/vggt.git
cd vggt
python3 -m pip install -e .

VGGT weights can be provided as a Hugging Face model ID such as facebook/VGGT-1B or as a local checkpoint directory.

Tip

Grounded SAM 2 is optional. Without question-specific grounding boxes, ScenePick runs the spatial-coverage-only variant.

🚀 Training

The training path is deliberately split into a deterministic ScenePick preparation stage and a model-training stage. This prevents VGGT reconstruction from being repeated in every epoch.

1. Prepare ScanForgeQA records

Training accepts JSON or JSONL. A canonical sample looks like:

{
  "sample_id": "scene0001_qa0001",
  "video_path": "videos/scene0001.mp4",
  "question": "What is the distance between the sofa and table?",
  "answer": "2.0 meters",
  "question_type": "absolute_distance",
  "selected_frames": [],
  "ground_truth_coordinates": {
    "sofa": [2, 3],
    "table": [6, 3]
  },
  "category_counts": {
    "sofa": 1,
    "table": 1
  },
  "grounding_boxes": {}
}

Required fields are video_path, question, and answer.

• selected_frames may be empty; ScenePick will populate it.
• ground_truth_coordinates enables the complete SpaceAlign spatial rewards.
• If scene_graph_path is provided instead, 10×10 grid coordinates are derived automatically.
• grounding_boxes enables question-aware semantic weighting.
• Common aliases such as video, source_video, qa_id, problem, and solution are accepted.

Run cached ScenePick preprocessing:

python3 scripts/prepare_scanforgeqa.py \
  --dataset /path/to/scanforgeqa_train.jsonl \
  --video-root /path/to/ScanForgeQA \
  --vggt-model facebook/VGGT-1B \
  --num-frames 32 \
  --max-candidates 64 \
  --output outputs/scanforgeqa_train_scenepick.jsonl

Questions sharing the same video reuse one VGGT reconstruction. Only one video's dense point maps are retained in memory at a time.

2. Train with SpaceAlign GRPO

The following command uses Qwen2.5-VL-7B-Instruct as an example backbone:

accelerate launch scripts/train_spacealign.py \
  --dataset outputs/scanforgeqa_train_scenepick.jsonl \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --output-dir outputs/spacealign-qwen2.5-vl-7b \
  --num-generations 4 \
  --batch-size 1 \
  --gradient-accumulation-steps 8 \
  --dtype bfloat16

The example configuration applies LoRA to q_proj and v_proj. For another backbone:

--model organization/another-vlm \
--lora-target-modules q_proj k_proj v_proj o_proj

SpaceAlign trains the model to return:

<spatial_map>{"object": [x, y]}</spatial_map>
<think>spatial reasoning</think>
<answer>final answer</answer>

The four reward components are logged independently:

Reward	Function
R_format	Checks the required response structure
R_task	Scores multiple-choice, text, or numerical correctness
R_abs	Measures normalized object-coordinate accuracy
R_rel	Measures pairwise direction and distance consistency

An unprepared dataset can also be passed directly to train_spacealign.py; add --vggt-model and ScenePick will run before training.

Validate records without loading model weights:

python3 scripts/train_spacealign.py \
  --dataset outputs/scanforgeqa_train_scenepick.jsonl \
  --dry-run

Important

GRPO generates multiple completions for every visual prompt and is substantially more memory-intensive than SFT. Use an Accelerate/DeepSpeed configuration appropriate for the available GPUs.

🔮 Inference

Inference combines ScenePick-selected frames with a SpatialMind prompt and the chosen vision-language model.

Prepared sample

python3 scripts/infer_spaceera.py \
  --dataset outputs/scanforgeqa_train_scenepick.jsonl \
  --sample-id scene0001_qa0001 \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --adapter outputs/spacealign-qwen2.5-vl-7b \
  --output outputs/prediction.json

Omit --adapter to use the original backbone. Replace --model to evaluate another compatible VLM.

Raw sample without selected frames

Add ScenePick parameters:

python3 scripts/infer_spaceera.py \
  --dataset /path/to/raw_samples.jsonl \
  --sample-id scene0001_qa0001 \
  --vggt-model facebook/VGGT-1B \
  --num-frames 32 \
  --max-candidates 64 \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --output outputs/prediction.json

When scene_graph_path is available, SpatialMind includes its structured scene representation. Otherwise, it instructs the VLM to perform local modeling, coordinate mapping, cognition generation, and question decomposition directly from the selected frames.

Inspect the complete model input without loading VLM weights:

python3 scripts/infer_spaceera.py \
  --dataset outputs/scanforgeqa_train_scenepick.jsonl \
  --sample-id scene0001_qa0001 \
  --dry-run \
  --output outputs/inference_input.json

🧩 Method components

🧠 SpatialMind prompting

SpatialMind decomposes spatial reasoning into two coordinated stages:

1. Scene decomposition: local modeling, coordinate mapping, and cognition generation.
2. Question decomposition: classify the spatial task and construct explicit reasoning steps.

SpatialMind structured prompting strategy.

The implementation supports relative distance, object count, appearance order, relative direction, object size, absolute distance, room size, and route planning.

🏗️ ScanForgeQA construction

ScanForgeQA provides a three-stage synthetic data pipeline:

1. Scene construction from structured room and object specifications.
2. Scan creation using orbit and navigation camera trajectories.
3. QA generation from scene geometry and object annotations.

ScanForgeQA dataset construction pipeline.

Minimal conference-version example:

spaceera build-scene \
  --scene-spec examples/scene_spec.json \
  --output outputs/scene_graph.json

spaceera build-scan \
  --scene-graph outputs/scene_graph.json \
  --output outputs/scan_sequence.json \
  --blender-script outputs/scan_camera.py

spaceera generate-qa \
  --scene-graph outputs/scene_graph.json \
  --video-path /path/to/rendered_scan.mp4 \
  --output outputs/qa_pairs.json

Supplying --video-path writes the video and scene-graph references required by integrated training.

🎞️ ScenePick sampling

ScenePick combines geometric coverage with question-related object semantics:

1. Uniformly pre-sample candidate frames.
2. Reconstruct depth, camera poses, and 3D points with VGGT.
3. Ground question-related objects and assign semantic weights.
4. Voxelize valid 3D points.
5. Greedily maximize weighted coverage.

ScenePick frame sampling strategy.

Run it directly on the included video:

spaceera scenepick \
  --video-dir examples/scene0277_02.mp4 \
  --model facebook/VGGT-1B \
  --num-frames 32 \
  --max-candidates 64 \
  --output outputs/scene0277_02_scenepick.json

For semantic sampling, add:

--grounding-json examples/grounding_boxes.json

Grounding JSON uses {frame_name_or_path: [[x1, y1, x2, y2], ...]}. Without it, all points receive equal weight and the command reproduces the spatial-coverage-only ablation.

🎯 SpaceAlign rewards

Core reward functions are available from spaceera.spacealign, while TRL-compatible batch adapters live in spaceera.spacealign.trl_rewards.

Evaluate the standalone example:

spaceera spacealign-reward \
  --input examples/spacealign_reward.json \
  --output outputs/reward.json

🧰 Utility commands

• reason_steps.py: inspect question decomposition.
• gen_scene_exp.py: inspect scene decomposition.
• nav_script.py: export an example Blender camera path.
• max_coverage_cal_batch.py: backward-compatible ScenePick entry point.

📚 Citation

Conference version

@inproceedings{NEURIPS2025_9541101f,
  author    = {Zhang, Haoyu and Liu, Meng and Li, Zaijing and Wen, Haokun and Guan, Weili and Wang, Yaowei and Nie, Liqiang},
  booktitle = {Advances in Neural Information Processing Systems},
  editor    = {D. Belgrave and C. Zhang and H. Lin and R. Pascanu and P. Koniusz and M. Ghassemi and N. Chen},
  pages     = {103202--103229},
  publisher = {Curran Associates, Inc.},
  title     = {Spatial Understanding from Videos: Structured Prompts Meet Simulation Data},
  url       = {https://proceedings.neurips.cc/paper_files/paper/2025/file/9541101fbc2f24bce5f2462b95db88c4-Paper-Conference.pdf},
  volume    = {38},
  year      = {2025}
}

Journal extension

@article{guan2026spaceerapp,
  author  = {Guan, Weili and Zhang, Haoyu and Liu, Meng and Xiang, Qianlong and Wang, Yaowei and Nie, Liqiang},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  title   = {SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Video},
  year    = {2026},
  pages   = {1--15},
  doi     = {10.1109/TPAMI.2026.3692302}
}

🙏 Acknowledgements

We thank the authors of VSI-Bench, VGGT, Grounded SAM 2, Blender, Hugging Face Transformers/TRL, and LLaMA-Factory for their released tools and resources.