DepthWarpVS: Depth-Guided Novel-View Synthesis for Glasses-Free 3D

DepthWarpVS is a real-time warp-and-refine pipeline for synthesizing multiple views from monocular RGB and depth input. It targets glasses-free 3D video communication, where geometry must remain interpretable and completion must be fast enough for interactive use.

Research Snapshot

Item	Summary
Research question	Can explicit depth reprojection and localized completion produce stable novel views more efficiently than regenerating every target view?
My contribution	I designed the end-to-end depth-warping system, the hole/valid/pollute reliability representation, the lightweight MGMI completion network, and the synthetic defect-generation pipeline used to train it.
Main reported result	In the current system measurement on one RTX 4090, MGMI completion takes approximately 5 ms per 4K target view, and the full 11-view prototype operates at approximately 15 FPS.
Research output	Related work was accepted as a co-authored paper at ICDT 2026: DWvs: Depth-Guided Image Warping and Hole Filling for Novel View Synthesis.

The timing result above is a system snapshot from the research prototype. It should not be treated as a cross-hardware benchmark until the public benchmark script, checkpoint, and exact runtime environment are released together.

Research Question

Single-image view synthesis has an unavoidable disocclusion problem: moving the virtual camera reveals pixels that were not visible in the source image. A full generative model can hallucinate every output view, but this is expensive and may change pixels that were already geometrically reliable.

DepthWarpVS tests a more constrained hypothesis:

1. use depth to explicitly project reliable source pixels;
2. detect holes and depth-boundary contamination during projection;
3. apply a lightweight refiner only to unreliable regions;
4. fuse the resulting views for a lenticular or other glasses-free display.

My Contributions

• Designed the depth-guided forward-splatting and visibility pipeline for monocular novel-view synthesis.
• Introduced three explicit reliability channels: hole, valid, and pollute.
• Developed MGMI, a mask-guided lightweight completion network that preserves already valid pixels.
• Built a geometry-driven data generator that creates pseudo holes, boundary contamination, edit masks, and targets from available video data.
• Integrated single-image, folder, and video inference with multi-view fusion and real-time display output.
• Implemented optional person segmentation, parsing, keypoint priors, edge sharpening, mixed precision, chunked rendering, and asynchronous video I/O.

Method

RGB + depth
  -> target-camera forward splatting
  -> visibility and reliability masks
  -> MGMI localized completion
  -> multiple synthesized views
  -> glasses-free 3D interlacing/fusion

The core implementation is organized around:

• main.py: supported end-to-end image/folder/video inference;
• models/splatting/softmax_splat.py: depth-aware splatting;
• models/refiner/MGMI.py: localized completion;
• data/mannequin_refine_dataset.py: refiner training samples;
• scripts/prepare_simwarp_new.py: synthetic warp defects;
• engine/trainer_refiner.py: training and validation loop.

Qualitative Results

These examples are qualitative. The checkpoint, complete benchmark table, and evaluation manifest are not included.

Installation

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

CUDA is recommended for splatting and MGMI inference. torch-scatter is optional; the implementation includes a fallback path.

Inference

Run from the repository root:

python main.py \
  --image /path/to/source.png \
  --depth /path/to/depth.png \
  --refiner_ckpt /path/to/refiner_ema_best.pth \
  --out outputs/multiview.png \
  --num_per_side 5 \
  --max_disp_px 25

Video mode:

python main.py \
  --video /path/to/rgb.mp4 \
  --depth_video /path/to/depth.mp4 \
  --refiner_ckpt /path/to/refiner_ema_best.pth \
  --out outputs/multiview.mp4 \
  --num_per_side 5 \
  --max_disp_px 25 \
  --focus_depth 5.9 \
  --ffmpeg_h264

Use python main.py --help for camera intrinsics, depth decoding, fusion, performance, and encoding options.

Training

The package-style training entry point should be run from the parent directory of this repository:

python -m depth_warp_vs.scripts.prepare_simwarp_new \
  --root /path/to/MannequinChallenge \
  --splits train,validation,test

python -m depth_warp_vs.scripts.train_refiner \
  --config depth_warp_vs/configs/mgmi_refiner_train.yaml

Update the dataset root in the YAML configuration before training. Checkpoints and third-party depth models are intentionally not committed.

Reproducibility

Included:

• source code for splatting, reliability masks, MGMI, training, and fusion;
• qualitative figures and deployment photographs;
• unit-level geometry, warp, loss, splatting, and end-to-end test files;
• deterministic seed fields in the main configurations.

Not included:

• an approved public checkpoint and a checksum;
• exact data split manifests and preprocessing versions;
• PSNR, SSIM, LPIPS, temporal consistency, latency, and memory tables;
• ablations for hole-only versus hole+pollute masks, MGMI, priors, and edge sharpening;
• repeated measurements across at least two GPU classes.

Data and Third-Party Assets

This repository does not redistribute the previously bundled RenderPeople mesh and texture files. RenderPeople's terms restrict redistribution of downloadable source models and require separate permission for computer-vision research. Use only datasets for which you have appropriate research and redistribution rights.

See THIRD_PARTY_NOTICES.md for upstream components and data responsibilities.

License

Original code in this repository is released under the MIT License. Third-party code, models, datasets, and media remain under their respective licenses.