FocalNet Sky Matting

FocalMatter Paper |

PyTorch implementation of a FocalNet-based sky matting and dynamic sky replacement pipeline.

This project uses a FocalMatter model for sky matte prediction and a video compositing pipeline for replacing the sky region with a new background. The model combines a FocalNet encoder with a compact decoder, top-level context aggregation, gated skip fusion, and focal refinement blocks.

Video

Requirements

See Requirements.txt.

Recommended environment:

python -m venv .venv
.venv\Scripts\activate
pip install -r Requirements.txt

Install a CUDA-enabled PyTorch build if you want GPU inference or training. The pipeline falls back to CPU through torch.device("cuda:0" if torch.cuda.is_available() else "cpu"), but video inference is much slower on CPU.

Setup

1. Clone this repository:

git clone <this-repository-url>
cd FocalNet-SkyMatting

2. Place the trained checkpoint files in checkpoints/.

Model file	Download link	Purpose
Pretrained checkpoint	Google Drive (519MB)	Trained FocalMatter weights for evaluation/inference.
FocalMatter checkpoint [fine-tunning]	Google Drive (173MB)	Trained FocalMatter weights for evaluation/inference.
FocalNet-Tiny SRF pretrained encoder [training from scratch]	Microsoft FocalNet GitHub release (109MB)	Initial ImageNet pretrained encoder weights.

For the current inference pipeline, video.py points to:

checkpoints/best_ckpt.pt

Run Demo Inference

Edit the SETTINGS object in video.py, then run:

python video.py

The pipeline writes:

demo.avi
demo-cat.avi

demo-cat.avi concatenates the input frame and composite result side by side.

Configure Your Own Run

All video inference parameters live in video.py. The most commonly edited fields are:

• input_clip: input image or video path.
• replacement_folder and replacement_name: replacement sky/background asset.
• model_weights_folder and model_weights_file: checkpoint location.
• network_frame_width and network_frame_height: neural network input size.
• render_width and render_height: output render size.
• replacement_center_fraction: crop/tiling scale for replacement backgrounds.
• use_auto_lighting, lighting_blend_strength, and color_blend_strength: blending controls.

Train The Matting Model

The current training loader expects the public sky-mask dataset:

with image and matte folders such as:

images/train
images/val
density_estimation+guided_filter
guided_filter

Update root_dir in utils.py if your dataset is stored elsewhere.

Dataset source: public sky-mask dataset download.

Run training:

python train.py 
  --dataset cvprw2020-ade20K-defg 
  --checkpoint_dir checkpoints 
  --vis_dir val_out 
  --in_size 384 
  --max_num_epochs 200 
  --lr 1e-4 
  --batch_size 8

FocalMatter Architecture

The FocalMatter model in model.py uses:

• FocalNet encoder with channel pyramid [96, 192, 384, 768].
• Lightweight top context aggregation with depthwise dilated convolutions.
• Skip-align gates to align and reweight encoder features before decoder fusion.
• Focal refinement blocks in the decoder.
• A small neck and alpha head for full-resolution matte prediction.

Note: the current runnable video pipeline forwards RGB frames through the model. The paper draft discusses an RGB+trimap formulation.

Limitations

• Sky matte prediction is most reliable on outdoor videos with visible sky.
• Night scenes, skyless frames, and flat textureless skies may produce weak masks or unstable background motion.
• Background stabilization depends on enough reliable tracked features in the replacement region.
• The current training loader contains a local dataset path that should be edited for a new machine.

Citation

@inproceedings{yang2022focalnet,
    title={Focal Modulation Networks},
    author={Yang, Jianwei and Li, Chunyuan and Zhang, Pengchuan and Wang, Xiaodong and Gao, Jianfeng and others},
    booktitle={NeurIPS},
    year={2022}
}