Unsupervised Spatiotemporal Data Inpainting
Reconstructing missing regions in video sequences without supervision
Implementation of Yin, Pajot, de Bézenac & Gallinari, “Unsupervised Spatiotemporal Data Inpainting”
Abstract
Inpainting spatio-temporal sequences is an active research topic that usually relies on supervision with large datasets. We reimplement an unsupervised approach that reconstructs missing regions in video sequences by training a generative model directly on occluded sequences.
A ResNet-type generator of 3D residual blocks with spatial self-attention completes each sequence, and only fills missing values while keeping observed pixels intact. Adversarial training with PatchGAN discriminators on individual frames and on whole sequences captures both per-frame detail and temporal coherence.
We train on geophysical data (sea surface temperature with cloud occlusions) and natural videos (FaceForensics++, KTH, BAIR) with raindrop, moving-bar and removed-pixel occlusions, evaluating with FID, FVD and mean absolute error.
Keywords: video inpainting · unsupervised learning · GAN · self-attention · PatchGAN · spatio-temporal data
Method
Let x and y be the true unknown and the occluded sequences. The goal is the most likely sequence given the observation, which splits into a prior on x and the likelihood of y given x.
- Prior. A generator G maps occluded sequences to completed ones; a random occlusion F with mask m̂ ~ pM re-masks the output, and adversarial training makes it indistinguishable from real occluded data.
- Likelihood. The generator only fills missing values and leaves observed pixels intact: G(y) = Gold(y) ⊙ m̄ + y ⊙ m.
- Generator. A ResNet-type network of 3D ResNet blocks followed by a spatial self-attention layer, encoding the masked sequence and decoding the reconstruction.
- Discriminators. PatchGAN discriminators run on individual frames (Df) and on whole sequences (Ds), classifying each N × N patch as real or fake.
Two supervised variants serve as references: a paired variant with corrupted–uncorrupted pairs (y, x), and an unpaired variant built from separate samples of each.
Data
Table 1: Datasets and occlusions
| Dataset | Type | Content | Occlusions |
|---|---|---|---|
| SST | Geophysical | Sea surface temperature | Clouds |
| FaceForensics++ | Natural video | 1,000 videos of faces on a static background | Raindrops, moving bar, removed pixels |
| KTH | Natural video | 2,391 clips of 6 human actions | Raindrops, moving bar, removed pixels |
| BAIR Robot Pushing | Natural video | 44,374 videos from a one-armed robot | Raindrops, moving bar, removed pixels |
Results
Frames are evaluated with the Fréchet Inception Distance (FID, Inception features trained on ImageNet), sequences with the Fréchet Video Distance (FVD, I3D features trained on Kinetics-400), and reconstruction error with the mean absolute error.
Key findings
- Hard problem: inpainting videos in a fully unsupervised context is a hard task and an active research area.
- Training cost: training is very sensitive, not stable, and GPU- and time-consuming.
- Frames and sequences: separate frame and sequence discriminators let the model capture both per-frame detail and temporal coherence.
Code & references
Code on GitHub (training and FID/FVD scripts) · Project poster (PDF) · Original paper on OpenReview.
