Superpixel Sampling Networks

 

Varun Jampani           Deqing Sun           Ming-Yu Liu           Ming-Hsuan Yang           Jan Kautz

 

Abstract

Superpixels provide an efficient low/mid-level representation of image data, which greatly reduces the number of image primitives for subsequent vision tasks. While various superpixel computation models exist, they are not differentiable, making them difficult to integrate into otherwise end-to-end trainable deep neural networks. In this work, we develop a new differentiable model for superpixel sampling that better leverages deep networks for learning superpixel segmentation. The resulting “Superpixel Sampling Network” (SSN) is end-to-end trainable, which allows learning task-specific superpixels with flexible loss functions and has fast runtime. Extensive experimental analysis indicates that SSNs not only outperforms existing superpixel algorithms on traditional segmentation benchmarks, but can also learns superpixels for other tasks. In addition, SSNs can be easily integrated into downstream deep networks resulting in performance improvements.


Overview of Superpixel Sampling Networks. A given image is first passed onto a deep network that extracts features at each pixel, which are then used by differentiable SLIC to generate the superpixels. Shown here are a couple of example SSN generated task-specific superpixels for semantic segmentation and optical flow.

Paper

Please consider citing if you make use of this work and/or the corresponding code:

@inproceedings{jampani:ssn:2018,
	title = {Superpixel Sampling Networks},
	author = {Jampani, Varun and Sun, Deqing and Liu, Ming-Yu and Yang, Ming-Hsuan and Kautz, Jan},
	booktitle = {European Conference on Computer Vision (ECCV)},
	month = September,
	year = {2018}
}

Code

We integrated superpixel sampling networks into Caffe neural network framework. Code is available online in ths github repository: https://github.com/NVlabs/ssn_superpixels.

Video