Unsupervised Learning of Affinity for Image Segmentation: An Inpainting based Approach

Unsupervised image segmentation is a fundamental problem that aims at clustering pixels into regions satisfying homogeneity (w.r.t color or texture), with no pixel annotations specified as ground truth. This paper proposes a novel unsupervised graph based segmentation framework based on a novel affinity matrix. The main hypothesis is that a convolutional neural network (CNN) trained to generate the contents of arbitrary missing image regions can provide powerful grouping cues among pixels in a given image. We encode this intuition with the generation of activation maps from state-of-the-art inpainting network, where the missing region corresponds to a superpixel of the input image. The activation weights are obtained for each pixel of every superpixel region and are utilised in graph construction over these superpixels. In contrast to the traditional graph based methods, that are primarily dependent on integrating local grouping cues, we propose to learn affinities based on high feature descriptiveness of CNNs, thus enabling reliable graph building for segmentation. Experimental results on the Berkeley segmentation database demonstrates the effectiveness of the proposed framework.