Efficient piecewise training of deep structured models for semantic segmentation

Recent advances in semantic image segmentation have mostly been achieved by training deep convolutional neural networks (CNNs) for the task. We show how to improve semantic segmentation through the use of contextual information, by combining the strengths of deep CNNs to learn powerful feature representations, with Conditional Random Fields (CRFs) which can capture contextual relation modeling. Unlike previous work, our formulation of "deep CRFs" learns both unary {\em and} pairwise terms using multi-scale fully convolutional neural networks (FCNNs) in an end-to-end fashion, which enables us to model complex spatial relations between image regions. A naive method for training such an approach would rely on direct likelihood maximization of the CRF, but this would require expensive inference at each stochastic gradient decent iteration, rendering the approach computationally unviable. We propose a novel method for efficient joint training of the deep structured model based on piecewise training. This approximate training method avoids repeated inference, and so is computationally tractable. We also demonstrate that it yields results that are competitive with the state-of-art in semantic segmentation for the PASCAL VOC 2012 dataset. In particular, we achieve an intersection-over-union score of $70.7$ on its test set, which outperforms state-of-the-art results that make use of the same size training set, thus demonstrating the value of our deep, multi-scale approach to modelling contextual relations.