|
|
图像语义分割方法研究进展
|
Abstract:
图像语义分割任务是将图像的每个像素分类到每一个实例中，每个实例对应一个类。该任务是场景理解概念的一部分，由于深度学习的图像语义分割方法能更好地解释图像的全局上下文，越来越受到计算机视觉和机器学习研究者的关注，并广泛应用于室内导航、自动驾驶，甚至虚拟或增强现实系统等领域。本文介绍图像语义分割的术语的概念，回顾传统和现有的深度学习方法，强调了它们在该领域的贡献和意义，以及语义分割算法的评价指标与常用数据集，最后，我们对当前语义图像分割任务中存在的一些问题进行讨论，并提出相关解决方法和研究展望。
The semantic image segmentation task consists of classifying each pixel of an image into an instance, where each instance corresponds to a class. This task is a part of the concept of scene understanding or better explaining the global context of an image. Image semantic segmentation is more and more being of interest for computer vision and machine learning researchers. Many applications on the rise need accurate and efficient segmentation mechanisms: indoor navigation, autonomous driving, and even virtual or augmented reality systems to name a few. This paper provides the concept of image semantic segmentation terms, and reviews traditional and existing deep learning methods with emphasizing their contributions and significance in this field, as well as the evaluation indicators of semantic segmentation algorithms and commonly used datasets. At last, we discuss some problems in current semantic image segmentation tasks, and propose relevant solutions and research prospects.
| [1] | Lu, X., Wang, W., Shen, J., et al. (2022) Zero-Shot Video Object Segmentation with Co-Attention Siamese Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44, 2228-2242. |
| [2] | Batra, A., Singh, S., Pang, G., et al. (2019) Improved Road Connectivity by Joint Learning of Orientation and Segmentation. 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, 16-20 June 2019, 10377- 10385. https://doi.org/10.1109/CVPR.2019.01063 |
| [3] | Sehar, U. and Naseem, M.L. (2022) How Deep Learning Is Empowering Semantic Segmentation: Traditional and Deep Learning Techniques for Semantic Segmentation: A Comparison. Multimedia Tools and Applications, 81, 30519-30544.
https://doi.org/10.1007/s11042-022-12821-3 |
| [4] | Sharif Razavian, A., Azizpour, H., Sullivan, J., et al. (2014) CNN Features Off-the-Shelf: An Astounding Baseline for Recognition. 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, 23-28 June 2014, 806-813.
https://doi.org/10.1109/CVPRW.2014.131 |
| [5] | Long, J., Shelhamer, E. and Darrell, T. (2015) Fully Convolutional Net-Works for Semantic Segmentation. 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, 7-12 June 2015, 3431-3440.
https://doi.org/10.1109/CVPR.2015.7298965 |
| [6] | Friedman, N., Geiger, D. and Goldszmidt, M. (1997) Bayesian Network Classifiers. Machine Learning, 29, 131-163.
https://doi.org/10.1023/A:1007465528199 |
| [7] | Liu, W., Rabinovich, A. and Berg, A.C. (2015) ParseNet: Looking Wider to See Better. |
| [8] | Chen, L.-C., Papandreou, G., et al. (2014) Semantic Image Segmentation with Deep Convolutionalnets and Fully Connected CRFs. |
| [9] | Schwing, A.G. and Urtasun, R. (2015) Fully Connected Deep Structured Networks. |
| [10] | Lin, G., Shen, C., et al. (2016) Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, 27-30 June 2016, 3194-3203.
https://doi.org/10.1109/CVPR.2016.348 |
| [11] | Noh, H., Hong, S. and Han, B. (2015) Learning Deconvolution Network for Semantic Segmentation. 2015 IEEE International Conference on Computer Vision, Santiago, 7-13 December 2015, 1520-1528.
https://doi.org/10.1109/ICCV.2015.178 |
| [12] | Badrinarayanan, V., Kendall, A. and Cipolla, R. (2017) SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39, 2481-2495.
https://doi.org/10.1109/TPAMI.2016.2644615 |
| [13] | Fu, J., Liu, J., et al. (2019) Stacked Deconvolutional Network for Semantic Segmentation. IEEE Transactions on Image Processing. https://doi.org/10.1109/TIP.2019.2895460 |
| [14] | Ronneberger, O., Fischer, P. and Brox, T. (2015) U-Net: Convolutional Networks for Biomedical Image Segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, Berlin, 234- 241. https://doi.org/10.1007/978-3-319-24574-4_28 |
| [15] | Milletari, F., Navab, N. and Ahmadi, S.-A. (2016) V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. International Conference on 3D Vision IEEE, Stanford, 25-28 October 2016, 565-571.
https://doi.org/10.1109/3DV.2016.79 |
| [16] | Lin, T.-Y., Dollar, P., et al. (2017) Feature Pyramid Networks for Object Detection. 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, 21-26 July 2017, 2117-2125. https://doi.org/10.1109/CVPR.2017.106 |
| [17] | Zhao, H., Shi, J., et al. (2017) Pyramid Scene Parsing Network. 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, 21-26 July 2017, 2881-2890. https://doi.org/10.1109/CVPR.2017.660 |
| [18] | Ren, S., He, K., et al. (2015) Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NIPS’15: Proceedings of the 28th International Conference on Neural Information Processing Systems, Volume 1, 91-99. |
| [19] | He, K., Gkioxari, Dollar, G.P. and Girshick, R. (2017) Mask R-CNN. 2017 IEEE International Conference on Computer Vision, Venice, 22-29 October 2017, 2961-2969. https://doi.org/10.1109/ICCV.2017.322 |
| [20] | Liu, S., Qi, L., et al. (2018) Path Aggregation Network for Instance Segmentation. 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, 18-22 June 2018, 8759-8768.
https://doi.org/10.1109/CVPR.2018.00913 |
| [21] | Chen, L.-C., Hermans, A., et al. (2018) Masklab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features. 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, 18-22 June 2018, 4013-4022. https://doi.org/10.1109/CVPR.2018.00422 |
| [22] | Lee, Y. and Park, J. (2020) CenterMask: Real-Time Anchor-Free Instance Segmentation. 2020 IEEE Conference on Computer Vision and Pattern Recognition, Seattle, 13-19 June 2020, 13906-13915.
https://doi.org/10.1109/CVPR42600.2020.01392 |
| [23] | Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K. and Yuille, A.L. (2018) DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40, 834-848. https://doi.org/10.1109/TPAMI.2017.2699184 |
| [24] | Chen, L.-C., Papandreou, G., Schroff, F. and Adam, H. (2018) Rethinking Atrous Convolution for Semantic Image Segmentation. |
| [25] | Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F. and Adam, H. (2018) Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. 2018 European Conference on Computer Vision, Munich, 8-14 September 2018, 801-818. https://doi.org/10.1007/978-3-030-01234-2_49 |
| [26] | Visin, F., Kastner, K., et al. (2015) ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks. |
| [27] | Byeon, W., Breuel, T.M., et al. (2015) Scene Labeling with LSTM Recurrent Neural Networks. 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, 7-12 June 2015, 3547-3555.
https://doi.org/10.1109/CVPR.2015.7298977 |
| [28] | Xiang, Y. and Fox, D. (2017) DA-RNN: Semantic Mapping with Data Associated Recurrent Neural Networks.
https://doi.org/10.15607/RSS.2017.XIII.013 |
| [29] | Chen, L.-C., Yang, Y., et al. (2016) Attention to Scale: Scale-Aware Semantic Image Segmentation. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, 27-30 June 2016, 3640-3649.
https://doi.org/10.1109/CVPR.2016.396 |
| [30] | Fu, J., Liu, J., et al. (2019) Dual Attention Network for Scene Segmentation. 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, 15-20 June 2019, 3146-3154. https://doi.org/10.1109/CVPR.2019.00326 |
| [31] | Choi, S., Kim, J.T. and Choo, J. (2020) Cars Can’t Fly Up in the Sky: Improving Urban-Scene Segmentation via Height-Driven Attention Networks. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, 13-19 June 2020, 9373-9383. https://doi.org/10.1109/CVPR42600.2020.00939 |
| [32] | Zhang, H., Wu, C., et al. (2020) Resnest: Split-Attention Networks. |
| [33] | Luc, P., Couprie, C., et al. (2016) Semantic Segmentation Using Adversarial Networks. |
| [34] | Hung, W.-C., Tsai, Y.-H., et al. (2018) Adversarial Learning for Semi-Supervised Semantic Segmentation. |
| [35] | Xue, Y., Xu, T., et al. (2018) SegAN: Adversarial Network with Multi-Scale L1 Loss for Medical Image Segmentation. Neuroinformatics, 16, 383-392. https://doi.org/10.1007/s12021-018-9377-x |
| [36] | Xu, J.J., Zhang, Z.X. and Hu, X.L. (2022) Extracting Semantic Knowledge from GANs with Unsupervised Learning. |
| [37] | Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., et al. (2017) A Review on Deep Learning Techniques Applied to Semantic Segmentation. |
| [38] | Xu, K., Wen, L., et al. (2019) Spatiotemporal CNN for Video Object Segmentation. 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, 15-20 June 2019, 1379-1388.
https://doi.org/10.1109/CVPR.2019.00147 |
| [39] | Cao, Z., Hidalgo, G., Simon, T., et al. (2018) OpenPose: Real-Time Multi-Person 2D Pose Estimation Using Part Affinity Fields. |
| [40] | Zhu, Y., Zhou, Y., Xu, H., et al. (2019) Learning Instance Activation Maps for Weakly Supervised Instance Segmentation. 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, 15-20 June 2019, 3111-3120.
https://doi.org/10.1109/CVPR.2019.00323 |
| [41] | Chen, K., et al. (2019) Hybrid Task Cascade for Instance Segmentation. 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, 15-20 June 2019, 4969-4978. https://doi.org/10.1109/CVPR.2019.00511 |
| [42] | Zhuang, P., Wang, Y. and Qiao, Y. (2020) Learning Attentive Pairwise Interaction for Fine-Grained Classification. 2020 AAAI Conference on Artificial Intelligence, New York, 7-12 February 2020, 13130-13137.
https://doi.org/10.1609/aaai.v34i07.7016 |
| [43] | He, J., Deng, Z., Zhou, L., et al. (2019) Adaptive Pyramid Context Network for Semantic Segmentation. 2019 IEEE/ CVF Conference on Computer Vision and Pattern Recognition, Long Beach, 15-20 June 2019, 7511-7520.
https://doi.org/10.1109/CVPR.2019.00770 |
| [44] | Liu, J., He, J., et al. (2020) Learning to Predict Context-Adaptive Convolution for Semantic Segmentation. 2020 European Conference on Computer Vision, Glasgow, 23-28 August 2020, 769-786.
https://doi.org/10.1007/978-3-030-58595-2_46 |
| [45] | Ke, L., Tai, Y.W. and Tang, C.K. (2021) Deep Occlusion-Aware Instance Segmentation with Overlapping Bi-Layers. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, 20-25 June 2021, 4018-4027.
https://doi.org/10.1109/CVPR46437.2021.00401 |
