|
|
基于深度学习的点云上采样算法研究
|
Abstract:
点云上采样能够提高点云分辨率并保持点云的特征，近年来越来越受到人们的重视。基于深度学习的点云上采样算法相较于基于优化的算法，能够更有效地学习点云的特征和结构，且对数据的先验要求不高，取得了先进的上采样效果。因此基于深度学习的点云上采样是当前许多学者主要研究的方向之一。本文综述了基于深度学习的点云上采样算法，阐述了点云上采样的整体框架以及改进的策略，并介绍了点云上采样效果的评价指标以及常用的数据集，最后探讨了点云上采样的未来的几个极具潜力的发展方向。
Point cloud upsampling improves the resolution of point cloud and maintains the feature of point cloud, which has attracted more and more attention in recent years. Compared with the optimization-based algorithms, the point cloud upsampling algorithms based on deep learning can more effectively learn the feature and structure of the point cloud and have low prior requirements for data, leading to the advanced effect of upsampling. Therefore, the point cloud upsampling based on deep learning is one of the main research directions of many scholars at present. In this paper, we summarize the point cloud upsampling algorithms based on deep learning and expound the holistic framework and improved strategies of point cloud upsampling. Then the evaluation metrics of point cloud upsampling effect and commonly used data sets are introduced. We finally discuss several potential development directions of point cloud upsampling in the future.
| [1] | Hoppe, H., DeRose, T., Duchamp, T., et al. (1992) Surface Reconstruction from Unorganized Points. Proceedings of the 19th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), Chicago, 26-31 July 1992, 71-78. https://doi.org/10.1145/133994.134011 |
| [2] | Kazhdan, M.M. and Hoppe, H. (2013) Screened Poisson Surface Reconstruction. ACM Transactions on Graphics (TOG), 32, 29:1-29:13. https://doi.org/10.1145/2487228.2487237 |
| [3] | Newcombe, R.A., Izadi, S., Hilliges, O., et al. (2011) KinectFusion: Real-Time Dense Surface Mapping and Tracking. IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Basel, 26-29 October 2011, 127-136.
https://doi.org/10.1109/ISMAR.2011.6092378 |
| [4] | Riegler, G., Ulusoy, A.O., Bischof, H. and Geiger, A. (2017) Octnetfusion: Learning Depth Fusion from Data. IEEE International Conference on 3D Vision (3DV), Qingdao, 10-12 October 2017, 57-66.
https://doi.org/10.1109/3DV.2017.00017 |
| [5] | Lang, A.H., Vora, S., Caesar, H., et al. (2019) Pointpillars: Fast Encoders for Object Detection from Point Clouds. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, 16-20 June 2019, 12697- 12705. https://doi.org/10.1109/CVPR.2019.01298 |
| [6] | Wang, Y., Chao, W.-L., Garg, D., et al. (2019) Pseudolidar from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, 16-20 June 2019, 8445-8453. https://doi.org/10.1109/CVPR.2019.00864 |
| [7] | Qi, C.R., Su, H., Mo, K. and Guibas, L.J. (2017) PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, 21-26 July 2017, 77-85. |
| [8] | Qi, C.R., Yi, L., Su, H. and Guibas, L.J. (2017) PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. International Conference on Neural Information Processing Systems (NIPS), Long Beach, 4-9 December 2017, 5099-5108. |
| [9] | Yu, L.Q., Li, X.Z., et al. (2018) Pu-Net: Point Cloud Upsampling Network. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, 18-23 June 2018, 2790-2799. |
| [10] | Yu, L., Li, X.Z., Fu, C.W., et al. (2018) Ec-net: An Edge-Aware Point Set Consolidation Network. ECCV 15th European Conference, Munich, 8-14 September 2018, 386-402. |
| [11] | Qian, Y., Hou, J., Kwong, S. and He, Y. (2020) Pugeo-net: A Geometry-Centric Network for 3D Point Cloud Upsampling. 16th European Conference, Glasgow, 23-28 August 2020, 752-769.
https://doi.org/10.1007/978-3-030-58529-7_44 |
| [12] | Sharma, R., Schwandt, T., Kunert, C., Urban, S. and Broll, W. (2021) Point Cloud Upsampling and Normal Estimation Using Deep Learning for Robust Surface Reconstruction. Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021), Volume 5, 70-79.
https://doi.org/10.5220/0010211600700079 |
| [13] | Qian, G.C., Abualshour, A., Li, G.H., et al. (2021) Pu-gcn: Point Cloud Upsampling Using Graph Convolutional Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, 19-25 June 2021, 11683-11692. https://doi.org/10.1109/CVPR46437.2021.01151 |
| [14] | Long, C., Zhang, W., Li, R., Wang, H., Dong, Z. and Yang, B. (2022) PC2-PU: Patch Correlation and Point Correlation for Effective Point Cloud Upsampling. Proceedings of the 30th ACM International Conference on Multimedia, Lisbon, 10-14 October 2022, 2191-2201. |
| [15] | 钟帆, 柏正尧. 采用动态残差图卷积的3D点云超分辨率[J]. 浙江大学学报(工学版), 2022, 56(11): 2251-2259. |
| [16] | Gu, F., Zhang, C.L., Wang, H.Y., et al. (2022) PU-WGCN: Point Cloud Upsampling Using Weighted Graph Convolutional Networks. Remote Sensing, 14, 5356. https://doi.org/10.3390/rs14215356 |
| [17] | Wang, Y.F., Wu, S.H., Huang, H., et al. (2019) Patch-Based Progressive 3d Point Set Upsampling. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, 16-20 June 2019, 5951-5960.
https://doi.org/10.1109/CVPR.2019.00611 |
| [18] | Li, R.H., Li, X.Z., Heng, P.-A. and Fu, C.-W. (2021) Point Cloud Upsampling via Disentangled Refinement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, 20-25 June 2021, 344-353. |
| [19] | Du, H., Yan, X., Wang, J., Xie, D. and Pu, S. (2022) Point Cloud Upsampling via Cascaded Refinement Network. |
| [20] | Li, R.H., Li, X.Z., Fu, C.W., et al. (2019) Pu-gan: A Point Cloud Upsampling Adversarial Network. IEEE International Conference on Computer Vision (ICCV), Seoul, 27 October-2 November 2019, 7202-7211. |
| [21] | Wu, H.K., Zhang, J.G. and Huang, K.Q. (2020) Point Cloud Super Resolution with Adversarial Residual Graph Networks. British Machine Vision Conference (BMVC), Manchester, 7-11 September 2020, 256-267. |
| [22] | Zhou, K., Dong, M. and Arslanturk, S. (2022) “Zero-Shot” Point Cloud Upsampling. 2022 IEEE International Conference on Multimedia and Expo (ICME), Taipei, 11-15 July 2022, 1-6.
https://doi.org/10.1109/ICME52920.2022.9859662 |
| [23] | Ye, S., Chen, D., Han, S., Wan, Z. and Liao, J. (2021) Meta-PU: An Arbitrary-Scale Upsampling Network for Point Cloud. IEEE Transactions on Visualization and Computer Graphics, 28, 3206-3218.
https://doi.org/10.1109/TVCG.2021.3058311 |
| [24] | Feng, W., Li, J., Cai, H., Luo, X. and Zhang, J. (2022) Neural Points: Point Cloud Representation with Neural Fields for Arbitrary Upsampling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, 19-24 June 2022, 18633-18642. https://doi.org/10.1109/CVPR52688.2022.01808 |
| [25] | Liu, X., Han, Z., Wen, X., et al. (2019) L2G Auto-Encoder: Understanding Point Clouds by Local-to-Global Reconstruction with Hierarchical Self-Attention. Proceedings of the ACM International Conference on Multimedia, Nice, 21-25 October 2019, 989-997. https://doi.org/10.1145/3343031.3350960 |
| [26] | Liu, X., Liu, X., Liu, Y.S. and Han, Z. (2022) Spu-net: Self-Supervised Point Cloud Upsampling by Coarse-to-Fine Reconstruction with Self-Projection Optimization. IEEE Transactions on Image Processing, 31, 4213-4226.
https://doi.org/10.1109/TIP.2022.3182266 |
| [27] | Zhao, W., Liu, X., Zhong, Z., Jiang, J., Gao, W., Li, G. and Ji, X. (2022) Self-Supervised Arbitrary-Scale Point Clouds Upsampling via Implicit Neural Representation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, 19-24 June 2022, 1999-2007. https://doi.org/10.1109/CVPR52688.2022.00204 |
| [28] | Qian, Y., Hou, J. and Kwong, S. (2021) Deep Magnification-Flexible Upsampling over 3D Point Clouds. IEEE Transactions on Image Processing, 30, 8354-8367. https://doi.org/10.1109/TIP.2021.3115385 |
| [29] | Mao, A., Du, Z., Hou, J., Duan, Y., Liu, Y.J. and He, Y. (2022) PU-Flow: A Point Cloud Upsampling Network with Normalizing Flows. IEEE Transactions on Visualization and Computer Graphics, 1-14.
https://doi.org/10.1109/TVCG.2022.3196334 |
| [30] | Nguyen, T., Pham, Q.H., Le, T., Pham, T., Ho, N. and Hua, B.S. (2021) Point-Set Distances for Learning Representations of 3D Point Clouds. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, 10-17 October 2021, 10478-10487. https://doi.org/10.1109/ICCV48922.2021.01031 |
| [31] | Jesorsky, O., Klaus, J.K. and Robert, W.F. (2001) Robust Face Detection Using the Hausdorff Distance. International Conference on Audio- and Video-Based Biometric Person Authentication, Heidelberg, 6-8 June 2001, 90-95.
https://doi.org/10.1007/3-540-45344-X_14 |
| [32] | Fan, H.Q., Su, H. and Guibas, L.J. (2017) A Point Set Generation Network for 3D Object Reconstruction from a Single Image. CVPR 2017, Honolulu, 21-26 July 2017, 605-613. |
| [33] | Sokolova, M., Japkowicz, N. and Szpakowicz, S. (2006) Beyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation. 19th Australian Joint Conference on Artificial Intelligence, Hobart, 4-8 December 2006, 1015-1021. https://doi.org/10.1007/11941439_114 |
| [34] | Achlioptas, P., Diamanti, O., Mitliagkas, I. and Guibas, L. (2018) Learning Representations and Generative Models for 3d Point Clouds. 35th International Conference on Machine Learning (ICML), Stockholm, 10-15 July 2018, 40-49. |
| [35] | Wu, Z., Song, S., Khosla, A., et al. (2015) 3D ShapeNets: A Deep Representation for Volumetric Shapes. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, 7-12 June 2015, 1912-1920. |
| [36] | Chang, A.X., Funkhouser, T., Guibas, L.J., et al. (2015) ShapeNet: An Information-Rich 3D Model Repository. |
| [37] | Lian, Z., Zhang, J., Choi, S., et al. (2015) Non-Rigid 3D Shape Retrieval. Proceedings of the 2015 Eurographics Workshop on 3D Object Retrieval, Zurich, 2-3 May 2015, 107-120. |
| [38] | Sketchfab. https://sketchfab.com |
| [39] | Yang, F.Z., Yang, H., Fu, J.L., et al. (2020) Learning Texture Transformer Network for Image Super-Resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, 13-19 June 2020, 5791- 5800. https://doi.org/10.1109/CVPR42600.2020.00583 |
| [40] | Chen, H.T., Wang, Y.H., Guo, T.Y., et al. (2021) Pre-Trained Image Processing Transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, 20-25 June 2021, 12299-12310.
https://doi.org/10.1109/CVPR46437.2021.01212 |
| [41] | Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2020) An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale. |
| [42] | Liu, Z., Lin, Y.T., Cao, Y., et al. (2021) Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, 10-17 October 2021, 10012-10022. https://doi.org/10.1109/ICCV48922.2021.00986 |
| [43] | Carion, N., Massa, F., Synnaeve, G., et al. (2020) End-to-End Object Detection with Transformers. In: European Conference on Computer Vision, Springer, Berlin, 213-229. https://doi.org/10.1007/978-3-030-58452-8_13 |
| [44] | Zhu, X.Z., Su, W.J., Lu, L.W., et al. (2020) Deformable DETR: Deformable Transformers for End-to-End Object Detection. |
| [45] | Qiu, S., Anwar, S. and Barnes, N. (2021) Pu-Transformer: Point Cloud Upsampling Transformer. Proceedings of the Asian Conference on Computer Vision, Macau, 4-8 December 2022, 2475-2493. |
