|
|
点云上采样技术研究
|
Abstract:
点云上采样旨在将稀疏、嘈杂、不完整的点云转换为密集、干净、完整的点云，这样有利于提高下游任务的性能。当前的点云上采样方法主要分为基于优化和基于学习的方法，本文针对基于深度学习的点云上采样算法进行了综述。本文从点云上采样的开山之作PU-Net引入，阐述了上采样算法的发展，并总结了插值算法在点云上采样中的应用，然后对不同特征扩展方法进行了比较，介绍了上采样算法的评价指标和常用数据集，最后对点云上采样的发展前景进行了展望。
Point cloud upsampling aims to convert sparse, noisy, and incomplete point clouds into dense, clean, and complete point clouds, which is conducive to improving the performance of downstream tasks. Current point cloud upsampling algorithms are mainly classified into optimization-based and learning-based methods, and this paper provides a review of deep learning-based point cloud up-sampling algorithms. This paper introduces PU-Net, the pioneer of point cloud upsampling. Then it describes the development of upsampling algorithms and summarizes the application of interpolation algorithms in point cloud upsampling. Then it compares the different feature expansion meth-ods, introduces the evaluation indexes of upsampling algorithms and the commonly used datasets. Finally, it looks forward to the development prospects of point cloud upsampling.
| [1] | Cai, Q., Pan, Y., Yao, T. and Mei, T. (2022) 3D Cascade RCNN: High Quality Object Detection in Point Clouds. IEEE Transactions on Image Processing, 31, 5706-5719. https://doi.org/10.1109/TIP.2022.3201469 |
| [2] | An, P., Liang, J., Ma, T., Chen, Y., Wang, L. and Ma, J. (2023) ProUDA: Progressive Unsupervised Data Augmentation for Semi-Supervised 3D Object Detection on Point Cloud. Pattern Recognition Letters, 170, 64-69.
https://doi.org/10.1016/j.patrec.2023.04.002 |
| [3] | Xie, G., Li, Y., Wang, Y., Li, Z. and Qu, H. (2023) 3D Point Cloud Object Detection Algorithm Based on Temporal Information Fusion and Uncertainty Estimation. Remote Sensing, 15, Article 2986. https://doi.org/10.3390/rs15122986 |
| [4] | Newcombe, R.A., Izadi, S., Hilliges, O., et al. (2011) KinectFusion: Real-Time Dense Surface Mapping and Tracking. IEEE International Symposium on Mixed and Aug-mented Reality (ISMAR), Basel, 26-29 October 2011, 127-136.
https://doi.org/10.1109/ISMAR.2011.6092378 |
| [5] | Riegler, G., Ulusoy, A.O., Bischof, H. and Geiger, A. (2017) Octnetfusion: Learning Depth Fusion from Data. IEEE In-ternational Conference on 3D Vision (3DV), Qingdao, 10-12 October 2017, 57-66.
https://doi.org/10.1109/3DV.2017.00017 |
| [6] | Charles, R.Q., Su, H., Kaichun, M. and Guibas, L.J. (2017) Point-Net: Deep Learning on Point Sets for 3D Classification and Segmentation. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, 21-26 July 2017, 77-85. https://doi.org/10.1109/CVPR.2017.16 |
| [7] | Qi, C.R., Yi, L., Su, H. and Guibas, L.J. (2017) PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. International Conference on Neural Information Pro-cessing Systems (NIPS), Long Beach, 4-9 December 2017, 5099-5108. |
| [8] | Yu, L., Li, X., Fu, C.W., Cohen-Or, D. and Heng, P.A. (2018) PU-Net: Point Cloud Upsampling Network. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, 18-23 June 2018, 2790-2799.
https://doi.org/10.1109/CVPR.2018.00295 |
| [9] | Wang, YF., Wu, S., Huang, H., Cohen-Or, D. and Sorkine-Hornung, O. (2019) Patch-Based Progressive 3D Point Set Upsampling. 2019 IEEE/CVF Conference on Com-puter Vision and Pattern Recognition (CVPR), Long Beach, 15-20 June 2019, 5951-5960. https://doi.org/10.1109/CVPR.2019.00611 |
| [10] | Li, R., Li, X., Heng, P.A. and Fu, C.W. (2021) Point Cloud Up-sampling via Disentangled Refinement. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, 20-25 June 2021, 344-353.
https://doi.org/10.1109/CVPR46437.2021.00041 |
| [11] | Zhang, P., Wang, X., Ma, L., Wang, S., Kwong, S. and Jiang, J. (2021) Progressive Point Cloud Upsampling via Differentiable Rendering. IEEE Transactions on Circuits and Systems for Video Technology, 31, 4673-4685.
https://doi.org/10.1109/TCSVT.2021.3100134 |
| [12] | Bai, Y., Wang, X., Ang Jr., M.H. and Rus, D. (2022) BIMS-PU: Bi-Directional and Multi-Scale Point Cloud Upsampling. IEEE Robotics and Automation Letters, 7, 7447-7454. https://doi.org/10.1109/LRA.2022.3183932 |
| [13] | Du, H., Yan, X., Wang, J., Xie, D. and Pu, S. (2022) Point Cloud Upsampling via Cascaded Refinement Network. In: Wang, L., Gall, J., Chin, T.J., Sato, I. and Chellappa, R., Eds., Computer Vision—ACCV 2022, Springer, Cham, 106-122. |
| [14] | Qian, G., Abualshour, A., Li, G., Thabet, A. and Ghanem, B. (2021) PU-GCN: Point Cloud Upsampling Using Graph Convolutional Networks. 2021 IEEE/CVF Con-ference on Computer Vision and Pattern Recognition (CVPR), Nashville, 20-25 June 2021, 11678-11687. https://doi.org/10.1109/CVPR46437.2021.01151 |
| [15] | Szegedy, C., Ioffe, S., Vanhoucke, V. and Alemi, A. (2017) Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. Proceedings of the AAAI Confer-ence on Artificial Intelligence, 31, 4278-4284.
https://doi.org/10.1609/aaai.v31i1.11231 |
| [16] | Szegedy, C., et al. (2015) Going Deeper with Convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, 7-12 June 2015, 1-9. https://doi.org/10.1109/CVPR.2015.7298594 |
| [17] | Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. and Wojna, Z. (2016) Rethinking the Inception Architecture for Computer Vision. 2016 IEEE Conference on Computer Vision and Pat-tern Recognition (CVPR), Las Vegas, 27-30 June 2016, 2818-2826. https://doi.org/10.1109/CVPR.2016.308 |
| [18] | Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M. and Sol-omon, J.M. (2019) Dynamic Graph CNN for Learning on Point Clouds. ACM Transactions on Graphics, 38, 1-12. https://doi.org/10.1145/3326362 |
| [19] | Gu, F., Zhang, C., Wang, H., He, Q. and Huo, L. (2022) PU-WGCN: Point Cloud Upsampling Using Weighted Graph Convolutional Networks. Remote Sensing, 14, Article 5356. https://doi.org/10.3390/rs14215356 |
| [20] | Li, R., Li, X., Fu, C.W., Cohen-Or, D. and Heng, P.A. (2019) PU-GAN: A Point Cloud Upsampling Adversarial Network. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, 27 October-2 November 2019, 7202-7211. https://doi.org/10.1109/ICCV.2019.00730 |
| [21] | Liu, H., Yuan, H., Hamzaoui, R., Gao, W. and Li, S. (2022) PU-Refiner: A Geometry Refiner with Adversarial Learning for Point Cloud Upsampling. ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore, 23-27 May 2022, 2270-2274. https://doi.org/10.1109/ICASSP43922.2022.9746373 |
| [22] | Zhou, K., Dong, M. and Arslanturk, S. (2022) “Ze-ro-Shot” Point Cloud Upsampling. 2022 IEEE International Conference on Multimedia and Expo (ICME), Taipei, 18-22 July 2022, 1-6.
https://doi.org/10.1109/ICME52920.2022.9859662 |
| [23] | Liu, H., Yuan, H., Hou, J., Hamzaoui, R. and Gao, W. (2022) PUFA-GAN: A Frequency-Aware Generative Adversarial Network for 3D Point Cloud Upsampling. IEEE Transactions on Image Processing, 31, 7389-7402.
https://doi.org/10.1109/TIP.2022.3222918 |
| [24] | Mao, A., Du, Z., Hou, J., Duan, Y., Liu, Y.J. and He, Y. (2022) PU-Flow: A Point Cloud Upsampling Network with Normalizing Flows. IEEE Transactions on Visualization and Computer Graphics, 29, 4964-4977.
https://doi.org/10.1109/TVCG.2022.3196334 |
| [25] | Hu, X., Wei, X. and Sun, J. (2023) A Noising-Denoising Framework for Point Cloud Upsampling via Normalizing Flows. Pattern Recognition, 140, Article ID: 109569. https://doi.org/10.1016/j.patcog.2023.109569 |
| [26] | Liu, X.H., Han, Z.Z., Wen, X., Liu, Y.S. and Zwicker, M. (2019) L2G Auto-Encoder: Understanding Point Clouds by Local-to-Global Reconstruction with Hierarchical Self-Attention. Proceedings of the 27th ACM International Conference on Multimedia, Nice France, 21-25 October 2019, 989-997. https://doi.org/10.1145/3343031.3350960 |
| [27] | Zhao, Y.F., Hui, L. and Xie, J. (2021) SSPU-Net: Self-Supervised Point Cloud Upsampling via Differentiable Rendering. Proceedings of the 29th ACM International Conference on Mul-timedia, 20-24 October 2021, 2214-2223.
https://doi.org/10.1145/3474085.3475381 |
| [28] | Liu, X., Liu, X., Liu, Y.S. and Han, Z. (2022) SPU-Net: Self-Supervised Point Cloud Upsampling by Coarse-to-Fine Reconstruction with Self-Projection Optimization. IEEE Transactions on Image Processing, 31, 4213-4226.
https://doi.org/10.1109/TIP.2022.3182266 |
| [29] | Feng, W., Li, J., Cai, H., Luo, X. and Zhang, J. (2022) Neural Points: Point Cloud Representation with Neural Fields for Arbitrary Upsampling. 2022 IEEE/CVF Conference on Com-puter Vision and Pattern Recognition (CVPR), New Orleans, 18-24 June 2022, 18612-18621. https://doi.org/10.1109/CVPR52688.2022.01808 |
| [30] | Zhao, W., et al. (2022) Self-Supervised Arbitrary-Scale Point Clouds Upsampling via Implicit Neural Representation. 2022 IEEE/CVF Conference on Computer Vision and Pat-tern Recognition (CVPR), New Orleans, 18-24 June 2022, 1989-1997. https://doi.org/10.1109/CVPR52688.2022.00204 |
| [31] | Yang, Y., Feng, C., Shen, Y. and Tian, D. (2018) Folding-Net: Point Cloud Auto-Encoder via Deep Grid Deformation. 2018 IEEE/CVF Conference on Computer Vision and Pat-tern Recognition, Salt Lake City, 18-23 June 2018, 206-215.
https://doi.org/10.1109/CVPR.2018.00029 |
| [32] | Han, B., Zhang, X. and Ren, S. (2022). PU-GACNet: Graph At-tention Convolution Network for Point Cloud Upsampling. Image and Vision Computing, 118, Article ID: 104371. https://doi.org/10.1016/j.imavis.2021.104371 |
| [33] | Zhao, T., Li, L., Tian, T., Ma, J. and Tian, J. (2023) APUNet: Attention-Guided Upsampling Network for Sparse and Non-Uniform Point Cloud. Pattern Recognition, 143, Article ID: 109796.
https://doi.org/10.1016/j.patcog.2023.109796 |
| [34] | Mao, A., Duan, Y., Wen, Y.H., Du, Z., Cai, H. and Liu, Y.J. (2023) Invertible Residual Neural Networks with Conditional Injector and Interpolator for Point Cloud Upsampling. Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI-23), Macao, 19-25 August 2023, 1267-1275. |
| [35] | Nguyen, T., Pham, Q.H., Le, T., Pham, T., Ho, N. and Hua, B.S. (2021) Point-Set Dis-tances for Learning Representations of 3D Point Clouds. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, 10-17 October 2021, 10458-10467. https://doi.org/10.1109/ICCV48922.2021.01031 |
| [36] | Jesorsky, O., Kirchberg, K.J. and Frischholz, R. (2001) Ro-bust Face Detection Using the Hausdorff Distance. In: Bigun, J. and Smeraldi, F., Eds., Audio- and Video-Based Bio-metric Person Authentication, Springer, Berlin, 90-95.
https://doi.org/10.1007/3-540-45344-X_14 |
| [37] | Fan, H., Su, H. and Guibas, L. (2017) A Point Set Generation Network for 3D Object Reconstruction from a Single Image. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, 21-26 July 2017, 2463-2471.
https://doi.org/10.1109/CVPR.2017.264 |
| [38] | Chang, A.X., Funkhouser, T., Guibas, L.J., et al. (2015) ShapeNet: An Information-Rich 3D Model Repository.
arXiv: 1512.03012. |
| [39] | Li, Z., Li, G., Li, T.H., Liu, S. and Gao, W. (2023) Semantic Point Cloud Upsampling. IEEE Transactions on Multimedia, 25, 3432-3442. https://doi.org/10.1109/TMM.2022.3160604 |
| [40] | He, Y., Tang, D., Zhang, Y., Xue, X. and Fu, Y. (2023) Grad-PU: Arbitrary-Scale Point Cloud Upsampling via Gradient Descent with Learned Distance Functions. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Van-couver, 17-24 June 2023, 5354-5363. https://doi.org/10.1109/CVPR52729.2023.00518 |
