Thermal-Visible Image Fusion for Robust Autonomous Driving Perception Under Adverse Weather Conditions
Keywords:
thermal-visible fusion; autonomous driving; adverse weather; multimodal perception; system architecture; robustness; governance; infrastructure; fairness; sustainabilityAbstract
Autonomous driving perception must remain dependable when rain, fog, snow, dust, glare, and low illumination degrade visible-band imagery. Thermal-visible image fusion offers a multimodal strategy for combining complementary sensing physics, yet its value depends less on pixel-level enhancement than on system-level integration, architectural trade-offs, and deployment constraints. This paper examines thermal-visible fusion as a socio-technical perception infrastructure for robust autonomous driving under adverse weather. It analyzes sensing modality complementarity, fusion architectures, robustness limits, sensor placement, computational sustainability, and governance implications. The discussion emphasizes that fusion cannot eliminate ambiguity; it redistributes uncertainty across sensors, models, and operational domains. Effective designs therefore require calibrated degradation awareness, graceful degradation policies, traceable confidence estimation, and infrastructure support for maintenance and validation. The paper also considers fairness in pedestrian detection, policy questions around safety certification, and cross-domain lessons from rescue robotics, rail, and aviation. It argues that thermal-visible fusion should be treated as an accountable layer within a broader mobility system rather than as a standalone algorithmic remedy. The conclusion identifies research priorities in uncertainty-aware fusion, embedded efficiency, adverse-weather datasets, and regulatory frameworks that reward transparent robustness rather than benchmark overfitting.
References
1. Levinson, J., Askeland, J., Becker, J., Dolson, J., Held, D., Kammel, S., Kolter, J. Z., Langer, D., Pink, O., Pratt, V., Sokolsky, M., Stanek, G., Stavens, D., Teichman, A., Werling, M., & Thrun, S. (2011). Towards fully autonomous driving: Systems and algorithms. In 2011 IEEE Intelligent Vehicles Symposium (IV) (pp. 163–168). IEEE. https://doi.org/10.1109/IVS.2011.5940562
2. Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving? The KITTI vision benchmark suite. In 2012 IEEE Conference on Computer Vision and Pattern Recognition (pp. 3354–3361). IEEE. https://doi.org/10.1109/CVPR.2012.6248074
3. Ma, J., Ma, Y., & Li, C. (2019). Infrared and visible image fusion methods and applications: A survey. Information Fusion, 45, 153–178. https://doi.org/10.1016/j.inffus.2018.02.004
4. Li, S., Kang, X., & Hu, J. (2013). Image fusion with guided filtering. IEEE Transactions on Image Processing, 22(7), 2864–2875. https://doi.org/10.1109/TIP.2013.2244222
5. Liu, Y., Chen, X., Wang, Z., Wang, Z. J., Ward, R. K., & Wang, X. (2018). Deep learning for pixel-level image fusion: Recent advances and future prospects. Information Fusion, 42, 158–173. https://doi.org/10.1016/j.inffus.2017.10.007
6. Zhang, H., Xu, H., Tian, X., Jiang, J., & Ma, J. (2021). Image fusion meets deep learning: A survey and perspective. Information Fusion, 76, 323–336. https://doi.org/10.1016/j.inffus.2021.06.008
7. Fritsche, P., Zeise, B., Hemme, P., & Wagner, B. (2017). Fusion of radar, LiDAR and thermal information for hazard detection in low visibility environments. In 2017 IEEE International Symposium on Safety, Security and Rescue Robotics (SSRR) (pp. 96–101). IEEE. https://doi.org/10.1109/SSRR.2017.8088146
8. Sakaridis, C., Dai, D., & Van Gool, L. (2018). Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision, 126(9), 973–992. https://doi.org/10.1007/s11263-018-1072-8
9. Bijelic, M., Gruber, T., Mannan, F., Kraus, F., Ritter, W., Dietmayer, K., & Heide, F. (2020). Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 11682–11692). IEEE. https://doi.org/10.1109/CVPR42600.2020.01170
10. Pfeuffer, A., & Dietmayer, K. (2018). Optimal sensor placement for multi-sensor vehicle perception. IEEE Sensors Journal, 18(17), 7196–7206. https://doi.org/10.1109/JSEN.2018.2857830
11. Teichmann, M., Weber, M., Zoellner, M., Cipolla, R., & Urtasun, R. (2018). MultiNet: Real-time joint semantic reasoning for autonomous driving. In 2018 IEEE Intelligent Vehicles Symposium (IV) (pp. 1013–1020). IEEE. https://doi.org/10.1109/IVS.2018.8500504
12. Wang Z, He D, Zhao L, et al. Difusionseg: Diffusion-driven semantic segmentation with multi-modal image fusion for enhanced perception[J]. Knowledge-Based Systems, 2025: 114481.
13. Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015). DeepDriving: Learning affordance for direct perception in autonomous driving. In 2015 IEEE International Conference on Computer Vision (ICCV) (pp. 2722–2730). IEEE. https://doi.org/10.1109/ICCV.2015.312
14. Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., Zhang, X., Zhao, J., & Zieba, K. (2016). End to end learning for self-driving cars. arXiv. https://arxiv.org/abs/1604.07316
15. Naphade, M., Chang, M.-C., Sharma, A., Anastasiu, D. C., Jagarlamudi, V., Chakraborty, P., Huang, T., Wang, S., Liu, M.-Y., Chellappa, R., Hwang, J.-N., & Lyu, S. (2020). The 2019 AI City Challenge. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (pp. 452–460). IEEE. https://doi.org/10.1109/CVPRW.2019.00062
16. Sualeh, M., & Kim, G.-W. (2019). Dynamic multi-LiDAR based multiple object detection and tracking. Sensors, 19(6), 1474. https://doi.org/10.3390/s19061474
17. Kocić, J., Jovičić, N., & Drndarević, V. (2018). An end-to-end deep neural network for autonomous driving designed for embedded automotive platforms. Sensors, 18(7), 2120. https://doi.org/10.3390/s18072120
18. Shalev-Shwartz, S., Shammah, S., & Shashua, A. (2017). On a formal model of safe and scalable self-driving cars. arXiv. https://arxiv.org/abs/1708.06374
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Advanced Artificial Intelligence Research
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
