Efficient Video Anomaly Localization through Persistent Memory Modeling and Temporal Feature Refinement
Keywords:
video anomaly localization, persistent memory, temporal feature refinement, efficient deep learning, socio-technical systems, edge computing, algorithmic fairnessAbstract
Video anomaly localization has become a critical component of intelligent surveillance and industrial monitoring infrastructures, yet existing approaches often suffer from prohibitive computational costs and a lack of temporal coherence, impeding real-world deployment in large-scale systems. This paper presents a novel framework that integrates persistent memory modeling with temporal feature refinement to enable efficient and accurate video anomaly localization. The persistent memory module maintains a compact, dynamically updated prototypical representation of normal behavioral patterns across long sequences, while the temporal feature refinement mechanism enhances discriminative power by adaptively aggregating multi-scale motion and appearance cues. By decoupling memory storage from frame-wise inference depth and employing lightweight temporal convolutions, the architecture achieves significant reductions in latency and memory footprint without sacrificing localization fidelity. The discussion extends beyond algorithm design to system-level considerations, encompassing deployment on heterogeneous edge-cloud continuums, governance of surveillance data, fairness audits under demographic and environmental variations, and sustainable engineering practices. Through an interdisciplinary lens, the paper analyzes structural trade-offs in memory capacity, update strategies, and feature granularity, offering a blueprint for anomaly localization systems that balance performance, robustness, and socio-technical accountability. The proposed design principles are contextualized within the broader landscape of video understanding, smart city governance, and responsible artificial intelligence, highlighting pathways toward ethically aligned and operationally resilient large-scale surveillance infrastructures.
References
1. Sultani, W., Chen, C., & Shah, M. (2018). Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 6479–6488).
2. Gong, D., Liu, L., Le, V., Saha, B., Mansour, M. R., Venkatesh, S., & Hengel, A. V. D. (2019). Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1705–1714).
3. Liu, W., Luo, W., Lian, D., & Gao, S. (2018). Future frame prediction for anomaly detection – a new baseline. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 6536–6545).
4. Hasan, M., Choi, J., Neumann, J., Roy-Chowdhury, A. K., & Davis, L. S. (2016). Learning temporal regularity in video sequences. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 733–742).
5. Chong, Y. S., & Tay, Y. H. (2017). Abnormal event detection in videos using spatiotemporal autoencoder. In Proceedings of the International Symposium on Neural Networks (pp. 189–196).
6. Ramachandra, B., Jones, M., & Vatsavai, R. (2020). A survey of single-scene video anomaly detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(5), 2293–2312.
7. Wang, X., Girshick, R., Gupta, A., & He, K. (2018). Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 7794–7803).
8. Bai, S., Kolter, J. Z., & Koltun, V. (2018). An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271.
9. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780.
10. Carreira, J., & Zisserman, A. (2017). Quo vadis, action recognition? A new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 6299–6308).
11. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (Vol. 30).
12. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the NAACL-HLT (pp. 4171–4186).
13. Crankshaw, D., Wang, X., Zhou, G., Franklin, M. J., Gonzalez, J. E., & Stoica, I. (2017). Clipper: A low-latency online prediction serving system. In Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation (pp. 613–627).
14. Barocas, S., Hardt, M., & Narayanan, A. (2019). Fairness and machine learning. fairmlbook.org.
15. Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019). Fairness and abstraction in sociotechnical systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency (pp. 59–68).
16. Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). "Why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1135–1144).
17. Li, G., Yuan, H., Chen, S., Hu, Q., Wang, J., & Jiang, K. (2026). MFT: Memory-Aware Fine-Tuning of SAM2 for Efficient Long-Sequence Video Object Segmentation. IEEE Signal Processing Letters.
18. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., ... & Girshick, R. (2023). Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4015–4026).
19. Luo, W., Liu, W., & Gao, S. (2017). A revisit of sparse coding based anomaly detection in stacked RNN framework. In Proceedings of the IEEE International Conference on Computer Vision (pp. 341–349).
20. Georgescu, M. I., Barbalau, A., Ionescu, R. T., Popescu, M., & Shah, M. (2021). Anomaly detection in video via self-supervised and multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12742–12752).
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Advanced Artificial Intelligence Research
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
