A Comprehensive Overview of Classical and New Perceivable Spatial and Temporal Artifacts in Compressed Video Streams

Modern video encoding tools use lossy coding schemes due to the constraints of

power and bandwidth. These coding approaches also create spatial and temporal artifacts

which ultimately decrease the received video quality. Understanding these classical and new

artifacts, especially produced due to new video encoding tools are very important for the

development of efficient video codec’s which can avoid these artifacts on the encoder side,

rather than compensating them on decoder side. There is still a lack of knowledge about new

artifacts due to the new video coding tools, besides the classical artifacts, i.e., blocking, ringing,

etc. Many existing papers in the literature describe these artifacts, but none of the paper

discusses these artifacts comprehensively. Therefore, this research provides a comprehensive

overview of all classical and new artifacts (spatial and temporal) produced by current and new

video coding tools, i.e., SVC (scalable video coding) and MVC (multi view coding) by H.264,

and NVC (next generation video codec’s - H.265/HEVC). The paper also describes the effect

of new coding tools on classical artifacts and how these artifacts are related to each other.

Overall, this research will assist in the design of more efficient adaptive quantization

algorithms and coding mechanisms to improve the video codec performance.

[1] Richardson, I.E.G. (2003) H.264 and MPEG-4 Video Compression. Chichester, England: Wiley & Sons Ltd.

[2] Sullivan, G., Ohm, J., Woo-Jin, H., and Wiegand, T., “Overview of the High Efficiency Video Coding (HEVC) standard,” IEEE Trans.

Circuits and Systems for Video Technology 22(12), 1649–1668 (2012).

[3] B. M. Zavodsky, “Time-variant video quality evaluation for mobile networks,” Phd Thesis, Fakulta elektrotechniky a informatiky

Katedra rdioelektroniky, Slovensk technick univerzita v Bratislave, 2006.

[4] Yuen, M. and Wu, H., “A survey of hybrid MC/DPCM/DCT video coding distortions,” Signal Processing 70 (3), 247–278 (1998).

[5] WeisiLin, C. Jay Kuo, “Perceptual visual quality metrics: A survey”, Journal of Visual

Communication and Image Representation, volume 22, Issue 4, May 2011, pages 297-312.

[6] Muhammad Shahid, Andreas Rossholm, Benny Lövström, and Hans Zepernick, “Noreference image and video quality

assessment : a classification and review of recent approaches”, EURASIP Journal on Image and Video Processing, 2014, pages 397-437.

[7] Andreas Unterweger, “Compression artifacts in modern video coding and state of the art means of compensation”, University of Salzburg, Austria, 2012.

[8] Chodisetti Rakesh Anil, “H.264 video coding artifacts; Measurement and reduction of flickering ,” Master thesis, Department of Electrical Engineering, Blekinge Institute of Technology, 2013.

[9] K. Zeng, T. Zhao, A. Rehman, and Z. Wang, “Characterizing perceptual artifacts in compressed video streams,

” IS&T/SPIE Human Vision and Electronic Imaging, vol. 9014, no. 10, pp. 2458–2462, Feb. 2014.

[10] Liqun Lin, Shiqi Yu, Tiesong Zhao, Member,” PEA265: Perceptual Assessment of Video Compression Artifacts”, Computer Vision and Pattern Recognition; Image and Video Processing, 1601–1612 (2019).

[11] Zeng, K., Rehman, A., Wang, J., and Wang, Z., “From H.264 to HEVC: Coding gain predicted by objective video quality assessment models,” in International Workshop on Video Processing and Quality Metrics for Consumer Electronics, Feb 2013.

[12] Hanwha Techwin, “White Paper: H.265 High Efficiency Coding Video Compression for Security Applications”, Ridgefield Park, NJ, USA, 2016

[13] K. Zhu, C. Li, V. Asari, and D. Saupe, “No-reference video quality assessment based on artifact measurement and statistical analysis,” IEEE Trans. Circuits Syst. Video Technol., vol. 25, no. 4, pp. 533–546, April, 2015.

[14] S. Yoo, K. Choi, and J. Ra, “Blind post-processing for ringing and mosquito artifact reduction in coded videos,” IEEE Trans. Circuits Syst. Video Technol., vol. 24, no. 5, pp. 721–732, May 2014.

[15] H Liu, N Klomp, I Heynderickx, “A no-reference metric for perceived ringing artifacts in images”, IEEE Trans. Circuits Syst. video technol. volume 4, issue 20, pages 529–539, October, 2010.

[16] N. Suresh, “Mean time between visible artifacts in visual communications,” Phd Thesis, Department of Electrical and Computer Engineering, Georgia tech. University, 2007.

[17] C. Dong, Y. Deng, C. Chang, and X. Tang, “Compression artifacts reduction by a deep convolution network,” in Proc. IEEE International Conference on Computer Vision (ICCV), Feb. 2016, pp. 576–584.

[18] J. Xia, Y. Shi, K. Teunissen, and I. Heynderickx, “Perceivable artifacts in compressed video and their relation to video quality,” Signal Process Image., vol. 24, no. 7, pp. 548– 556, Aug. 2009.

[19] M. Rohil, N. Gupta, P. Yadav, “An improved model for no-reference image quality assessment and a no-reference video quality assessment model based on frame analysis”, Signal, Image and Video Processing, vol 6, no.7, pp. 610-619, July 2019.

[20] J. Liu , D. Liu, W. Yang , S. Xia , X. Zhang , and Y. Dai,” A Comprehensive Benchmark for Single Image Compression Artifact Reduction”, IEEE Transactions on image processing, vol. 29, pp. 225-241, 2020

[21] K. Yang ; S. Wan ; Y. Gong ; Y. Yang ; Y. Feng, “Temporal Distortions in H.265/HEVC”, Asia-Pacific Signal and Information Processing Association Annual

Summit and Conference (APSIPA), Jeju, South Korea, 13-16 Dec. 2016.

[22] Y. C. Gong, S. Wan, K. Yang, F. Z. Yang, and L. Cui, “An efficient algorithm to eliminate temporal pumping artifact in video coding with hierarchical prediction structure,” J. visual communication, vol. 25, no. 7, pp. 1528–1542, Oct. 2014.

[23] Y. Yang, J. Xiang. and H. Wau., “Robust filtering technique for reduction of temporal fluctuation in H.264 video sequences,” IEEE Trans. Circuits and Systems for Video Technology 20(3), 458–462, Nov, 2010.

[24] J. Lee, J. Park, H. Wang, “A New Distortion Measure for Motion Estimation in Motion- Compensated Hybrid Video Coding. Signal Processing”, Image Communication, vol. 2, no. 6, pp. 75–84, 2011.

[25] P. V. Pahalawatta and A. M. Tourapis, “Motion estimated temporal consistency metrics for objective video quality assessment,” in Quality of Multimedia Experience, International Workshop on, pp. 174–179, 2009.

[26] S. Wan, Y. Gong, and F. Yang, “Perception of Temporal Pumping Artifact in Video Coding with the Hierarchical Prediction Structure,” in Proc. IEEE Int. Conf. Multimedia and Expo, pp. 503-508, Jul. 2012.

[27] Y. Gong, S. Wan, K. Yang, B. Li, and H. Wu, “Perception based Quantitative Definition of Temporal Pumping Artifact”, in Proc. of the 19th Int. Conf. on Digital Signal Processing, pp.870-875, August 2014.

[28] Y. Qi and M. Dai , “The Effect of Frame freezing and Frame skipping on video quality”, Proceedings of the 2006 International Conference on Intelligent Information Hiding and Multimedia Signal Processing, Jakarta, Indonesia, 13-16 April, 2006.

[29] Z. Akhtar and T. Falk, “Audio-Visual Multimedia Quality Assessment: A Comprehensive Survey”, IEEE access, vol. 11, no. 7, pp. 412-440, July, 2017

[30] G. Vetro, A. Wiegand, T. Sullivan, “Overview of the Stereo and Multiview Video Coding Extensions of the H.264/AVC Standard”. Proceedings of IEEE, Special Issue on 3D Media and Displays, vol. 99, no. 4, pp. 626–642, 2011.

[31] P. Merkle, Y. Smolic, P. Wiegand, “The effects of multiview depth video compression on multiview rendering”, Signal Processing: Image Communication, vol. 24, no. 1, pp. 73– 88, 2008.

[32] H. Boujut, B. Pineau, J. Hadar, O. Ahmed, “Weighted-MSE based on saliency map for assessing video quality of H.264 video streams”, Proceedings of the SPIE “Image Quality and System Performance VIII”, vol. 7, no. 6, pp. 686–696. 2011.

[33] R. Farrugia, A. Debono,”A Hybrid Error Control and Artifact Detection Mechanism for Robust Decoding of H.264/AVC Video Sequences”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 20, no. 5, pp. 756–762, 2010.