Video quality

Video quality is a characteristic of a video passed through a video transmission/processing system, a formal or informal measure of perceived video degradation (typically, compared to the original video). Video processing systems may introduce some amount of distortion or artifacts in the video signal, which negatively impacts the user's perception of a system. For many stakeholders such as content providers, service providers and network operators, the assurance of video quality is an important task.

In order to develop novel audio and video quality metrics, the concept of QoE need to be integrated to the model of those metrics; for instances, the user preference for the multimedia content should be considered.^[1]^[2]^[3]^[4]

Video quality evaluation is performed to describe the quality of a set of video sequences under study. Video quality can be evaluated objectively (by mathematical models) or subjectively (by asking users for their rating). Also, the quality of a system can be determined offline (i.e., in a laboratory setting for developing new codecs or services), or in-service (to monitor and ensure a certain level of quality).

From analog to digital video

Since the world's first video sequence was recorded and transmitted, many video processing systems have been designed. Such systems encode video streams and transmit them over various kinds of networks or channels. In the ages of analog video systems, it was possible to evaluate the quality aspects of a video processing system by calculating the system's frequency response using test signals (for example, a collection of color bars and circles).

Digital video systems have almost fully replaced analog ones, and quality evaluation methods have changed. The performance of a digital video processing and transmission system can vary significantly and depends, amongst others, on the characteristics of the input video signal (e.g. amount of motion or spatial details), the settings used for encoding and transmission, and the channel fidelity or network performance.

Objective video quality

Objective video models are mathematical models that approximate results from subjective quality assessment, in which human observers are asked to rate the quality of a video. In this context, the term model may refer to a simple statistical model in which several independent variables (e.g. the packet loss rate on a network and the video coding parameters) are fit against results obtained in a subjective quality evaluation test using regression techniques. A model may also be a more complicated algorithm implemented in software or hardware. The terms model and metric are often used interchangeably.

In general, the aforementioned models are based on criteria that can be measured objectively – that is, free from human interpretation. They can be automatically evaluated by a computer program. Unlike a panel of human observers, an objective model will always output the same quality score for a given set of input parameters.

Classification of objective video quality metrics

Objective metrics can be classified by the amount of information available about the original signal, the received signal, or whether there is a signal present at all:^[5]

Full Reference Methods (FR): FR metrics compute the quality difference by comparing the original video signal against the received video signal. Typically, every pixel from the source is compared against the corresponding pixel at the received video, with no knowledge about the encoding or transmission process in between. More elaborate algorithms may choose to combine the pixel-based estimation with other approaches such as described below. FR metrics are usually the most accurate at the expense of higher computational effort.
Reduced Reference Methods (RR): RR metrics extract some features of both videos and compare them to give a quality score. They are used when all the original video is not available, or when it would be practically impossible to do so, e.g. in a transmission with a limited bandwidth. This makes them more efficient than FR metrics.
No-Reference Methods (NR): NR metrics try to assess the quality of a distorted video without any reference to the original signal. Due to the absence of an original signal, they may be less accurate than FR or RR approaches, but are more efficient to compute.
- Pixel-Based Methods (NR-P): Pixel-based metrics use a decoded representation of the signal and analyze the quality based on the pixel information. Some of these evaluate specific degradation types only, such as blurring or other coding artifacts.
- Parametric/Bitstream Methods (NR-B): These metrics make use of features extracted from the transmission container and/or video bitstream, e.g. MPEG-TS packet headers, motion vectors and quantization parameters. They do not have access to the original signal and require no decoding of the video, which makes them more efficient. In contrast to NR-P metrics, they have no access to the final decoded signal. However, the picture quality predictions they deliver are not very accurate.
- Hybrid Methods (Hybrid NR-P-B): Hybrid metrics combine parameters extracted from the bitstream with a decoded video signal. They are therefore a mix between NR-P and NR-B models.

Use of picture quality models for video quality estimation

Some models that are used for video quality assessment (such as PSNR or SSIM) are simply image quality models, whose output is calculated for every frame of a video sequence. This quality measure of every frame can then be recorded over time to assess the quality of an entire video sequence. While this method is easy to implement, it does not factor in certain kinds of degradations that develop over time, such as the moving artifacts caused by packet loss and its concealment. A video quality metric that considers the temporal aspects of quality degradations, like the MOVIE Index, may be able to produce more accurate predictions of human-perceived quality.

Examples

No-reference metrics

An overview of recent no-reference image quality models has been given in a journal paper by Shahid et al.^[5]

Natural scene statistic based NR picture quality prediction models developed by the Laboratory for Image and Video Engineering (LIVE) are competitive with leading FR models on the world's largest public-domain subjective picture quality databases. These include the DIIVINE,^[6] BRISQUE,^[7] BLIINDS,^[8] and NIQE^[9] models, which differ primarily in the form of picture transform they use.

Simple full-reference metrics

The most traditional ways of evaluating quality of digital video processing system (e.g. a video codec) are FR-based. Among the oldest FR metrics are signal-to-noise ratio (SNR) and peak signal-to-noise ratio (PSNR), which are calculated between every frame of the original video signal and the video passed through a system (e.g., an encoder or a transmission channel). PSNR is the most widely used objective image quality metric, and the average PSNR over all frames can be considered a video quality metric. However, PSNR values do not correlate well with perceived picture quality due to the complex, highly non-linear behavior of the human visual system.^[10]

More complex full-reference metrics

With the success of digital video, a large number of more precise metrics were developed. These metrics are inherently more complex than PSNR, and need more computational effort to calculate predictions of video quality. Among those metrics are for example UQI, VQM, SSIM and the MOVIE Index.

Based on the results of benchmarks by the Video Quality Experts Group (VQEG) (some in the course of the Multimedia Test Phase (2007–2008) and the HDTV Test Phase I (2009–2011)), some metrics were standardized in ITU-T as:

ITU-T Rec. J.147 (FR), 2002
ITU-T Rec. J.246 (RR), 2008
ITU-T Rec. J.247 (FR), 2008 (see PEVQ)
ITU-T Rec. J.341 (FR), 2011 (see VQuad-HD)
ITU-T Rec. J.342 (RR), 2011

Bitstream-based metrics

The above metrics still require access to the original video bitstream before transmission, or at least part of it. In practice, an original stream may not always be available for comparison, for example when measuring the quality from the user side. For a more efficient estimation of video quality in such cases, parametric/bitstream-based metrics were also standardized as ITU-T Rec. P.1201 and P.1202.

Use in practice

Few of these standards have found successful commercial application, including PEVQ and VQuad-HD. However, two models invented at the Laboratory for Image and Video Engineering (LIVE): the Primetime Emmy Award-winning Structural Similarity (SSIM) video quality measurement tool and the MOVIE Index, as well as the older Tektronix PQA models are used by broadcast and post-production houses throughout the television and cinematic industries.

Training and performance evaluation

Since objective video quality models are expected to predict results given by human observers, they are developed with the aid of subjective test results. During development of an objective model, its parameters should be trained so as to achieve the best correlation between the objectively predicted values and the subjective scores, often available as mean opinion scores (MOS).

The most widely used subjective test materials are in the public-domain and include still picture, motion picture, streaming video, high definition, 3-D (stereoscopic) and special-purposes picture quality related datasets. These so-called databases are created by various research laboratories around the world. Some of them have become de facto standards, including several public-domain subjective picture quality databases created and maintained by the Laboratory for Image and Video Engineering (LIVE).

In theory, a model can be trained on a set of data in such a way that it produces perfectly matching scores on that dataset. However, such a model will be over-trained and will therefore not perform well on new datasets. It is therefore advised to validate models against new data and use the resulting performance as a real indicator of the model's prediction accuracy.

To measure the performance of a model, some frequently used metrics are the linear correlation coefficient, Spearman's rank correlation coefficient, and the root mean square error (RMSE). Other metrics are the kappa coefficient and the outliers ratio. ITU-T Rec. P.1401 gives an overview of statistical procedures to evaluate and compare objective models.

Uses and application of objective metrics

Objective video quality metrics can be used in various application areas. In video codec development, the performance of a codec is often evaluated in terms of PSNR or SSIM. For service providers, objective metrics can be used for monitoring a system. For example, an IPTV provider may choose to monitor their service quality by means of objective metrics, rather than asking users for their opinion, or waiting for customer complaints about bad video quality.

An objective metric should only be used in the context that it was developed for. For example, a model that was developed using a particular video codec is not guaranteed to be accurate for another video codec. Similarly, a model trained on tests performed on a large TV screen should not be used for evaluating the quality of a video watched on a mobile phone.

Other approaches

When estimating quality of a video codec, all the mentioned objective methods may require repeating post-encoding tests in order to determine the encoding parameters that satisfy a required level of visual quality, making them time consuming, complex and impractical for implementation in real commercial applications. There is ongoing research into developing novel objective evaluation methods which enable prediction of the perceived quality level of the encoded video before the actual encoding is performed [1].

Subjective video quality

The main goal of many objective video quality metrics is to automatically estimate the average user's (viewer's) opinion on the quality of a video processed by a system. Procedures for subjective video quality measurements are described in ITU-R recommendation BT.500 and ITU-T recommendation P.910. Their main idea is the same as in Mean Opinion Score for audio: video sequences are shown to a group of viewers and then their opinion is recorded and averaged to evaluate the quality of each video sequence. However, the testing procedure may vary depending on what kind of system is tested.

References

^ Rodriguez, Demostenes; Rosa, Renata Lopes; Bressan, Graça (26 August 2012). Quality metric to assess video streaming service over TCP considering temporal location of pauses (pdf). IEEE Transactions on Consumer Electronics. IEEE. pp. 985–992. doi:10.1109/TCE.2012.6311346. Retrieved November 25, 2016.
^ Rodriguez, Demostenes; Rosa, Renata Lopes; Bressan, Graça; WANG, ZHOU (December 2014). The impact of video-quality-level switching on user quality of experience in dynamic adaptive streaming over HTTP (pdf). EURASIP J WIREL COMM. Springer. doi:10.1186/1687-1499-2014-216. Retrieved November 25, 2016.{{cite conference}}: CS1 maint: unflagged free DOI (link)
^ Rodriguez, Demostenes; Bressan, Graça (January 2012). Video quality assessments on digital TV and video streaming services using objective metrics (pdf). IEEE Latin America Transactions. IEEE. doi:10.1109/TLA.2012.6142458. Retrieved November 25, 2016.
^ Rodriguez, Demostenes; Rosa, Renata Lopes; Costa, Eduardo; Abrahão, Julia; Bressan, Graça (November 2014). Video quality assessment in video streaming services considering user preference for video content (pdf). IEEE Transactions on Consumer Electronics. IEEE. doi:10.1109/TCE.2014.6937328. Retrieved November 25, 2016.
^ ^a ^b Shahid, Muhammad (2014-02-16). "No-reference image and video quality assessment: a classification and review of recent approaches". EURASIP Journal on Image and Video Processing.
^ Moorthy, A.K.; Bovik, A.C. (2010-05-01). "A Two-Step Framework for Constructing Blind Image Quality Indices". IEEE Signal Processing Letters. 17 (5): 513–516. doi:10.1109/LSP.2010.2043888. ISSN 1070-9908.
^ Saad, M.A.; Bovik, A.C.; Charrier, C. (2012-08-01). "Blind Image Quality Assessment: A Natural Scene Statistics Approach in the DCT Domain". IEEE Transactions on Image Processing. 21 (8): 3339–3352. doi:10.1109/TIP.2012.2191563. ISSN 1057-7149.
^ Mittal, A.; Moorthy, A.K.; Bovik, A.C. (2012-12-01). "No-Reference Image Quality Assessment in the Spatial Domain". IEEE Transactions on Image Processing. 21 (12): 4695–4708. doi:10.1109/TIP.2012.2214050. ISSN 1057-7149.
^ Mittal, A.; Soundararajan, R.; Bovik, A.C. (2013-03-01). "Making a #x201C;Completely Blind #x201D; Image Quality Analyzer". IEEE Signal Processing Letters. 20 (3): 209–212. doi:10.1109/LSP.2012.2227726. ISSN 1070-9908.
^ Winkler, Stefan. "The evolution of video quality measurement: from PSNR to hybrid metrics". IEEE Transactions on Broadcasting. doi:10.1109/TBC.2008.2000733.