Subjective Video Quality Assessment

Subjective quality assessment is of paramount importance for the characterization of the performance of every media processing or transmission system whose end user is a human being. The main drawback of these methods is always their cost, which limits the number of variants or parameters of the investigated methods that can be explored. Hence, normative protocols are usually too cumbersome to assess the effects of minor algorithmic variations and not feasible for quotidian lab experimentation.

GPM in collaboration with Zygmunt Pizlo from Purdue University has developed a simplified protocol and associated metrics based on Signal Detection Theory (SDT) for subjective Video Quality Assessment (VQA) is proposed with the aim of filling the gap existing between the lack of discrimination abilities of objective Quality Estimates (specially when perceptually motivated processing methods are involved) and the costly normative subjective quality tests.

The proposed protocol employs a reduced number of assessors and provides a quality ranking of the methods being evaluated. It is intended for providing the rapid experimental turn around necessary for developing algorithms.

GPM graphical Interface

A java-based graphical interface to design and administrate the test is also available here.

The multiplatform application requires JMF (Java Media Framework): a Java library that supports video in several formats (AVI, MPEG, and MOV). A user guide provides instructions to design and configure the experiments. The number of sessions, number of sequences per session, video format and path where videos are stored must be supplied at this stage. Then, each pair of videos is presented sequentially on a light gray plain background after which the system requires an answer for the VQA question. During the experiments, the application collects the results in a temporary file so that the sessions can be paused and restarted at any point of the test. The final results of the test are provided in a plain text file.

References

M. de Frutos-López, A. B. Mejía-Ocaña, S. Sanz-Rodríguez, C. Peláez- Moreno, F. Díaz-de-María, and Z. Pizlo, “A simplified subjective video quality assessment method based on Signal Detection Theory,” Picture Coding Symposium, Kraków, Poland, May 2012 (acepted).

M. de Frutos-López, A. B. Mejía-Ocaña, S. Sanz-Rodríguez, C. Peláez- Moreno, F. Díaz-de-María, and Z. Pizlo, “A simplified subjective video quality assessment method for rapid experimental turn-around,” IEEE Transactions on Circuits and Systems for Video Technology, 2012 (submitted).