About VoLTE PEVQ Testing
Click here to see this page in full context

Landslide Performance Test System – Release 24.3 September 2024

About VoLTE PEVQ Testing

 

PEVQ must be replaced by VMAF reservation option - About VoLTE VMAF Testing

 

PEVQ to VMAF Transition

In 2024 OPTICOM is stopping PEVQ support. Landslide we will deprecate PEVQ and only allow it in previously saved tests. For all new tests, users should use VMAF - About VoLTE VMAF Testing,  Media Qos Settings

For customers that have the PEVQ license, there will be special rules:

  1. Any existing test that does not have PEVQ enabled, or any test created with 24.1, will show the PEVQ panel as disabled with a tooltip on the enabler indicating it is deprecated to use or ask for VMAF.
  2. Any existing test, previously saved with PEVQ enabled will show the enabled PEVQ panel.   But if you turn it off and save the test, you will fall into rule 1.

If you have both licenses, PEVQ and VMAF will be mutually exclusive and once a test is saved with PEVQ disabled the next time it is opened it will be disabled per rule 1.

When a user clicks OK on a test case and PEVQ is turned on, there will be a WARNING. 
"Landslide 24.1 is the final release supporting PEVQ. Please consider transition to VMAF." 

Each time the TC capable of PEVQ (PevqEn=true) is launching the following warning message will be reporting to RUN LOG:

"Landslide 24.1 is the final release supporting PEVQ. Please consider transition to VMAF."   

 

The purpose is to measure video quality in VoLTE environment by means of OPTICOM’s Perceptual Evaluation of Video Quality - PEVQ.  PEVQ is based on three International standards: ITU-T J.144, ITU-T J.247, and ITU-T J.246. It compares Reference Video Signal (X(t)) and Degraded Video Signal (Y(t)), the signal that passed through network subsystem being encoded and potentially impaired. For true digital subsystems as EPC and IMS being impaired means RTP packet lost or received late – out of Jitter Buffer time. PEVQ measures MOS (Mean Opinion Score) in the range of 1 (poor) to 5 (excellent). The best value ever obtained for video is 4.759. Besides MOS the algorithm accomplishes more specific calculations. They are:

 

 

PEVQ Algorithm assumes that both inputs, Reference and Degraded Signals, are AVI container with video stream in “rawvideo” format. Main diagram of flowing and transforming of Reference video signal into Degraded signal is shown below in Figure 1-1.

 

 

 

Structure of the PEVQ algorithm is shown in Figure 1-2.

 

The PEVQ algorithm was validated by OPTICOM © for the following video frames formats:

 

 

Minimal frame rate  is 2 fps; there is no hard limit for the upper boundary.

 

The algorithm is extremely time consuming and depends on frame resolution and frame rate mostly. Table below gives approximate time per one PEVQ measurement. As it has seen from the Table it is of seconds scale.  Just a reminder, POLQA accomplishing voice QoM measurements takes a fraction of second. 

 

Format

Description

C100/S2

1 core

4 cores

QCIF

PevqRef_qcif.avi/ PevqRef_qcif.avi: 25 Fps, 220 frames/8.8 sec

13.8 sec

4.13 sec

CIF

PevqRef_cif.avi/ PevqRef_cif.avi: 25 Fps, 220 frames/8.8 sec

50.2 sec

13.8 sec

VGA

PevqRef_vga.avi/ PevqRef_vga.avi: 25 Fps, 220 frames/8.8 sec

145 sec

39.2 sec

Processing Speed for PEVQ (before being optimized by OPTICOM®)

 

The OPTICOM provided new/improved version of the PEVQ library, highly optimized. This version can run only on one CPU core that is perfectly fits to that only one CPU core is reserved to accomplish PEVQ calculation. The results are very impressive against the library of the previous version:

Format

Description

C100/S2

1 core

QCIF

PevqRef_qcif.avi/ PevqRef_qcif.avi: 25 Fps, 220 frames/8.8 sec

2.3 sec

CIF

PevqRef_cif.avi/ PevqRef_cif.avi: 25 Fps, 220 frames/8.8 sec

10.25 sec

VGA

PevqRef_vga.avi/ PevqRef_vga.avi: 25 Fps, 220 frames/8.8 sec

40.2 sec

Processing Speed for PEVQ (after being optimized by OPTICOM®)

VoLTE or SIP capable Test Cases

SUT

 

See, also section “One- and two-arm testing”.

Restrictions

One- and two-arm testing

PEVQ measurements should be applicable to both: one-arm & two-arm testing. 

In two-arm testing the SUT (SGW or/and PGW) is true digital communication system that may encounter just RTP packets lost or late arrival to the destination. To create this condition PEVQ measurements should be combined with intensive RTP video traffic: a few UEs receiving RTP traffic should be designated to perform PEVQ measurements.

 

There are two VoLTE’s two arm-testing environment:

There are two VoLTE’s one-arm testing environments where the X(t) can be distorted

“rtpvideo” DMF

 

The PEVQ specific tests are based on the RTP traffic generated/processed by “rtpvideo” DMF only.

X(t) and Y(t) Signals as input for PEVQ

 

The data flow and video signal transition is shown in Figure 1.2

 

Data flow and video signal transition

The data flow and video signal transition is shown in Figure 1.1

 

Transmit side performs the following steps:

1.     Encodes X(t) signal presented in “rawvideo” format by means of H.263, H.264, H.265 or VP8 codecs.

2.   Packetizes the encoded video signal into RTP stream: one or more RTP packet (s) per video frame.

3.     Sends packetized video frames out according “fps” characteristic of the signal.

 

Receive side performs the following steps:

1.     Receives the packets and collects them in Jitter Buffer.

2.     Decodes the RTP stream to exactly the same format it was encoded from.

3.     Writes the decoded signal to file – Y(t) signal.

4.     Applies PEVQ algorithm to reference (X(t)) and impaired (Y(t)) signals that produces PEVQ score as well as some additional results.

 

PEVQ Capacity/Performance

The PEQV algorithm is extremely time consuming that means that the performance and so the capacity is very limited and realistically we can obtain PEQV score from one or two UEs only.  The Table 1.1 shows processing speed for video files of different formats.

 

To support PEVQ scores in real-time the following enhancement should be considered:

     1)   Introducing inter RTP file  gap that will allow to combine receiving RTP stream and PEVQ processing in real-time

     2)   Reserving more then one CPU core to QoM thread performing PEVQ algorithm.

PEVQ and POLQA

1)     One TC can run POLQA & PEVQ:  TCpolqa+pevq

2)     Any two TCs of a Test Session can run: one – POLQA and another PEVQ: TC1polqa & TC2pevq

3)     Any two Test Sessions running in context of one LS process can run separately POLQA and PEVQ:  TC(TS1)polqa & TC(TS2)pevq

Self adjusting mechanism of PEVQ score calculation

Self-adjusting mechanism that allows to perform as many PEVQ as possible. The mechanism evenly distributes the measurements among all the UEs.

GUI Provisioning

These are the following GUI components:

1)     CPU core reservation to perform PEVQ calculation: “Test Server | Configuration”

2)     “System Status” that reports “Remaining PEVQ ECs”

3)     Specifying AVI file in “rawvideo” format as the reference video signal (X(t)): “VoLTE TC | Gm | Media | RTP Traffic | ‘rtpvideo’ DMF”

4)     Actual PEVQ enabler

PEVQ Resource Reservation

POLQA and PEVQ share the same Resource Reservation mechanism to  allocate CPU core to perform POLQA and/or PEVQ calculation.

 

 

System Status

System Status shall include “Remaining PEVQ ECs” as it shown in Figure below:

 

rtpvideo” DMF

To support PEVQ the “rtpvideo” DMF | RTP Video” form has to be updated:

GUI must allow to place AVI file in “Media TDF” field.  This is the Reference video file (X(t)). It has to be specified in all Streaming modes: Transmit (“Tx Only”), Receive (“Rx Only”) , and Bidirectional

 

 

PEVQ is enabled on a manner similar to how POLQA is enabled. Video frame similar to Voice frame in “QoM”.

“Enable (POLQA) and “Enable (PEVQ)” are mutually inclusive: do allow both controls being ON at the same time.

Measurements

PEVQ measurements are added to “L5-7 Client| Rtp Video” and “L5-7 Server| RTP Video” Measurement Groups. All the measurements below those are of floating type are reporting as

uint64Value = (uint64) fValue * 1000. Upon receiving TAS must divide the value by 1000 and report on GUI as floating number with accuracy of up to 3 digits after floating point. In fact, 4.1, 4.15, or 4.153

RTP Counters

Description

RTP MOS-PEVQ Min/Ave/Max

MOS for PEVQ. Accuracy is 0.001

RTP MOS-PEVQ Meas Count

Number of times calculation of MOS-PEVQ was succeeded

RTP MOS-PEVQ Meas Count (Failed)

Number of times calculation of MOS-PEVQ was failed

RTP MOS-PEVQ Meas Count (Overriden)

Number of times calculation of MOS-PEVQ was not performed because of overload condition

RTP MOS-PEVQ Duration (sec)

Min/Ave/Max

 

Time  since RTP stream was fully collected to the moment when it’s MOS gets calculated.

Accuracy is 0.001 seconds.

 

RTP PEVQ Decorrelation Indicator

Min/Ave/Max

 

Value of the temporal distortion indicator averaged over all frames. Range 1..10. Accuracy 0.001

 

RTP PEVQ PSNR-Y (dB)

Min/Ave/Max

 

Averaged value of the PSNR for the luminance values.

Range 0..100. Accuracy is 0.001

 

RTP PEVQ PSNR-Cb (dB)

Min/Ave/Max

 

Averaged value of the PSNR for the Cb component of the YCbCr color space.

Range 0..100. Accuracy is 0.001

 

RTP PEVQ PSNR-Cr (dB)

Min/Ave/Max

 

Averaged value of the PSNR for the Cb component of the YCbCr color space.

Range 0..100. Accuracy is 0.001

 

RTP PEVQ Blockiness

Min/Ave/Max

 

Value of the blockiness indicator, averaged over all frames. Range 0..10. Accuracy 0.001

 

RTP PEVQ Jerkiness

Min/Ave/Max

Value of the jerkiness indicator, averaged over all frames. Range: 0..10. Accuracy 0.001

 

EC (Effective Channels) consumed

PEVQ has the same license concept as POLQA: based on EC (Effective Channels)

 

PEVQ TDF Library of VIDEO files

The PEVQ TDFLibrary is based on OPTICOM ® PEVQ Distribution Package includes video files.

The PEVQ TDF Library was created from two files enlisted below. The VGA based file was not included as it size exceeds 100 MB and is not supported by the GUI (at least as of 14.6).

 

File name

fps

Format

Size (KB)

Duration

(seconds)

Bitrate

(kb/s)

Resolution

PevqRef_cif.avi

25

rawvideo

65,354

8.8

60838

352x288

PevqRef_qcif.avi

25

rawvideo

16,340

8.8

15218

176x144

 

© 2024 Spirent Communications Inc. All Rights Reserved.