Test Summary Measurements
Click here to see this page in full context

Landslide Performance Test System – Release 24.3 September 2024

Test Summary Measurements

The values on the Test Summary tab report the aggregate measurements across all protocols used in the test.

Measurement Values

In the default view, the Reports tab displays the Summary view, which summarizes the measurements for the test session. If your test includes multiple test cases, you can view the report for a single test case by changing the report view...

Unless otherwise noted, cumulative values report the number of occurrences or rates calculated since the start of the test, and per-interval values report the number of occurrences or rates calculated during the interval. You can view either by changing the measurement values...

Measurement Index

Test Time measurements are available in all test cases.

The following sections contain measurements that are available in all test cases that include control plane traffic:

The following sections contain measurements that are only available in certain tests or test configurations:

Test Times

The first Test Summary measurements report the time involved in the test.

Elapsed Time The amount of time between the start of the test and the end of the report interval.
Test Iteration Reports the current iteration in a repeating test session. It is only displayed when a Number of Iterations is defined for the test session.
Actual Time The actual time shown on the Client. Actual Time is determined by the TAS. If the TAS time is wrong, Actual Time is reported wrong.  
Start Time

The date and time that a test case is started on the Test Server:

  • Start Time indicates each test case start time and is determined by the Test Server.
  • If the TS is configured for UTC time, and thus reports it with UTC, the Start Time is converted to local time zone (TAS for end of test reports, Client if viewing in the GUI), if not, the Start Time indicates the time reported by the TS. 
Local Time The Local time is the system time (stored in milliseconds internally) displayed in the appropriate time zone and time on the Client. (Retrieving the Raw Value from Tcl API returns a long integer system time).
Test Duration The length of time the test session will run (when a duration is defined).
Est. Time Remaining The time remaining in the test duration (when a duration is defined).

^ Back to Top

Session Rates

The following measurements record the session connect and disconnect rates for all tests that include control plane traffic. (See UMTS measurements for UMTS rate measurements.)

Attempted Connect Rate

 The rate at which the test attempted to establish MN sessions in the previous interval, as defined by Activation Rate.

Attempted Disconnect Rate

 The rate at which the test attempted to disconnect MN sessions in the previous interval, as defined by Deactivation Rate. This measurement is not displayed until the first session disconnect is attempted.

NOTE: When the test definition includes an option that adjusts the defined connect and disconnect rates, the adjusted attempted rates will be reported.

Actual Connect Rate

 The actual rate at which MN sessions were successfully established in the previous interval. A significant variance between this rate and the Attempted Connect Rate can indicate communication delays with the SUT or a heavy load on the SUT CPU. You can Connect to the SUT to check the CPU load.

NOTE: In a Data Verification test, a session is not considered to be connected until data connectivity has been verified. This can significantly increase Actual Connect Rate.

Actual Disconnect Rate

 The actual rate at which MN sessions were successfully disconnected in the previous interval. This measurement is not displayed until the first session disconnect is attempted. A significant variance between this rate and the Attempted Disconnect Rate can indicate communication delays with the SUT or a heavy load on the SUT CPU. You can Connect to the SUT to check the CPU load.

Average Session Connect Time

The Average Session Connect time reported in the Test Summary tab is the time taken by the traffic layer to get established.

The average time — measured from transmission of the first request to receipt of the final acknowledgement — required to successfully activate an MN session. This value is averaged for all successful session activations across all protocol layers.

Calculation on Test Summary Tab:

For example, if Create PDP Request is sent at time t, and Create PDP Response is received at time (t + n), the MN address received is written in the Create PDP response in the traffic layer and then traffic layer is established. That is, the traffic layer is established at time (t + n + m).

  • The average connect time reported in the Test Summary tab:
    m = (t + n +  m) - (t + n)

For IPV4 traffic state transition (traffic layer establishment) is immediate and for IPV6 traffic,  traffic layer establishment does not occur until a Router Advertisement is received.

Calculation on GTP Tab:

The average connect time reported in the GTP tab is the time taken by the GTP layer to get established. For example, if a Create PDP request is sent at time t, and the Create PDP Response is received at time (t + n). The average connect time reported on the GTP tab is calculated as n = (t + n) - t.

NOTE: In a Data Verification test, a session is not considered to be connected until data connectivity has been verified. This can significantly increase Average Session Connect Time.

 

LTE Calculation: An example:

The Session Connect Time from signalling perspective define the time period when the UE moves from EMM_Deregistered to EMM_Registered and ECM_Connected state.

UE =  EMM_deregistered

t = Point in time when UE will send the signalling message toward MME:  

S1AP/NAS-EPS  id-initialUEMessage  “Attach Request  PDN Connectivity Request”

2. UE =   ECM_Connected

3. t + n = point in time when SGW sends the originated by node signalling message toward MME: GTPv2 Modify Bearer Response (in this case SGW and PGW are simulated by Landslide)

or

t + n = Point in time when EnodeB sends the originated by node signalling message toward MME: S1AP/NAS-EPS id-uplinkNASTransport  “Attach complete  Activate default EPS bearer context accept”  (in this case SGW and PGW are  not simulated by Landslide)

4. UE = EMM_Registered and ECM_Connected

   
   

 

For only one subscriber, the Average Session Connect Time is calculated as:

  • n = (t+n) - t (which is the same calculation as on the GTP tab shown above)

  • Logically there is no “Average Session Connect Time” but “ Session Connect Time” or “Average Session Connect Time” = “ Session Connect Time”.

For more than one subscribers should be:  

Sum of Session Connect Time per session (a=n; ak=nk):
Average Session Connect Time per interval= / (number of established sessions per interval (15 seconds))

NOTE: The cumulative rate (other than in L5-7 measurements) is calculated by the TAS  as (Value (n) – Value (0) ) / (Time (n) – Time(0)), and reports the average rate over the whole test and up to the interval. The P-I value shows the actual rate that occurred within a given interval.

 

Average Session Disconnect Time

 The average time — measured from initiation to the release of the final connection — required to successfully deactivate an MN session. This value is averaged for all successful disconnects across all protocol layers.

 

^ Back to Top

 

Data Throughput

Total Data Sent + Received Packets/Sec

 The total number of sent and received Packets per second.   This measurement is available in MME Nodal, SGW Nodal, PGW Nodal, GGSN Nodal, Wifi Offload Gateway Nodal, IP Application Node and Network Host test cases with Local or Remote network host.

Total Data Sent + Received Bits/Sec

 The total number of sent and received Bits per second. This measurement is available in MME Nodal, SGW Nodal, PGW Nodal, GGSN Nodal, Wifi Offload Gateway Nodal, IP Application Node and Network Host test cases with Local or Remote network host.

 

^ Back to Top

MN Session Counters

The following measurements report current session status, and the number of MN session connects and disconnects through the life of the test. See the following sections for application-specific or additional session measurements:

Authentication Sessions Established

The number of MN sessions currently established. As MN sessions transition from an idle state to an active state, this measurement increases as Authentication Sessions Pending decreases. The reverse is true when sessions transition to an idle (pending) state.

Authentication Sessions Pending

The number of MN sessions that are currently in an idle state. As MN sessions transition to an active (established) state, this measurement decreases as Sessions Established increases. The reverse is true when sessions transition to an active state, or to the disconnected state at the end of the test.

Session Errors

The number of MN session errors generated during the test, including errors from all reported protocol layers.

Attempted Session Connects

The number of MN sessions that the test has attempted to connect.

TIP: In a Capacity Test, this value should always match the MN sessions defined. When a session loading activity is used this value should be the number of sessions times the number of session loading runs.

Attempted Session Disconnects

The number of established MN sessions that the test has attempted to disconnect. Disconnects that were initiated by a SUT are not included in this measurement. Assuming that the test initiates all disconnects, this value will not match Attempted Session Connects if all sessions were not successfully established, but it should correspond to Actual Session Connects at the end of the test.

Actual Session Connects

The number of Attempted Session Connects that were successful. This value will not match Attempted Session Connects if all sessions were not successfully established, but it should correspond to Actual Session Disconnects at the end of the test.

Actual Session Disconnects

The number of established and pending MN sessions that have been successfully disconnected regardless of which entity initiated the disconnect. At the end of the test the pending sessions, including those that could not be established due to an error, are disconnected. At this point the value should match Attempted Session Connects.

NOTES:

  • The sum of Sessions Established and Sessions Pending will not necessarily equal the total number of MN sessions at any given point in time during the test. Some sessions could be in a transitory state when the measurements are collected.

  • If a session cannot be established, it is left in the pending state.

  • Session Errors will not necessarily match the difference between Attempted Session Connects and Actual Session Connects, as a session can fail due to multiple errors.

^ Back to Top

Handoff and Handoff Rate Test Measurements

The following measurements are available when a handoff , handoff rate , or Session Loading With Mobility test is executed.

Actual Handoff Rate

The actual rate at which MN sessions were successfully handed off in the previous interval. This measurement is not displayed until the first session handoff is attempted.

Handoff Attempts or Returning Home Attempts

The number of MN session handoffs that have been attempted during the test.

NOTE: The sum of successes and failures will not necessarily equal attempts at any given point in time during the test. Some sessions could be in a transitory state when the measurements are collected.

Handoff Successes or Returning Home Successes

The number of MN sessions that have been successfully handed off.

Session Not Ready for Handoff

The number of times a handoff attempt is made when the MN session cannot be handed off due to the following reasons:

If the test performs continuous handoffs, another attempt to handoff the session will be made when the session is ready. If the test performs a single handoff, no further attempt will be made to handoff the session.

Handoff Failures

The number of failures that occurred while processing MN session handoffs. This measurement is not displayed until the first session handoff is attempted.

^ Back to Top

Revocation Measurements

The following measurements are available when a MIP Revocation test is executed.

Actual Revocation Rate

The actual rate at which MN registrations were revoked in the previous interval. This measurement is not displayed until the first revocation is attempted.

Revocation Attempts

The number of revocations that were attempted. This measurement is not displayed until the first revocation is attempted.

Revocation Successes

The number of MN registrations that were revoked. This measurement is not displayed until the first revocation is attempted.

Revocation Failures

The number of revocation failures. This measurement is not displayed until the first revocation is attempted.

Sessions Not Ready For Revocation

The number of times a revocation attempt is made when the MN session has not been fully established, indicating that the test is trying to initiate revocations faster than the SUT can establish a session. No further attempt will be made to revoke the session and when it is established, it will remain in the established state.

^ Back to Top

AAA Testing

The following measurements are available when a AAA test case is used.

Accounting Started Sessions

The number of accounting sessions currently established. As accounting sessions are started, this measurement increases.

Accounting Stopped Sessions

The number of accounting sessions that have been stopped in the interval. As accounting sessions are stopped, this measurement increases until all sessions have been stopped and then it is reset to 0.

NOTES:

  • The sum of Accounting Started Sessions and Accounting Stopped Sessions will not necessarily equal the number of MN sessions while sessions are stopping. Some sessions could be in a transitory state when the measurements are collected.

  • When you use a short accounting message cycle — shorter than the time required to establish all of the MN sessions — Accounting Stopped Sessions can begin to increment before all of the accounting sessions are established. Since the message cycle for each MN session uses an independent timer, the Accounting Started Time can expire in the oldest sessions before all of the sessions have been established.

 

^ Back to Top

AMF /SMF Testing -5G

Actual Session Registers

Actual Session Deregisters

Actual Session Register Time

Actual Session Deregister Time

Actual Session Register Time (P-I)

Actual Session Deregister Time (P-I)

Session Registers  (Sessions/Second) (P-I)

 Session Deregisters (Sessions/Second) (P-I)

NOTES:

In AMF Nodal - Test Summary, Average Session Register/De-register Time of MM based session will be shown:

  • Session Register Time : Time taken by UE - NAS 5G (MM) Layer to get registered.

  • Session De-register Time : Time taken by UE - NAS 5G (MM) Layer to get de-registered.

  • AMF Nodal won't report "Average Session Connect/Disconnect Time"

In SMF Nodal - Test Summary, Average Session Time of SM based session will be shown:

  • Session Connect Time : Time taken by UE - IP/SM/Nsmf to get established.

  • Session Disconnect Time : Time taken by UE - IP/SM/Nsmf to get released.

 

Service Based Nodal / Node Testing - 5G

Connect Time  -  T1 = the point in time when the first request message is sent to bring the session up; T2 = the point in time when the last response message is received to bring up the session. The Connect Time = T2 – T1.

 

Disconnect Time  -  T1 = the point in time when the first request message is sent to tear down the session; T2 = the point in time when the last response message is received to tear down the session. The Disconnect Time = T2 – T1.

^ Back to Top

ASN Nodal Testing

The following measurement is available in an ASN Nodal test.

Sessions Network Idle

The number of mobile stations that are currently in an idle state.

^ Back to Top

CDMA2000 Testing

The following measurements are available in the End-to-End Mobile IP, FA Nodal, Simple IP, and Simple IP VPN test cases.

Main Sessions Established

The number of main service instances currently established. As MNs establish their main service instances, this measurement increases as Sessions Pending decreases. The reverse is true when sessions transition to an idle (pending) state.

Auxiliary Sessions Established

The number of auxiliary service instances currently established. As MNs establish their auxiliary service instances, this measurement increases as Sessions Pending decreases. The reverse is true when sessions transition to an idle (pending) state.

^ Back to Top

IP Data Testing

The following measurements are available when the IP Application Node or PDG Micro-Mobility test cases are used.

Client Data Nodes Established

The number of MNs that are currently sending or receiving data packets. As MNs transition from an idle state to an active state, this measurement increases as Client Data Nodes Pending decreases. The reverse is true when MNs transition to an idle (pending) state.

Client Data Nodes Pending

The number of MNs that are currently in an idle state. As MNs transition to an active (started) state, this measurement decreases as Client Data Nodes Established increases. The reverse is true when sessions transition to an idle state.

Basic Data Flow Resources Started

The number of DMFs that have been successfully started when one of the Basic Data protocols is used. This measurement will increment as each DMF for a Client Node is successfully started.

Enhanced Data Flow Resources Started

The number of DMF transactions that have successfully executed for all Client Nodes when one of the Advanced Data protocols is used. This measurement will increment as each transaction is executed.

MPD Files Sent

The number of MPD files that have been successfully sent. This measurement will increment as each MPD file successfully sent.

Total Media Segments Sent

The number of Media Segments that have been sent. This measurement will increment as each Media Segment is Sent. This includes segments sent with Error.

Good Media Segments Sent

The number of Media Segments that have been successfully sent. This measurement will increment as each Media Segment is successfully Sent.

Error Media Segments Sent

The number of Media Segments that have been successfully sent with Error. This measurement will increment as each Media Segment is Sent with error.

^ Back to Top

L2TP Testing

The following measurements are available when L2TP is used in a test.

L2TP Tunnels Established

The number of VPN tunnels currently established. As tunnels transition from an idle state to an active state, this measurement increases as L2TP Tunnels Pending decreases. The reverse is true when tunnels transition to an idle (disconnected) state.

L2TP Tunnels Pending

The number of VPN tunnels that are currently in an idle state. As tunnels transition to an active (established) state, this measurement decreases as L2TP Tunnels Established increases. The reverse is true when tunnels transition to an idle (disconnected) state.

NOTES:

  • The sum of L2TP Tunnels Established and L2TP Tunnels Pending will not necessarily equal the number of MN sessions at any given point in time during the test. Some tunnels could be in a transitory state when the measurements are collected.

  • If a tunnel cannot be established, it is left in the pending state.

^ Back to Top

SGW Nodal Testing

Average Connect

The Average Connect is calculated as each session is established. When a session starts, the establishment procedure saves/makes a note of the start time. The session is considered established when all control messages (on S11/S5 tabs) required to bring the session and bearers up have been exchanged. The elapsed time is then calculated and noted/saved. The elapsed time is summed as each session establishes and the average connect time is calculated as the sum of the elapsed time(s) dived by the number of sessions established.

The average connect time is = The sum of the elapsed time(s) / The number of sessions established.

Average Disconnect

The Average disconnect is calculated as each session is torn down. When a session ends, the disconnect procedure saves/makes a note of the end time. The session is considered disconnected when all control messages (on S11/S5 tabs) required to bring the session and bearers down have been exchanged. The elapsed time is then calculated and noted/saved. The elapsed time is summed as each session tears down and the average disconnect time is calculated as the sum of the elapsed time(s) dived by the number of sessions disconnected.

The average disconnect time is = The sum of the elapsed time(s) / The number of sessions disconnected.

See also Average Connect/Disconnect measurements on S11/S5 tab.

Radio Loss Enabled

Available when Pause Charging on S1 Release on SGW Nodal ODC is enabled. Used for "Pause Charging due to Radio Loss". Per 3GPP TS 29.274, 13.5.0 clause 8.12.  

 

Actual Session Connects Including Dedicated Bearers

Available when UE Initiates Dedicated Bearers on SGW Nodal Test Cnfg is enabled. Indicates the actual number of session connects including Dedicated Bearers. The UE is considered connected successfully when all bearers are established. 

Actual Session Connect Rate Including Dedicated Bearers (sessions/Second)

Available when UE Initiates Dedicated Bearers on SGW Nodal Test Cnfg is enabled. Indicates the actual session connect rate (in Session per Second) including Dedicated Bearers. The UE is considered connected successfully when all bearers are established.

 

^ Back to Top

UMTS Testing

The following measurements report RNC link status, MN connection and session status, and MN rates for UMTS testing.

Mobile Nodes Connected

The number of MNs that are currently attached to the SGSN and that have an active Iu connection. As MNs transition from an idle state to an attached state when the test begins, this measurement increases as Mobile Nodes Idle decreases. If Iu connections are released, this measurement decreases and MNs transition to the idle state, increasing Mobile Nodes Idle.

When MNs are detached, this measurement decreases as both Mobile Nodes Idle and Mobile Nodes Detached increase. If an MN is idle, its Iu connection is re-established in order to process the detach, and this measurement will show the connections becoming active before the detach.

Mobile Nodes Idle

The number of MN sessions that are currently in an idle state. As MNs are initially attached to the SGSN when the test begins, this measurement decreases as Mobile Nodes Connected increases. If an MN cannot be attached, it remains in the idle state.

Mobile Nodes Detached

The number of formerly connected MNs that have been detached from the SGSN. As MNs are detached, this measurement increases as Mobile Nodes Connected decreases.

Attempted Mobile Node Connects

The number of MNs that the test has attempted to attach to the SGSN.

TIP: In a Capacity Test, this value should always match the number of MNs defined. When a session loading activity is used this value should be the number of MNs times the number of session loading runs.

Attempted Mobile Node Detaches

The number of attached MNs that the test has attempted to detach. This value will not match Attempted Mobile Node Connects if all MNs were not successfully attached, but it should correspond to Actual Mobile Node Connects at the end of the test.

Actual Mobile Node Connects

The number of Attempted Mobile Node Connects that were successful. This value will not match Attempted Mobile Node Connects if all MNs were not successfully attached, but it should correspond to Attempted Mobile Node Detaches at the end of the test.

Actual Mobile Node Detaches

The number of attached MNs that were successfully detached. This value will not match Attempted Mobile Node Connects if all MNs were not successfully attached, but it should correspond to Actual Mobile Node Connects at the end of the test.

Attempted Mobile Node Connect Rate

The rate at which the test attempted to attach MNs to the SGSN in the previous interval, as defined by Attach Rate.

Attempted Mobile Node Detach Rate

The rate at which the test attempted to detach MNs from the SGSN in the previous interval, as defined by Detach Rate.

Actual Mobile Node Connect Rate

The actual rate at which MNs were successfully attached to the SGSN in the previous interval. A significant variance between this rate and the Attempted Mobile Node Connect Rate can indicate communication delays with the SUT or a heavy load on the SUT CPU.

Actual Mobile Node Detach Rate

The actual rate at which MN sessions were successfully detached from the SGSN in the previous interval. This measurement is not displayed until the first detach is attempted. A significant variance between this rate and the Attempted Mobile Node Detach Rate can indicate communication delays with the SUT or a heavy load on the SUT CPU.

Attempted PDP Activation Rate (Sessions/Second)

The rate at which the test attempted to establish PDP contexts with a GGSN in the previous interval, as defined by PDP Activation Rate.

Attempted PDP Deactivation Rate (Sessions/Second)

The rate at which the test attempted to deactivate established PDP contexts in the previous interval, as defined by PDP Deactivation Rate.

Actual PDP Activation Rate (Sessions/Second)

The actual rate at which PDP contexts were established with the GGSN in the previous interval. A significant variance between this rate and the Attempted PDP Activation Rate can indicate communication delays with the SUT or a heavy load on the SUT CPU.

Actual PDP Deactivation Rate (Sessions/Second)

The actual rate at which established PDP contexts were successfully deactivated in the previous interval. This measurement is not displayed until the first deactivation is attempted. A significant variance between this rate and the Attempted PDP Deactivation Rate can indicate communication delays with the SUT or a heavy load on the SUT CPU.

PDP Contexts Active

The number of PDP contexts currently established with the GGSN. As contexts are established, this measurement increases as PDP Contexts Available decreases. As contexts are deactivated, this measurement decreases as PDP Contexts Available increases.

PDP Contexts Available

The number of PDP contexts that are currently in an idle state. As contexts are established, this measurement decreases as PDP Contexts Active increases. If a context cannot be established, it remains in the idle state.

Attempted PDP Activations

The number of PDP contexts that the test has attempted to establish with the GGSN. If all MNs successfully attach to the GGSN, this value should match the total number of PDP contexts defined for the test.

Attempted PDP Deactivations

The number of established PDP contexts that the test has attempted to deactivate. This value will not match Attempted PDP Activations if all contexts were not successfully established, but it should correspond to Actual PDP Activations at the end of the test.

Actual PDP Activations

The number of Attempted PDP Activations that were successful. This value will not match Attempted PDP Activations if all PDP contexts were not successfully established, but it should correspond to Attempted PDP Deactivations at the end of the test.

Actual PDP Deactivations

The number of established PDP contexts that were successfully deactivated. This value will not match Attempted PDP Activations if all PDP contexts were not successfully established, but it should correspond to Actual PDP Activations at the end of the test.

Average GMM Attach Time

The average time — measured from transmission of the first request to receipt of the final acknowledgement — required to successfully attach an MN. This value is averaged for all successful attaches.

Average GMM Detach Time

The average time — measured from transmission of the first request to receipt of the final acknowledgement — required to successfully detach an MN. This value is averaged for all successful detaches.

Average PDP Activation Time

The average time — measured from transmission of the first request to receipt of the final acknowledgement — required to successfully activate a PDP context. This value is averaged for all successful session activations.

NOTE: In a Data Verification test, a session is not considered to be connected until data connectivity has been verified. This can significantly increase Average PDP Activation Time.

Average PDP Deactivation Time

The average time — measured from initiation to the release of the final connection — required to successfully deactivate a PDP context. This value is averaged for all successful deactivations.

RNC Established

The number of RNC links currently established with the SGSN. As links transition from an idle state to an established state when the test begins, this measurement increases as RNC Idle decreases. As links are torn down at the end of the test, this measurement decreases as RNC Idle increases.

RNC Idle

The number of RNC links that are currently in an idle state. As links are established with the SGSN when the test begins, this measurement decreases as RNC Established increases. If a link cannot be established, it remains in the idle state.

^ Back to Top

WiMax Testing

The following measurements are available in a WiMax Nodal test.

BS Links Established

The number of Base Station links currently established with the WAC. As links transition from an idle state to an established state when the test begins, this measurement increases as BS Links Pending decreases. As links are torn down at the end of the test, this measurement decreases as BS Links Pending increases.

BS Links Pending

The number of Base Station links that are currently in an idle state. As links are established with the WAC when the test begins, this measurement decreases as BS Links Established increases. If a link cannot be established, it remains in the idle state.

Sessions Network Idle

The number of mobile stations that are currently in an idle state.

^ Back to Top

WiFI Testing

The following measurements are available in a Wifi Offload Gateway Nodal and in some cases, SGW Nodal)

Access Points Established (Available in WiFi and SGW Nodal Test cases)

The number of Access Points that are currently connected to the WiFi Gateway. As Access Points transition from an idle state to a connected state when the test begins, this measurement increases as Access Points Idle decreases.

Access Points Idle (Available in WiFi and SGW Nodal Test cases)

The number of Access Points that are currently in an idle state. As Access Points are established with the WiFi Gateway, this measurement decreases as Access Points Established increases. If an Access Point cannot get connected to WiFi Gateway, it remains in the idle state.

Average Handoff Time (Available in WiFi Nodal Test case only)

The average time - measured from mobility from source AP to successfully moved to target AP.

^ Back to Top

SUT Query Option

SUT CPU Utilization

The percent of CPU usage on the SUT as reported using SUT Query.

^ Back to Top

© 2024 Spirent Communications Inc. All Rights Reserved.