Customizing Report Options


By default, a Test Report is saved to your results directory on the TAS whenever a test session that has successfully started is completed. You can customize the Report options for any test session, specify the granularity and format of the Test Report (what to subtotal) and choose to also generate a Per Session Report for the test session or periodically save a Test Report during the test.

TIP: You can also capture a Test or Per Session Report while the test is running.

Sections:

 

To configure the Report options:

Select the Reports tab..., before executing the test session, and click the Options button. The Reporting Options window opens. The button is disabled while the test is running.

Select the report level from the View drop-down list. You can capture a Summary report, individual reports for All Test Cases, or Summary and Each Test Case in Separate Files for a individual test cases in the list or Summary and Each Test Case in One File.   

Select the file type, Excel (.xls) or Text (.csv), from the Format drop-down list.

The selected file type saves all counters from a test session into one file when you select the Summary and Each Test Case in One File.

The Excel (XLS) option saves tests with 10 or fewer test cases that has requested a maximum of 1000 intervals, due to memory constraints.  

The CSV option does not have any limitations since this is generated in a memory efficient manner.

NOTE: To protect against Out Of Memory errors, the TAS checks the memory usage when saving a report and stops adding test cases to an Excel file if less than 100M of memory is available.  This results with a .xls file (Summary and Each Test Case in One XLS file) that is missing some test cases and/or the config data.  A Report Log will also be generated in such cases.  

 

Per Interval: Select to set the report values to per-interval. If the box is cleared, cumulative report values will be captured. The cumulative report also includes the Per-Interval values for Rate and Div (calculated) measurements.

You can calculate per-interval values for the normal counter measurements by subtracting the consecutive interval values from the cumulative values provided in the report.

NOTE: The cumulative rate is calculated by the TAS as  (Value (n) – Value (0) ) / (Time (n) – Time(0)), and reports the average rate over the whole test and up to the interval. The P-I value shows the actual rate that occurred within a given interval. See also L5-7 Per Interval Measurements.

Auto Save Every: Select to periodically save your Test Report during the test. By default, a Test Report will be saved every 4 hours. Modify the Hours field to change the report cycle.

NOTE: When you enable AutoSave, the TAS will save the TAS-Generated results (XLS/CSV) to the Results Website using same name that you get when the test completes, on a regular period that you define. You will get a Real-Time Log and Run log that indicates the Auto-Save is initiated.

 

Interval Period:  Set the Interval Period that determines the reporting frequency. By default, the interval period you select is based on your Test Elapse Time.

Options: 5 (s), 10 (s), 15 (s), 30 (s), 1 (m), 2 (m), 5 (m), 10 (m), 15 (m)

Default : 15 (s)

NOTEs:

  • When you select intervals less than 15s with large numbers of test cases or high performance tests, your Test Server, TAS and/ or your network can be more easily overloaded.
  • When you select intervals less than 15s your saved history time will be reduced. For example, for 5s interval period with 5 test cases, your maximum storage will be approximately 30 hours initial storage versus 80 hours (if configuration is set at 20000 intervals) for 15s interval.

 

For example, if a test is started at 14:02, Landslide reporting is based on the test starting time, and if every 5 min interval period is selected, then the next report interval is 14:07, 14:12, and so on.

Use actual time to mark intervals: Select to indicate that the TAS / System time is used to determine the Interval boundaries/reporting frequency.

When not selected, the reporting interval is based on the Test Elapsed time (default)

For example, if a test is started at 14:02, reporting is based on the actual time, and every 5 min interval period is selected, then the next report interval is 14:05, 14:10, and so on.

Select Generate a zip file containing the test case diagrams, click OK to save this setting and then from the Test Case Editor, click OK  if you wish to save the network work diagram within the test case.

When Generate a zip file containing the test case diagrams is selected, then anytime you click OK on a Test Case editor that has a network diagram, that diagram is saved along with the test session.

NOTE: Selecting just the Generate a zip file containing the test case diagrams option does not save the images. The images are saved only when you Click OK on a Test Case editor when Generate a zip file containing the test case diagrams is already selected on the Reporting Options window.

The Test Case network diagram is saved in the test session as jpg files compressed into a TEST_NAME_PREFIX.tcdiagrams.zip file and included with the rest of the test results, see the included examplesee the included example.

For example:

05/18 14:15:31.004:TAS: Generating final report(s)...

log: 05/18 14:15:40.074:TAS:Test is completed, result files:

05-18_02.12.09.PM__BasicTrafficTest_diagrams__RID-1.log.txt

05-18_02.12.09.PM__BasicTrafficTest_diagrams__RID-1.tcdiagrams.zip

05-18_02.12.09.PM__BasicTrafficTest_diagrams__RID-1.xls

Requested maximum Number of Intervals: Select to indicate the maximum number of intervals reported in the database.

The following is a recommended intervals based on the number of tests cases per 15 (s) Interval Period:

  • 100 for <= 300 test case, 0.41 hours initial storage

  • 250 for <= 250 test case, 1.04 hours initial storage

  • 500 for <= 200 test case, 2.08 hours initial storage

  • Default / 1000 for <= 100 test case, 4.16 hours initial storage

  • 2000 for <= 50 test case, 8.33 hours initial storage

  • 5000 for <= 15 test case, 20.83 hours initial storage

  • 10000 for <= 10 test case, 41.66 hours initial storage

  • 20000 for <= 5 test case, 83.33 hours initial storage

NOTEs: When you select intervals less than 15s your saved history time will be reduced. For example, for 5s interval period with 5 test cases, your maximum storage will be approximately 30 hours storage versus 80 hours (if configuration is set at 20000 intervals) for 15s interval.

Options 100 and 250 can be used for Live Network testing which only run for 5 to 15 minutes. 5s intervals over 15 minutes needs 180 intervals. 15s intervals over 5 minutes needs just 20.

 

A warning displays if you set up above the recommended limits that could result in TAS running out of memory.

The Initial Storage information displayed is an estimate of how long the test will run before deleting half of the intervals and doubling the interval time, based on initial 15second intervals and the number of requested intervals.

Estimated Measurement Data Points Required: NNNNN

Shows an estimate of how many measurement data points the current test could possibly use based on Requested Maximum Number of Intervals you selected. This is for your information purposes only.

The number is the minimum number of bytes the TAS will use when initializing the test session.  

Select Generate TAC to create a after test runs, server-side report on TAS. See About Test Reports for additional methods for generating TAC report.

Select Only when CriteriaStatus is FAILED. When enabled the end of test TAC report would only be generated  if you had configured PF Criteria and the final status was "FAILED". Otherwise no TAC report is generated. The System wide, force-tac-report TAS setting overrides this and will force a TAC Report.

For example: TestResultsName.tac.zip.

 Filename Created Size (bytes) Pass/Fail
07-25_04.36.57.AM__Basic_Sequencer__RID-2.log.txt 2013-07-25_04:41 44719 N/A
07-25_04.36.57.AM__Basic_Sequencer__RID-2.tac.zip 2013-07-25_04:41 323886 N/A
07-25_04.36.57.AM__Basic_Sequencer__RID-2.xls 2013-07-25_04:41 778752 N/A

The Generate TAC Report saved on TAS has the following characteristics:

  • Does not include Client side debug logs or logs as in TAC Reports from a Client.
  • File names match real results files on TAS
  • Generated files are full license .sql files.
  • Files are generated as per your selection .xls or .csv (on the Report Options window).
  • Three latest TAS debug logs are available: tas.0.log, tas.1.log, and tas.2.log.
  • All TS-Generated Results files are available (TS-generated files larger than 10M may not be available).
  • Full log files are available on the Test Results page.
  • Includes the .tsConfig file
Disable End of Test Retrievals from Test Servers - Select to prevent delaying end of test logic to wait for SSH connections, i.e. if no TS generated results are needed, including PCAP captures. Provides the ability to turn "OFF" the TS generated result file transfer at the end of test.

Generate HTML Report: Select to generate an HTML report (post-test). Select Use Results Manager Template to include the Results Manager Template in the HTML report.

Attributes GenerateHtmlReport and HtmlRptUseTemplate are also available via RESTful and Tcl APIs. 

The report is self-contained and shareable via email. Report does not require meas.db generation.

The report is based on one or more of the following:
   a) Results Manager Template.
   b) Favorite Measurements and Charts configured in the test.
   c) Top level Test details, name, description, run times, etc..

File is named using the same pattern as Run Log, XLS, ending with .html.
 <START_TIMESTAMP>_RUN-<RID>_<NAME>.html.

Example: 20-12-11_14.21.15_RID-9_SMFNodal_4_MT_Scenario_with_4_portsERK_b2b.html

Additional details in Landslide Result Manager templates in Test Sessions document.

Generate End-of-Test Database: Select to save a test report database (meas.db) that can be opened and viewed with the Landslide Results Manager report viewer.

NOTE:   

 The database is written out at the end of the test, based on the intervals stored in memory. This could increase the time it takes to complete a test session.

The Run Log and Real-Time Log will contain messages about Database generation. To help minimize the impacts, you can use Report Filters.

The database generation will only write measurements that match the initial Report Filters for the test. If you want to generate the database for a very large test, setup report filters that show only the required measurements.

Template File - With Generate End-Of-Test Database or Use Results Manager Template enabled, and a Template file selected and defined, the TAS will automatically load the report into a results manager and generate the HTML file. You must create the template in Report Manager, then load it as you would load a TDF into the TAS and select it for the test. Then Report Manager would need to be running on the TAS as a service of sorts, and have a RESTful API that the TAS can call. Additional details in Landslide Result Manager templates in Test Sessions document.
 

VisionWorks Results: Select to enable Landslide Test Analytics Reporting. Additional configuration details can be found in the topic VisionWorks Solution. This is a licensed feature.

NOTEs:

  • To enable all the features, must have UE Info Level = 10 on Test Server Configuration and VisionWorks Results enabled on Report Options. For USER PLANE, must add the correct DMFs and enable L4-7 Per Session End of Test results.
  • May get NULL Values for items that are not reported by your test case. When setting a UE Info Level X (1 to 10), that means that the UE Info component will include fields A, B, and C. But that does not mean the test will always report B or C, thus the UE Info report will show a column for B or C and it would be null. for example, if using level 9 and you do not do a POLQA on Default Bearer, you will get POLQA columns, but they will always be null.
  • When VisionWorks Results reporting is enabled, all of the Pass Fail Criteria and UE Info Criteria in the test session are included in the VisionWorks TEST.CRITS result file and additionally there are Test Case records summarizing each test case and a Test Session record summarizing the test session, as per the Data Model. The user can assign a SUT to a given Test Case or Test Session by putting the SUT name in the Test Case or Test Session name after ##, e.g. MyTestName##MySutName. If the SUT name is found in the TAS by name, the TAS will also populate the SUT_TYPE and SUT_IP with the associated values found in the TAS DB associated with that SUT. Test setup details can be found in the Visionworks Solution Topic.
  • The TEST.CRITS files will always be written out, even if the test does not start. (VisionWorks Results reporting must be enabled). The file will include the TEST_INIT_FAILED record.

Intent of Test : Choose the Service Assurance Test Type that Spirent VisionWorks Analytics should use to determine displays. The Intent of Test can be set specifically for individual test cases. See additional details in About the Test Case Settings Window.   Options: Data, Enterprise, VoLTE , etc   Data - Select for Date Service Assurance Web Performance Data Analytics. Enterprise - Select for Enterprise Service Assurance Machine-To-Machine Data Analytics. Configuration details can be found in the topic VisionWorks Solution. VoLTE- Select for VoLTE Service Assurance - Voice Quality Service monitoring Analytics.   SMS-MMS- Select for SMS-MMS Service Assurance - SMS-MMS monitoring Analytics. Network-Latency- Select for Network-Latency Service Assurance - Network Latency monitoring Analytics. OTT-Apps - Select for OTT-Apps Service Assurance - OTT Apps Quality Service monitoring Analytics. (Example : INSTAGRAM, YOUTUBE, etc). OTT-Video- Select for OTT-Video Service Assurance - OTT-Video monitoring Analytics. NB-IOT - Select for Device Type support on MME Nodal Test Cnfg (CIoT), S1-MME Interface - NAS Tab Emerg-Svcs - Select for Emergency Services Service Assurance - Emergency Services monitoring Analytics. EXT_APPS_VPN - Select for UE emulation to a customer specified VPN server using OpenVPN - Available on MME Nodal test case -  External App - OpenVPN. INTERCARRIER - Select for MMS Client for MMS and EMAIL messages - Available on MME Nodal test case -  External App - MMS Client. VoNR - Select for VoNR Service Assurance - VoNR monitoring Analytics. Available on AMF Nodal N2/N3 and N6 support for monitoring and activation and IP Application Node for fault isolation for VoNR to separate issues between IMS core and 5G core. Voice-EPS - Select for Voice-EPS Service Assurance - Voice-EPS monitoring Analytics. Video-Streaming - Select for Video-Streaming Service Assurance - Video-Streaming monitoring Analytics. Conference - Select for Conference Service Assurance - Conference monitoring Analytics. Custom- (future use case).

Note: When the user selects a Global Forced Intent of Test (e.g. Conference, SMS-MMS) in the Test Session Intent of Test, that will force every single record to use that Intent of Test. All other rules will not matter. 

Expose VisionWorks Results to Analytics: Automatically enabled. Select to copy VisionWorks files to the Analytics transfer folder, deselect if you do not want results sent to Analytics. Unchecking this box will put the files in the TAS results Area. The APIs flag is called VisionWorksCopy.
Tags - VisionWorks Tags - Also available on the Test Case Settings Window
  • Select Tags to enter the VisionWorks Tags. A Pop-up window allows you to capture Tag #, Name and Value.

 

  • The test case defined Tags will have higher precedence than the test session. Any Tags left blank in the test case will end up using the Tags from the test session. For example given: Test Session TAG1=Region=EAST,  TC0 TAG1=Region-WEST, TC1 TAG1=<empty>, records produced by TC0 will have TAG1=Region=WEST and records produced by TC1 will have TAG1=Region=EAST, and any TEST.CRITS records that only apply at the test session level will have TAG1=Region=EAST.
  • TAG Names can be up to 32 letters, numbers or underscores (_), but cannot start with a number.
  • TAG Values can be up to 32 characters (with some special characters allowed, as indicated in the GUI).
  • There are some predefined built-in TAG names available to select from (Network_TypeSUT_NAME, Analytics_Module, ST, VoLTE_Type, Region, LCCOption,  and Type), and a Custom tag name entry is available. Analytics_Module (5G) and Network Type values (SGi, N6) are required for IP Application Node based tests.  SUT_NAME is required to display the SUT in VisionWorks Test Case Analytics.  
    • Network_Type - SGi value for SGi interface (EPC) and N6 for N6 interface (NGC) for those respective KPIs. 
    • SUT_NAME - Identify a SUT name in the Test Case Analytics context and link it to an entity through reference data (entity id field)
    • Analytics_Module - 5G (used by Analytics for CRIT and PROC.)
    • VoLTE_Type - GETS (used by Analytics in the context of GETS call)
    • ST - used by Analytics in the context of Speedtest. KPIs for Speedtest server autoselection will look for this tag to be set to Auto vs KPIs for Speedtest server preset in test case will look for this tag to be set to Preset
    • Region, LCC, Option, Type, Network_type - Any User Input

 

TS CPU % Reporting - Select an option for CPU reporting. Options : AVG TS CPU %, AVG TS-Process CPU % , AVG TS-CORE CPU %. Additional details can be found in TS CPU % Measurements document. TS CPU Percentage Measurements.
 

Select Sub-Totals by L4-7, L3, Mobile Access Transport, and/or Core Network Nodes.   For Layer 4 - 7. Select an options (None, Mobile Node, Subscriber Group or DMF) from the L4-7 drop-down list.

Landslide standard live measurements provide KPIs that are the total values for all objects that are being emulated and all SUTs being tested within a test case. There is a Summary View that is the totals across all test cases in the Test Session. The user can also enable the reporting of subtotals in some specific protocols and dimensions. On the report options, there are three sections related to this:  Subtotals, Per Session and Per Mobile Node Options and TS Generated Per MN/Session Report(s):

The top section Subtotals controls what subtotal dimensions will be reported by the Test Servers. Checking boxes or choosing values in the combo boxes will enable specific subtotals to be reported. When reporting subtotals, the Live measurements will not support large numbers of subtotals.Only the first 8-12 subtotals will be sent to the live measurements, but all subtotals will be included in end of test files if enabled. Also if you enable multiple dimensions for a given layer, only one dimensional subtotal is sent to the live measurements, but the end of test results, if enabled, will include multiple files, one for each dimension.

The middle section Per Session and Per Mobile Node Options, lets you filter subtotals for just a particular range of Sessions or Mobile Nodes. If for example you are only interested in the live reporting of the subtotals you can see the range starting a the 500th UE so UE 500-509 are reported to the GUI. Note: Filtering Sessions does not affect Per-Bearer or per Network-Slice since these both only a few subtotals each.

The bottom section TS Generated Per MN/Session Report(s) control whether the Test Server will generate end of test files for these subtotals. If enabled the Test Server will generate separate files for each subtotal dimension and layer and will include ALL subtotals that occurred during the test as specified in Per Session and Per Mobile Node Options section.      

Note : Enabling Subtotal result without Per Session measurements is not a valid configuration.

Table showing which files the sub-totals are reported in when TS Generated results are enabled is located below.

Select the Per Session checkbox for L4-7, L3, Mobile Access Transport, and/or Core Network Nodes, and subtotals Per Session will be generated as per the option in Step 6. See additional details above.

NOTE:

  • Enabling per-session logging may result in a low system performance (depending on your test session/test suite configuration, number of nodes, DMFs, and so on).
  • Fireball DMFs only support Per-DMF Subtotals.     

The subtotal (Per-Session) selected affects both what you see in the GUI Reports and what you get in the TS Generated report. In the GUI Reports:

Displays subtotals for only one option for any given measurement/layer.  That is:

  1. DMF from the L4-7 dropdown list

    If you select DMF from the L4-7 dropdown list  the report displays the DMF subtotals of only the first few DMFs based on the range you specify on the Per Session and Per Mobile Node Options pane on the Report Options dialog box or for All Sessions on the GUI.

    When Sub-totals are being displayed, only the first 10 to 12 of DMF, Subscribers, Bearers, UEs, etc (what ever is being sub totalled) will show up in the GUI reports.

    Additional details / limitations in the About the Reports Tab - DMF Names

    NOTE:

    DMF Subtotals are based solely on the Mainflows list plus additional DMFs from Volte/eMBMS, not the Instances on Assignments tab.  Starting with DMF_0 would be the first DMF in the Mainflows list. For example if you had 3 Mainflow DMFs plus running Gm Volte RTP, you could have DMF_0 thru DMF_2 from Mainflows, and DMF_3 would be the VoLTE RTP subtotal. Subflow totals do not support additional instances of a DMF.  If you add extra Instances of DMF Mainflows on the Assignment tab the data shown in the subflow tabs will not be accurate.

    Client Per PDU - Available on AMF Nodal when L4-7 = DMF. Select for  “Basic” and “ENHANCED” class DMF types to support L4-7 client per-PDU sub-subtotal measurement.

    L4 Client and L5-7 Client, each DMF subtotal is divided into several sub-subtotals based on PDU#, like DMF_0 will be divided into several table, like DMF_0-PDU_0, DMF_1-PDU_1, DMF_2-PDU_2, etc. 

    Example:

    Mobile Node L4-7:

    Per Session

    If you select Mobile Node L4-7 dropdown list, the report displays subtotals of only the first few Mobile Nodes or Sessions based on the range you specify on the Per Session and Per Mobile Node Options pane on the Report Options dialog box or for All Sessions or All Mobile Nodes.

    When Sub-totals are being displayed, only the first 10 to 12 of DMF, Subscribers, Bearers, UEs, etc (what ever is being sub totalled) will show up in the GUI reports.

    Subscriber Group from the L4-7 dropdown list

    Available in DRA Nodal Test Case.

    When per Subscriber Group (UE group Statistics) will be enabled then, all the message counters will be pegged in their respective UE group tabs along with the Total counters under “Totals” Tab which is under DRA Session Main tab. The “Totals” and “Group_X (where X varies from 1 to 10)” tabs will be automatically created at run time under DRA Session Main Tab in Reports as like other test cases e.g. OCS Node Profiles, PCRF Node Monitors etc.

    NOTE: The group specific measurements with be shown in the corresponding tabs as per Group Index like 1, 2, 3…..up to total number of groups. So measurements for first group will be displayed under Group_1 tab and so on.

    For example, suppose if there are 3 UE groups then there will be 4 tabs created at run time under DRA Session Main tab as shown below:

    1.     Totals: It will have the total number of measurements for all subscribers.

    2.     Group_1: It will have the counters pegged only related to the subscribers under Group 1.

    3.     Group_2: Same as no. 2 but for Group 2.

    4.     Group_3: Same as no. 2 but for Group 3.

    When test is running, in reports Tab user can see the Total counters under “Totals” tab and the split counters as per Subscriber Group as shown :

     

    L4-7 dropdown list:

     None

    If you select None from the L4-7 dropdown list and select the Per Node for Core Networks, report displays subtotals per Node (that is, per SUT in CGF Node testing).

    If you select None from the L4-7 dropdown list and select Per Session, the report will display Session subtotals.

    If you select None from the L4-7 dropdown list and do not select Per Session, the report will not contain L5-7 subtotals.

  1. L3 - DMF

    If you select DMF from the L3 dropdown list  the report displays the DMF subtotals of only the first few Sessions based on the range you specify on the Per Session and Per Mobile Node Options pane on the Report Options dialog box or for All Sessions on the GUI.

    When Sub-totals are being displayed, only the first 10 to 12 of DMF, Subscribers, Bearers, UEs, etc (what ever is being sub totalled) will show up in the GUI reports.

    Additional details / limitations in the About the Reports Tab - DMF Names

    NOTE:

    • DMF Subtotals are based solely on the Mainflows list plus additional DMFs from Volte/eMBMS, not the Instances on Assignments tab.  Starting with DMF_0 would be the first DMF in the Mainflows list. For example if you had 3 Mainflow DMFs plus running Gm Volte RTP, you could have DMF_0 thru DMF_2 from Mainflows, and DMF_3 would be the VoLTE RTP subtotal. Subflow totals do not support additional instances of a DMF.  If you add extra Instances of DMF Mainflows on the Assignment tab the data shown in the subflow tabs will not be accurate.
    • Fireball DMFs only support Per-DMF Subtotals.     

    Select Per Session for L3 subtotals. Per Session and Per Mobile Node Options pane (Select All Sessions or Range of Sessions) becomes available for input.

    L3 - None

    Select Per Session for L3 subtotals. Per Session and Per Mobile Node Options pane (Select All Sessions or Range of Sessions) becomes available for input.

    If you select None from the L3 dropdown list and select the Per Node for Core Networks, report displays subtotals per Node (that is, per SUT in CGF Node testing).

    If you select None from the L3 dropdown list and select Per Session, the report will display Session subtotals.

    If you select None from the L3 dropdown list and do not select Per Session, the report will not contain L3 subtotals.

    Mobile Access Transport

    Select Per Session , Per Bearer or Per Network Slice for Mobile Access Transport subtotals. If Per Session is selected, the Per Session and Per Mobile Node Options pane (Select All Sessions or Range of Sessions) becomes available for input. "Per Network Slice" snd "Per Session" are mutually exclusive.

    Select Per Network Slice for the capability to report the SM related measurements at per Network Slice Level for AMF Nodal test case. A new Subtotal Tab Name "NSSAI" will be added under "Reports". It is only used when the Per Network Slice measurement is enabled and applicable to the "SM", "SM Cause" and "NGAP Transfer Cause" Tabs. An example is shown below:

    When enabled, the NAS 5G Sm Measurements will display the Average Session Connect Time, Average Session Disconnect Time and the Network Slice Info - SST + Sd (NSSAI). The NSSAI information is only obtained during the Pdu Establishment Procedure. Supported for up to 8 PDUs.

     

 
  1. Per Bearer

    Selecting this check box of GTP-C Bearers will generate subtotals of each default bearer and dedicated bearers belonging to each default bearer.

    NOTE: This option will be applied to only SGW Nodal and PGW Nodal Test Case.

    Example:

    when "Default Bearers per Session" is 2 and "Dedicated Bearers per Default" is 2,

     1) If this option is not selected,

       - S11 and/or S5 Measurement in SGW Nodal, PGW Nodal, SGW Node and PGW Node will show:

       * Totals  : counts of all bearers of all UEs

       * Default  : counts of all default bearers of all UEs

       * Bearer_1 : counts of all 1st dedicated bearers of all default bearers of all UEs

       * Bearer_2 : counts of all 2nd dedicated bearers of all default bearers of all UEs

     2) If this option is selected,

       - S11 and/or S5 Measurement in SGW Nodal and PGW Nodal will show:

     * Totals    : counts of all bearers of all UEs

     * Default_0  : counts of all 1st default bearers of all UEs

     * Bearer_0_1 : counts of all 1st dedicated bearers of all 1st default bearers of all UEs

     * Bearer_0_2 : counts of all 2nd dedicated bearers of all 1st default bearers of all UEs

     * Default_1  : counts of all 2nd default bearers of all UEs

     * Bearer_1_1 : counts of all 1st dedicated bearers of all 2nd default bearers of all UEs

     * Bearer_1_2 : counts of all 2nd dedicated bearers of all 2nd default bearers of all UEs

     * Default_2  : counts of all 3rd default bearers of all UEs

     * Bearer_2_1 : counts of all 1st dedicated bearers of all 3rd default bearers of all UEs

     * Bearer_2_2 : counts of all 2nd dedicated bearers of all 3rd default bearers of all UEs

     

    • S11 and/or S5 Measurements in SGW Node and PGW Node will show the same result as #1 result because this option will be applied to SGW Nodal and PGW Nodal test cases.

    • When selecting the options in Mobile Access Transport, only Per Bearer option or only Per Session option or Both (Per Session & Per Bearer) will be allowed.

    • If only PerBearer checkbox is selected in Subtotals Panel, Enable “File Mode” will become available in the TS Generated Per MN/Session Report (s) pane. A file such as “04-05_03.43.45.PM__RID-85__ts0_tc0_S11_BearerContext.csv” will be generated. Note) This file is not a new file but the file which is generated together with “04-05_03.43.45.PM__RID-85__ts0_tc0_S11_GTP_Bearer.csv” when selecting “Mobile Access Transport – Per Session” check-box. It includes OMs of Subtotal “Default” in S11 and/or S5 Measurement.

    Example – Screen shot of the SGW Nodal Test Case - S11 Measurement Tab of when the option is selected:

     

    Example of Subtotal File – xxxx_ S11(and/or S5)_BearerContext.csv :

    Subtotals of Mobility Target GTPv2 in PGW Nodal Test will not be displayed on GUI on the fly.

    Core Network Nodes Select Per Node for subtotals per Core Network Nodes.
 

 

 

For live reporting of subtotals, only one subtotal can be reported live. The tables below show the prioritization when using multiple subtotal dimensions:

Mobile Access Transport:
Per Session Per Bearer Per Network Slice Live Subtotals
ON OFF OFF Per.UE
OFF ON OFF Per.Bearer/Per.PDU
OFF OFF ON Per.Slice
ON ON OFF Per.Bearer/Per.PDU
ON ON ON N/A Per Session and Per Network Slice is mutual exclusive from GUI
OFF ON ON Per.NetworkSlice

 

L4-7:
Group By Per Session Client per PDU Live Subtotals
None ON OFF Per.UE Per.DMF
None ON ON  
None OFF ON Per.UE.Per.PDU (subtotal for each separate PDU on each UE)
DMF ANY ANY Per.DMF
Mobile Node ANY ANY Per.UE
Subscriber Group ANY ANY Per.GROUP

 

L3:
Group By Per Session Live Subtotals
None ON Per.UE
DMF ANY Per.DMF
 

Per Session and Per Mobile Node Options pane and TS Generated Per MN/Session Report(s) pane become active only when you select Per Session. Select or complete options as required.

The middle section Per Session and Per Mobile Node Options, lets you filter subtotals for just a particular range of Sessions or Mobile Nodes. If for example you are only interested in the live reporting of the subtotals you can see the range starting a the 500th UE so UE 500-509 are reported to the GUI. Note: Filtering Sessions does not affect Per-Bearer or per Network-Slice since these both only a few subtotals each.

Per Session and Per Mobile Node Options:

  1. All Sessions

    Available if the Per Session check box is selected. Select to include ALL Sessions in the generated report.

    When Sub-totals are being displayed, only the first 10 to 12 of DMF, Subscribers, Bearers, UEs, etc (what ever is being sub totaled) will show up in the GUI reports.

     

    Range of Sessions

    Starting Session / # of Sessions

    Available if the Per Session check box is selected. Select to include a Range Sessions in the generated report.

    Enter Starting Session and # of Sessions.

     

    All Mobile Nodes

    Available if the Per Session check box is selected and Mobile Node (L4-7). Select to include ALL Mobile Nodes in the generated report.  

    Range of Mobile Nodes

    Starting MN / # of Mns

    Available if the Per Session check box is selected and Mobile Node (L4-7). Select to include a range of Mobile Nodes in the generated report.  Enter Starting MN (mobile Node) and # of Mns.

 

 

TS Generated Per MN/Per Session Report

The section TS Generated Per MN/Session Report(s) control whether the Test Server will generate end of test files for these subtotals. If enabled the Test Server will generate separate files for each subtotal dimension and layer and will include ALL subtotals that occurred during the test as specified in Per Session and Per Mobile Node Options section.  

  1. Enable (includes end-of-test report and ability to generate during test)

    Available if any Per Session check box is selected or L4-7 Mobile Node or L3 DMF. When checked, the test server will generate .csv files containing the subtotals.


    Here is a table showing which files the sub-totals are reported in when TS Generated results are enabled:

    L4-7 Group By DMF

    SrcBasic_flow.csv – Per.DMF (Sum of all UEs) Basic Data DMF KPIs

    SrcEnhanced_flow.csv – Per.DMF (Sum of all UEs) Advanced Data DMF KPIs

    Mobile Node (Each row is total of all DMFs for a given UE)
    SrcBasic_MN.csv – Per.UE Basic Data DMF KPIs
    SrcEnhanced_MN.csv – Per.UE Advanced Data DMF KPIs 
    SrcBasic_socket_MN.csv – Per.UE Basic Data DMF L4/Socket KPIs
    SrcEnhanced_socket_MN.csv – Per.UE Advanced Data DMF L4/Socket KPIs
    Subscriber Group (Each Row is Subscriber Group)
    DRA_Session.csv – Per.Group DRA Session KPIs
    Per-Session ON

    (Each row is individual UE DMF Instance)

    SrcBasic.csv – Per.UE (Sum of all DMF) Basic Data DMF KPIs

    SrcEnhanced.csv – Per.UE (Sum of all DMF) Advanced Data DMF KPIs

    SrcBasic_socket.csv – Per.UE (Sum of all DMF) Basic Data DMF L4/Socket KPIs

    SrcEnhanced_socket.csv – Per.UE (Sum of all DMF) Advanced Data DMF L4/Socket KPIs

    L3 Group By DMF IpInstance_flow.csv – Per.DMF (Sum of all UEs) Basic Data L3 KPIs
    Per-Session ON

    IpInstance.csv – Per.UE (Sum of all DMFs on given Source IP) Basic Data L3 KPIs

    IpInstance_dmf.csv - for Per UE Per DMF statistics in Client side and HostInstance_dmf.csv for Per NH Per DMF in NH side.

    Mobile Access Transport Per-Session ON Depends upon the Node or Test Case
    MME Nodal

    MN_Bearer_Context.csv

    MN_Nas_Session.csv

    Enb_S1ap_Session.csv

    AMF Nodal

    Ue_Sm_Nas_Session.csv

    Ue_Mm_Nas_Session.csv

    Ue_Mm_Nas_Session.csv

    Per-Bearer ON

    MME Nodal Only applies for GTP bearers:

    S11_BearerContext.csv

    S5_GTP_BearerContext.csv

    S11S5_Bearer_Contexts.csv

    Per-Network Slice ON AMF Nodal: Ue_Sm_Nas_Session.csv
    Core Network Nodes Per-Node ON Depends upon the Node or Test Case
    MME Nodal

    LTE_Enb_User_Node.csv - Per.UE  ??L5+ KPIs 

    LTE_Enb_Appl_socket.csv - Per.UE L4/Socket KPIs

    LTE_Enb_Appl.csv - Per.UE L5+ KPIs

    AMF Nodal Gnb_Node.csv - Per.UE  L5+ KPIs
    Gnb_Node_socket.csv - Per.UE  L4/Socket KPIs
    Gnb_Instance.csv - Per.UE  L3 IP KPIs

     

     

     

    Name of .csv files generated for client side for L4-7:

    • L4-7 - SrcEnhanced socket.csv (L4), SrcEnhanced flow.csv (per dmf) SrcEnhanced.csv (per session)
    • L4-7 - SrcBasic socket.csv (L4), SrcBasic flow (per dmf) SrcBasic.csv (per session)

    Name of .csv files generated for Network Host for L4-7:

    • L4-7 - DstEnhanced socket.csv (L4), DstEnhanced flow.csv (per dmf) DstEnhanced.csv (per session)
    • L4-7 - DstBasic socket.csv (L4), DstBasic flow (per dmf) DstBasic.csv (per session)

    Name of .csv files generated for client side for L3:

    • L3  IpInstance.csv  Per-Session (client) - Per.UE (Sum of all DMFs on given Source IP) Basic Data L3 KPIs
    • L3  IpInstance_dmf.csv - for Per UE Per DMF statistics in Client side
    • L3  IpInstance_flow.csv – Per.DMF (Sum of all UEs) Basic Data L3 KPIs

    Name of .csv files generated for Server side for L3:

    • L3  HostInstance.csv  (server)
    • L3  HostInstance_dmf.csv for Per NH Per DMF in NH side

    NOTE: Basic.csv files will be generated when using Basic DMFs. Enhanced.csv files are generated when using Advanced or Lite DMFs.

     

    File Mode

    Options: Overwrite or Append.

    When file mode is Overwrite, the csv files for the running tests will be written fresh each time the user generates subtotal (per-session) results, including multi-iteration tests and end of test generation.  Resulting in one line of final value for each Sub-totaled Instance (Session, MN, etc.).

    When file mode is Append the csv files for the running tests will be appended too each time, resulting in many lines for each Instance, once value for each generation.

    NOTEs:

    • This affects how the TS saves it's generated files either overwriting over previously generate files for the same running test session, or appending to previously generated file. This option has nothing to do with files being overwritten in the Results Website.
    • To understand how this works, you should run a multi-iteration test and manual generate per-session results while the test is running and view the results area, to see how the new files appear. For middle of test generations, files are prefixed with a timestamp, so mid-test generated file will be replaced each time. BUT for end of test or end of test iterations, files will be given timestamp, so you will end up with unique copy each iteration.

     

    Include Per Session Data

    If you select the this check box, the ,csv files will contain only the first few subtotals based on the range or ALL subtotals as specified on the Per Session and Per Mobile Node Options pane on the Report Options dialog box.

    Include Mobile Node Data

    Available if you select  Mobile Node L4-7 dropdown list and select the Per Session check box. Select to Include Mobile Node Data based on the range you specify or All Mobile Nodes on the Per Session and Per Mobile Node Options pane on the Report Options dialog box.

    Include L3 Rates

    Select to include the Rates measurements in the L3 files (IpInstance.csv and HostInstance.csv).

     

 

TIP: Any limit on gathering reports with the L4-7 Subtotals and other reporting options when running a TC with maximum capacity depends on your system capacity (OS and memory) and license.

It is not recommend to gather reports with the above options on a Test Server with Max Mode (multi-threading) capacity.

Optimized File Transfers - Optimize transferring files from the Test Server to the TAS. 

 

TS Compress Results

Select to enable the TS (Test Server) to compress TC (Test Case) end of test files into tcN_test_result.tar.gz tarball for each TC to facilitate fast transfers to the TAS (TS level files like PCAPs are excluded).

On TS the tarball will be named like this:

/home/spcoast/sNNN/tc0_test_result.tar.gz

 

On TAS, prefixed with Test and TS Index:

23-10-03_15.59.43_RID-1_ts0_tc0_test_result.tar.gz
23-10-03_15.59.43_RID-1_ts0_tc1_test_result.tar.gz

 

Inside the tc0_test_result.tar.gz  would expected to be files like this:

tc0_HostInstance.csv 
tc0_DstBasic_socket.csv
tc0_DstBasic_flow.csv
tc0_DstBasic.csv

TAS Extract Files

Select to enable the TAS to extract the tarball once retrieved and then delete the tarball once extracted. If "VisionWorks Conversion" is enabled and #1 is enabled, #2 will be required or else error:  "VisionWorks cannot work unless tarballed results are extracted".

Available for selection is TS Compress Results is enabled.

 

Do Not Delete Tarball

Select to prevent the TAS from deleting the tcN_test_result.tar.gz tarball when files are extracted. Files will be preserved in results.

Available for selection is TAS Extract Files is enabled.

 

To globally override what is set in each Test Session, TAS Setting (Edit Settings):   optimized_results_transfers can be set to ON/OFF to force the feature ON or OFF.

When the TAS is started and the TAS Setting is set properly, a tasoutput.txt message will be logged: "Optimized Results Transfers forced: ON" (or OFF respectfully).
If set ON and the Test Session has "TS Compress Results" turned off, the Test will be converted to the default optimized transfer mode: TS Compress Results Enabled, TAS Extract Files Enabled and Do Not Delete Tarball Disabled.  If set OFF and the Test Session has "TS Compress Results" turned on, the Test will be converted to disable optimized transfer mode.   In both cases, a Run Log Message will be logged if Test mode was changed due to the TAS Setting. 
 

Kafka - Select for Landslide TAS kafka to send Live Measurements.  Additional details in Customizing Your Environment - Kafka section - Live Measurements.

Select "Do Not Send Any Updates to Kafka" - When this is checked, expect no Kafka messages from the Test Session for any/all Topics not just measurements.    

Select Force Measurements Updates - Options : On, Off, Throttled. When Throttled is enabled, enter the Throttling Interval in seconds. Default : 300 secomds.

For Throttled mode, it will cache the measurements data, before the timer out, the cache data will be updated if it has the measurement data or it will add a new one based on the measurement id. This the is way to merge the coming measurements, just to keep the latest ones. When configured timer run out, it will send the merged one to Kafka.
If Throttling is enabled and a test session completes before timer expires, we will force send that final message asynchronously. 

 

Kafka Settings (Live Measurements) Test Session Settings (this section) Result
Meas topic disabled ANY (Not forced/default/disabled, ON, OFF, Throttled) No Meas Kafka Messages
Meas Topic ON Not forced Standard Kafka Messages
Meas Topic ON Forced OFF No Meas Kafka Messages
Meas Topic ON Forced ON Standard Kafka Messages
Meas Topic ON Forced Throttled Throttled Kafka Messages
Meas Topic OFF Not forced No Meas Kafka Messages
Meas Topic OFF Forced OFF No Meas Kafka Messages
Meas Topic OFF Forced ON Standard Kafka Messages
Meas Topic OFF Forced Throttled Throttled Kafka Messages
Meas Topic Throttled Not forced Throttled to Kafka Settings Interval  Kafka Messages
Meas Topic Throttled Forced OFF No Meas Kafka Messages
Meas Topic Throttled Forced ON Standard Kafka Messages
Meas Topic Throttled Forced Throttled Throttled to Test Session Interval  Kafka Messages

 

NOTES:

If you abort the test session, Per Session Reports will not be generated.

If you capture Per Session Reports close to the end of the test and the captured reports do not finish before the Auto-Saved report generation begins, the captured reports may be incomplete or overwritten.

If network conditions abnormally delay the transfer of files from the test server to the TAS, the test may finish before all Per Session Reports are generated.

If the test session uses Automation Control to execute a test case more than once, a Per Session Report is generated when the test case reaches the Cleanup step. The report generated with the final iteration of the test case is included with the Auto-Saved Report. You can retrieve the reports for each iteration by capturing a Per Session Report after the Cleanup step and before the Init step that begins the next iteration.

 

After the test session has been stopped and the final measurements have been collected, the specified reports are generated. Select the Logs tab to view the report generation progress and the list of files generated. When the reports are complete, you can view or download them.

Notes to keep in mind when viewing results files:

NOTEs:

  • Each session has its own traffic start timing.
  • We will save the starting time at the point of traffic start/resume for each session, and calculate the “accumulated time” at the point of traffic pause/stop. So the “accumulated time” here means the total traffic active time during the test run.
  • For example, after the test session init/start/running, it starts traffic at T0, then pauses at T1, then resumes at T2, and finally stop the traffic and the whole test at T3. The accumulated traffic live time is: (T1 – T0) + (T3 – T2).

 

Related Topics

  1. About Test Reports
  2. Adding Automation Control
  3. Capturing Reports
  4. Changing the Measurement Values
  5. Changing the Report View
  6. Displaying Intervals
  7. Graphing Measurements