When a test case reaches the Started state, the test server begins sending the test measurements to the TAS, and the Reports tab... is added to the Test Session window. The TAS accumulates and calculates the measurements defined by each test case in the test session and reports the measurement values in time slices (intervals) throughout the course of the test.
Two report views are available in the Test Session window: a Summary report and a Test Case report.
Summary report displays the aggregate measurements for all test cases in the test session.
Test Case report displays only the measurements for the selected test case. You can change the report view at any time.
The test measurements are displayed in a grid made up of rows of measurements and columns of interval values and are logically grouped together on measurement tabs. In general, the measurements collected report the volume and rates achieved during test activities, the messages exchanged between the MNs and the physical or simulated devices included in the test, errors encountered during the test, and the throughput of the simulators. You can further customize the report display by hiding measurements, including entire report tabs, that are of little interest in a particular scenario.
The Test Summary measurement tab is included in all reports. The Test Summary tab displays high level rate and volume measurements for a test case or test session. Definitions for all measurements are grouped in topics by measurement tab in the Measurement Reference.
NOTE: When more than one test case uses the same protocols or types of emulators or when one test case uses multiple emulators, the measurements on the applicable measurement tab report the aggregate values for all instances of the protocol or emulator. |
Measurements are initially displayed in 15-second intervals. The right-most column in the measurement grid contains the measurements for the Current interval. As each interval expires, the current interval values are transferred to a column that is inserted to the left of the Current interval and labeled with a consecutive interval number.
If the test reaches 1000 intervals (1 — 999 plus Current), the interval period is increased to 30 seconds and half of the collected intervals are discarded. New intervals are created every 30 seconds until the number of intervals reaches 1000 again, and then the interval period is increased to 60 seconds, and half of the collected intervals are discarded. This process continues throughout the test whenever the number of intervals reaches 1000: the interval is doubled and half of the collected intervals are discarded.
If you configure the test session to run for a number of iterations, a special Final Report interval is created at the completion of each iteration to record the final measurements of each test iteration. If any Final Report intervals have been saved when the intervals are halved, additional timed intervals will be discarded to make room for the Final Report intervals. These intervals are not discarded unless 500 Final Report intervals have already been saved.
NOTE: Tests that run for more than 500 iterations do not contain all of the Final Report intervals. |
The measurement values displayed in the report intervals can be viewed in two different ways: the measurements collected since the test was started (cumulative values), and the measurements collected in each interval (interval values). The interval values are calculated by subtracting the previous interval's cumulative measurements from the Current interval's measurements. Therefore, when the previous interval value is larger than the Current interval value, a negative value is calculated. For example, Sessions Established will show a negative interval value as the sessions are disconnected.
You can choose to display cumulative values in all measurements, interval values in all measurements, or change the displayed values for individual measurements.
NOTE: Measurements that report averages should not be set to interval values. |
You can export the measurements from a test in two types of reports:
A Test Report contains the test session measurements as displayed on the Reports tab and grouped by report tab. The report can be generated in either of the views available for a displayed report — Summary or individual test case. In addition to the test measurements, a Test Report also includes the parameter settings in the test definition arranged by parameter tab.
A Per Session Report contains cumulative measurements for each emulated mobile node. Per Session Reports are made up of multiple files and each file contains the measurements from one report tab and one test case.
A final Test Report is automatically saved on the TAS when a test session reaches the Complete state. By default, a cumulative, Summary level report is generated in an Excel format. You can customize the type of Auto-Save report for each test session, setting the granularity, file format, and measurement value type, and optionally include a Per Session Report. You can also choose to periodically save a Test Report while the test is running.
The amount of disk space that is available for Test Results is based on your current license: To customize the amount of disk space allocated for results, see topic Edit Settings . Additional details in the About the System Status Window.
Use the TAS Settings Editor to update:
.
NOTES:
|
You can also capture a Test Report or Per Session Report while the test is running and export it to your machine. A variety of Test Report options allow you to capture only the measurements from the selected tab, save a subset of report intervals rather than all intervals, or save interval values rather than cumulative values. With Per Session Reports, you can capture all sessions or a range of sessions.
A TAC Report, is designed as an aid to troubleshooting a test session with your support representative. A TAC Report is a zipped file containing the cumulative Test Report for each test case, the exported test session, the Test Log and other Client and / or TAS Logs. This single file can then be sent to the support engineer. When you generate a TAC Report, you will be prompted for a location and file name. There is an option to also Include TS Logs. Go to Reports -> Generate TAC Report :
Pop-up allows for selection of :
A TAC Report can be generated at any time, before, after or during a test session execution. You do not even need a valid test session, valuable information can still be collected. There is an option to Include TS Logs. See Pop-up dialog above.
To generate a TAC report unrelated to a test session, create a new test session, go to Reports -> Generate TAC Report. This will provide a TAC report for everything but a test session.
To generate a TAC report for a specific Test Session but only needed after the test completes:
Reports -> Generate TAC Report after test completes.
If unable to do it manually, you can pre-enable Generate TAC from Reporting Options , then the TAS will generate a TAC report that will end up in the Results Website.
To generate a TAC report while a Test Session is running, Reports -> Generate TAC Report while test is running.
To generate a TAC report for a test session generically without running, just open the test session and Reports -> Generate TAC Report.
All of the above can be executed via Tcl API, read the Landslide Tcl API Object and Perform Function Reference for specific instructions.
#2-4 can be executed via RESTful API, Using the RESTFul API contains reference information about the Landslide RESTFul API system. Refer to Swagger UI on Landslide Home Page, it contains the latest most accurate information for anything it covers.
When there is not a Test Session at fault, or when the Test Session is not stable enough to generate the TAC Report, use this option to generate the report. Go to Help -> Generate Generic TAC Report from the Menu:
This is the equivalent of opening a new Test Session window and generating a TAC report for it but removes the idea that you need to be associated with a Test Session for a TAC report to provide help information.
TAC reports can sometimes contain a second Test Session in the STE. When the Test Session being reported is using the standard TS port-subnets, a copy of the Test Session is added to the STE that enables reserving + overriding ports to include the original TS Ports/Subnets in the Overridden Port-Subnets. This makes it easier/quicker for troubleshooting because you do not need to go look at XLS or TasStatus.csv to track down the TS being used and then the ETH ports involved to learn the Subnets. Just use the Test Session with the name starting with PORTS_
When it comes to logs you can retrieve for support, there are the 3 main options:
We also have the ability to get TAS-side TAC report, by enabling it on the Report Options of Test or using RESTful API to trigger TAC report, but this version of the TAC report will not contain any Client-side logs. Use it as last resort when you have Test Session based issues and unable to get TAC report from the Client (or Tcl API, if Tcl API based testing).
In all cases, the user should attempt to Generate the TAC Report immediately. Reproduce the problem, then immediately Generate the TAC report from any of the above methods.