Monitoring the Test Progress

When you run a test session, a number of progress indicators are available in the Test Session window that keep you apprised of the test session state, the state of each test case, and the actions that have been performed. Click to learn more about Test Session States. See additional information on Directly Commanding Test Cases after they have been initialized.

NOTE: TAS will disconnect a client only if it has stopped communicating with it. The client sends a heartbeat message every 30 seconds which resets the idle timer. The idle timer is 72 hours and a client will never be disconnected if it is monitoring a test.

A test case passes through several states during the course of a test. The table below explains the activities performed during each state by the different types of test cases.

NOTE: Test case progress reports in 5 second increments for Delay commands and On Demand Commands (Sequencer commands).

State	Nodal Test Cases	Node Test Cases
Uninitialized	The test case has not been run or has been cleaned up after execution.
Init	The test system is reserving the resources required to execute the test case and initializing the required protocol stacks.
Initialized	Initialization is complete and the test case is ready to start attempting MN sessions.	Initialization is complete and the test case is ready to start the service provided.
Starting	MN session establishment is underway and the test case has attempted to connect at least one MN session.	The service provided is started.
Running	An attempt has been made to connect all defined MN sessions. The sessions may or may not have been successfully established. When Sequencer commands are being executed, Running Sub-States are reported. For example: RUNNING:3#3:WAITING RUNNING:1#2:PAUSING RUNNING:2#5:PAUSED RUNNING:1#1:Sync_0	The service was successfully started and is ready to receive requests. See Running Sequencer Command Index for details.
Stopping	The test case has been stopped and is attempting to disconnect established MN sessions.	The service is stopping.
Stopped	All MN sessions have been successfully disconnected or the test case timed out while waiting for disconnect responses from the SUT. When Data Traffic is included in the test and Auto Stop Control Layer is checked, MN sessions are disconnected automatically when the traffic fails or when a finite number of transactions is defined. When Sequencer command Auto-Stop is executed, the test case is stopped, stops traffic, test sessions, disconnects all sessions, completes the stop process and reports State=STOPPED	The service has stopped. See log when Auto-Stop is executed.
Cleanup	The test system releases the resources used by the test case.

States:

Test Case State

The test case state is displayed in the Session Builder tab... Each test case row includes the test case state and a progress bar that indicates the time remaining if the test case is working to reach the Running or Stopped state.

If Automation Control is used, the progress bar displays the number of the current step. If On Demand Commands/Sequencer is also used the Test Case state column displays the command Id/number and the command counter (n#n) (counter = iteration of the sequencer command step loop).

The progress bar provides additional information and counts down the time remaining in a timed test or the time remaining in the current step in a test that uses Automation Control/Command Step.

Additional details in About Criterion State Machines.

Example:Example:

This example illustrates the test session and test case progress. Also, identifies the command step being executed.

Test Session State

The test session state is displayed under the menu bar in the Test Session window... It indicates the Overall State of the test session. If a test session contains one test case, the test session state is the same as the test case state. If a test session contains more than one test case and does not use Automation Control, the test session state represents the lowest state of the test cases. A test session with 3 test cases, one of which is still Starting while the other two are Running, for example, will be in the Starting state.

If Automation Control or On Demand Commands/Sequencer is used, Overall State displays the number of the current step. A progress bar provides additional information and counts down the time remaining in a timed test or the time remaining in the current step in a test that uses Automation Control.

The Overall State also indicates Test Errors/Warnings on top of Test Session window. When running of a test, if the TAS or TS reports errors or warnings in the Run Log, a warning displays in red. The status clears when you navigate to the Logs tab.

The Overall Pass/Fail Criteria Status for the test session is the summary of :

the Overall Pass/Fail Criteria (i.e. the Pass/Fail tab - About the Pass/Fail Tab),
the UE Info Criteria (the Info tab (About the Info Tab ) , Criteria subtabs)
the IEV pass/fail status (IE Validation Tool)

NOTE: The Test’s Overall Pass/Fail Criteria State will be PENDING if all Criteria are PENDING; FAILED if any Criteria are FAILED; PASSED if at least one Criterion is PASSED and none are FAILED.

However, if you are on the Logs tab when the error or warning occurs, the Overall StateOverall State does not display any error/warnings.

Test sessions have one additional state — Complete — iindicating the test session is 100% completed, the test results are available, and the test server and other resources are free to execute another test. One nuance, if — Complete Error — is reported, it indicates there were some errors reported during test initialization or cleanup. Check the logs for the reason.

Additional details in About Criterion State Machines.

Example:Example:

In this example, the test session is running but all test cases have not reached the "running" state.

Directly Commanding Test Cases

When the test session is running, you may right-click on any of the test cases in the grid to display an Action Menu with options: Init, Start, Stop and Cleanup.

Once a test case has been initialized, you can manually stop/start it inside a test session using Directly Command Mode.

In the session builder tab, once the test case has been initialized, right click the test case and select “Show Directly command test case Panel”:

NOTES:

The TAS supports multiple selections of Test cases however they must all be in the same state or similar state for the Direct Command panel to allow commands to be executed. Each command will be fully executed separately as GUI command, but automatically, as if the user selected each TC individually and clicked the button.
Users should be careful to select Test Cases that are in the same state (or similar state). The GUI will make no provisions to prevent a command that might be out of order or invalid, and will rely on the system to return an error in these cases.

Whenever it is necessary and if the test case is already in a “Running” state, it's possible to manually stop it using the “stop” button:

To re-start the test case, cleanup the procedure and Init before starting:

Once the state is “Stopped” click “Cleanup”, then since the state is “Uninitialized” click “Init”, and once the test case is “Initialized” click “Start” and run the test case again.

Test log messages

The test log messages displayed in the Logs tab... provide more detailed information on the test progress. The test log includes test control messages generated by the TAS and responses from the test servers, including informational messages, any errors that may prevent the test session from running, and errors encountered during the test. You can save and then print the test log to aid in troubleshooting a test configuration, and the test log is automatically saved on the TAS when the test session is complete.

When sending commands to a running Test Session that ultimately ends up going through to the Test Server, it is possible that you may need to retry that command due to 3 possible reasons. The types of commands this applies to includes ODCs (execute procedures), Updates (parameter changes), TC Commands (init/start/stop/cleanup), UE Info Query (from GUI only). See diagram below.

The three command errors that indicate a retry should be attempted:
#1. TS is busy handling another command --> Each TS-Process can only support one blocking-for-response command at a time, instead of BLOCKING for X seconds, we respond to the caller immediately and let them decide when to retry.
#2. ERROR: Last command have not finished yet!!! --> The Test Case single control thread for ODCs is currently busy executing previous command, instead of queuing up this request we respond to the caller immediately and let them decide when to retry.
#3. UDP Command Timed Out --> RE: Network issues or TS temporarily unresponsive.

#4. TS recycled --> This can occur if the TS is recycling, and TAS knows it is waiting for Registration and/or the first READY Status

#5. TS IP Address not established --> This can occur if the DHCP mode TS is not resolved/registered yet

Learn more about test logs

Wireshark Network Analyzer

With The Wireshark Network Analyzer, you can monitor the test by capturing the packets sent and received by a test server.

Learn how to use Wireshark

Send ODC, Update or TC-Command to Test Server :

Learn More About: