About Test Logs


The Test Log collects the operational messages generated by the TAS and the test servers from the time a test session is started until it is complete. Test Log collects the messages related to the control of the test session and test cases, as well messages from the Test Servers specific to the testing.   The Test Log is provided in the Test Results as the log.txt file, e.g. 23-03-30_16.13.03__RID-1__Single_TC1.log.txt. The Test Log (Run Log) differs from the Real-Time Logs (System Logs), which are not focused on individual Test Sessions. The Real-Time Logs collect system-level actions and some session-level messages and is not associated with Test Sessions.

The Logs tab... is added to the Test Session window and displayed when you click Run. You can monitor the test's progress as the messages are added to the log. These messages include:

While the test session is running, two logs are available in the Logs tab that display the messages generated while you are connected to the test session: Current Logs tab and the Log History tab.

Current Logs Displays only the most recent 200 messages, and the Log History tab contains the messages not displayed in Current Logs.
Log History Available even after the test is complete, and it then displays all messages collected throughout the duration of the test. If you reconnect to a disconnected session, you can retrieve the entire test log, which is then displayed on the TAS Logs tab.

Errors/Warnings

Available when running of a test, if the TAS or TS reports errors or warnings. See Test Server CPU Utilization Reporting below for additional details when test sessions print test server High CPU Utilization Warnings in the Logs tab of the test sessiors.  

When the test is complete, the Test Log is automatically saved on the TAS and you can view or download it at any time.

Message Structure

Test Log messages are color-coded to indicate whether the message contains informational (white), warning, (yellow), or error (orange) data. The messages follow a common format:

Prefix String

Provided inside the end of test log.txt file, to indicate the log type.   

log - Informational level log. Example : log: 04/02 15:53:53.909:ts0(Coast70):p2(2)::tc6: Begin Test Case

 

warning – Warning level log. Example : warning: 04/02 16:24:08.505:TS(Coast70[1]): The Test Server status indicates the test stopped running and is now ready


error – Error level log . Example : error: 04/02 15:54:16.387:ts0(Coast70):p3(3)::tc11: ERROR: Requested connect rate (3000.00) is too high


test_progress – Test Session Control/Automation progress, seconds remaining in a Delay Step or before something times out. Example : 100:Waiting on Predecessor (in step STARTING)


test_state – Overall Test State. Example : test_state:STARTING:04/02 15:53:57.146:STARTING


tc_progress – Test Case progress, estimated seconds remaining until current action completes or next state expected. Example : tc_progress: TS[0] TC[0] -> 50:1 Sec(s) remaining (Test Initialization)


tc_state – Test Case State. Example : tc_state: TS[0] TC[0] -> INITIALIZING


port_capture – Activity related to Port Capture starting/stopping/transferring. Example : port_capture: 03/31 13:41:01.257:TAS:ts1(CI092) eth2 capturing started


action – Activity initiated by the user, Continue, Stop, Abort. Example : action: Continue:Resume the test

Time stamp

month, day, hour, minute, second, millisecond, converted to the test system's local time zone.

Format MM/dd HH:mm:ss.SSS, e.g. 03/31 13:41:01.257

The message source

TAS e.g. 03/31 13:41:01.257:TAS:ts1(CI092) eth2 capturing started
TAS is the source, ts1(C1092) is just part of the message.

or
TS<test server index (test server name)>:P<test server process index (test process index>: TC<test case index within the test server>

e.g. 04/02 15:54:16.387:ts0(Coast70):p3(3)::tc11: ERROR: Requested connect rate (3000.00) is too high

Message text Begin Test Case, State=INITIALIZING, State=INITIALIZED, STARTED, etc.

NOTE: The message queue from the TAS to the Client for a given test session is reported as Session-to-client message queue. These messages are informational and sometimes may be dropped to allow the higher priority messages (progress, state, measurements) to get through/to be reported.

The logs are only dropped from being sent to the Client, but they are stored on the TAS. You may retrieve the complete TAS Run Log, via Logs->Get TAS Log, or wait till the test is over and Retrieve-All-Results.   

The Session-to-client message queue is full, log message (s) from TAS to Client will be dropped indicates one or more of the following:

  • The Client is too slow to read the messages from the TAS

  • The Client is locked up

  • The network is overloaded

  • The test session, test cases, test servers, etc., are sending too many logs to the TAS and overloading the queue (at that particular time)

When the blockage clears up, you should get a message, Queue is nominal, N log message from TAS to Client lost  to indicate how many logs were dropped.

TS_INDEX_IN_TEST

Index of the Test Server within the Test Session configuration

TS_NAME

Name of the test server

TS_PROCESS_INDEX

Index of the Test Server Process within the Test Server

TEST_PROCESS_INDEX

Index of the Test Server Process within the Test Session

TS_INDEX_IN_TEST

Index of the Test Server within the Test Session configuration

TS_PROCESS_INDEX

May be 0, 1, 2, the actual process on the TS that the test case is running on.

TEST_PROCESS_INDEX

May be 0, 1, 2, the relative index of the process within the test session’s process indexes.  For example if three test session are running on a test server, each on their own process, the logs would look like this:

  • First test to run (it’s first process 0, is running on TS process 0):

log: 02/06 11:03:51.892:ts1(Coast88):p0(0):tc0: State=INITIALIZING

  • Second test to run (it’s first process 0, is running on TS process 1):

log: 02/06 11:03:51.892:ts1(Coast88):p1(0):tc0: State=INITIALIZING

  • Third test to run (it’s first process 0, is running on TS process 2):

log: 02/06 11:03:51.892:ts1(Coast88):p2(0):tc0: State=INITIALIZING

Example: For Command Sequencer

Test Case Sub-state indicates Current Command indexTest Case Sub-state indicates Current Command index (For Command Sequencer)

log: 07/02 15:20:41.711:ts1(Coast45 10.8.45.1):p0(0):tc0: State=STARTED

tc_state: TS[0] TC[0] -> RUNNING

log: 07/02 15:20:41.766:ts0(Coast44 10.8.44.1):p0(0):tc0: State=RUNNING

tc_state: TS[1] TC[0] -> RUNNING

log: 07/02 15:20:41.754:TAS: Switching to step RUNNING

log: 07/02 15:20:41.711:ts1(Coast45 10.8.45.1):p0(0):tc0: State=RUNNING

test_state:RUNNING:07/02 15:20:41.754:RUNNING

test_progress: 100:Running indefinitely, stop test manually

log: 07/02 15:20:42.766:ts0(Coast44 10.8.44.1):p0(0):tc0: MME/PGW Nodes connected: true

log: 07/02 15:20:42.766:ts0(Coast44 10.8.44.1):p0(0):tc0: Starting GTP Protocol...

tc_state: TS[0] TC[0] -> RUNNING

log: 07/02 15:20:42.766:ts0(Coast44 10.8.44.1):p0(0):tc0: State=RUNNING

log: 07/02 15:20:42.766:ts0(Coast44 10.8.44.1):p0(0):tc0: Executing Command Sequencer...

tc_state: TS[0] TC[0] -> RUNNING:0#0

log: 07/02 15:20:42.766:ts0(Coast44 10.8.44.1):p0(0):tc0: State=RUNNING:0#0

log: 07/02 15:20:42.766:ts0(Coast44 10.8.44.1):p0(0):tc0: Executing Command 0

log: 07/02 15:20:42.766:ts0(Coast44 10.8.44.1):p0(0):tc0: onDemandCommand { ControlBearer { "Attach" "100.0" "1" "10" } }

log: 07/02 15:20:42.767:ts0(Coast44 10.8.44.1):p0(0):tc0: Starting Attach ...

log: 07/02 15:20:42.867:ts0(Coast44 10.8.44.1):p0(0):tc0: Attach done ...

log: 07/02 15:20:42.867:ts0(Coast44 10.8.44.1):p0(0):tc0: Getting the next command...

tc_state: TS[0] TC[0] -> RUNNING:1#1

log: 07/02 15:20:42.867:ts0(Coast44 10.8.44.1):p0(0):tc0: State=RUNNING:1#1

log: 07/02 15:20:42.867:ts0(Coast44 10.8.44.1):p0(0):tc0: Executing Command 1

log: 07/02 15:20:42.867:ts0(Coast44 10.8.44.1):p0(0):tc0: Delaying 5 seconds

tc_state: TS[0] TC[0] -> RUNNING:2#2

log: 07/02 15:20:47.867:ts0(Coast44 10.8.44.1):p0(0):tc0: State=RUNNING:2#2

log: 07/02 15:20:47.867:ts0(Coast44 10.8.44.1):p0(0):tc0: Executing Command 2

log: 07/02 15:20:47.867:ts0(Coast44 10.8.44.1):p0(0):tc0: onDemandCommand { ControlBearer { "PdnConnect" "100.0" "1" "10" "2" "2" } }

log: 07/02 15:20:47.867:ts0(Coast44 10.8.44.1):p0(0):tc0: Starting PdnConnect ...

log: 07/02 15:20:47.967:ts0(Coast44 10.8.44.1):p0(0):tc0: PdnConnect done ...

log: 07/02 15:20:47.967:ts0(Coast44 10.8.44.1):p0(0):tc0: Getting the next command...

tc_state: TS[0] TC[0] -> RUNNING:3#3

log: 07/02 15:20:47.967:ts0(Coast44 10.8.44.1):p0(0):tc0: State=RUNNING:3#3

log: 07/02 15:20:47.967:ts0(Coast44 10.8.44.1):p0(0):tc0: Executing Command 3

log: 07/02 15:20:47.967:ts0(Coast44 10.8.44.1):p0(0):tc0: Delaying 5 seconds

log: 07/02 15:20:47.967:ts0(Coast44 10.8.44.1):p0(0):tc0: All Dedicated Bearers are Established

tc_state: TS[0] TC[0] -> RUNNING:4#4

log: 07/02 15:20:52.967:ts0(Coast44 10.8.44.1):p0(0):tc0: State=RUNNING:4#4

log: 07/02 15:20:52.967:ts0(Coast44 10.8.44.1):p0(0):tc0: Executing Command 4

log: 07/02 15:20:52.967:ts0(Coast44 10.8.44.1):p0(0):tc0: Start DMF 0

log: 07/02 15:20:52.968:ts0(Coast44 10.8.44.1):p0(0):tc0: Resuming traffic for DMF 0

log: 07/02 15:20:52.968:ts0(Coast44 10.8.44.1):p0(0):tc0: Starting traffic for Paused DMF 0

log: 07/02 15:20:52.968:ts0(Coast44 10.8.44.1):p0(0):tc0: processResumeData - startingInstance:20, endingInstance:29. DataResumeRate:3000.0.

log: 07/02 15:20:52.972:ts0(Coast44 10.8.44.1):p0(0):tc0: Traffic Reconfiguration Complete

log: 07/02 15:20:52.972:ts0(Coast44 10.8.44.1):p0(0):tc0: Getting the next command...

tc_state: TS[0] TC[0] -> RUNNING:4#5

log: 07/02 15:20:52.972:ts0(Coast44 10.8.44.1):p0(0):tc0: State=RUNNING:4#5

log: 07/02 15:20:52.972:ts0(Coast44 10.8.44.1):p0(0):tc0: Executing Command 4

log: 07/02 15:20:52.972:ts0(Coast44 10.8.44.1):p0(0):tc0: Start DMF 1

log: 07/02 15:20:52.973:ts0(Coast44 10.8.44.1):p0(0):tc0: Resuming traffic for DMF 1

log: 07/02 15:20:52.973:ts0(Coast44 10.8.44.1):p0(0):tc0: Starting traffic for Paused DMF 1

log: 07/02 15:20:52.973:ts0(Coast44 10.8.44.1):p0(0):tc0: processResumeData - startingInstance:30, endingInstance:39. DataResumeRate:3000.0.

log: 07/02 15:20:52.977:ts0(Coast44 10.8.44.1):p0(0):tc0: Traffic Reconfiguration Complete

log: 07/02 15:20:52.977:ts0(Coast44 10.8.44.1):p0(0):tc0: Getting the next command...

tc_state: TS[0] TC[0] -> RUNNING:4#6

log: 07/02 15:20:52.977:ts0(Coast44 10.8.44.1):p0(0):tc0: State=RUNNING:4#6

tc_state: TS[0] TC[0] -> RUNNING:4#4

log: 07/02 15:20:52.967:ts0(Coast44 10.8.44.1):p0(0):tc0: State=RUNNING:4#4

ts<TS_INDEX_IN_TEST>(<TS_NAME>)P<TS_PROCESS_INDEX>(<TEST_PROCESS_INDEX>):tc<TC_INDEX_WITHIN_TS>: State=<RUNNING:INDEX#COUNT>

Test Case Sub-state Current Command index with SyncPointTest Case Sub-state Current Command index with SyncPoint (For Command Sequencer step SyncPoint)

tc_state: TS[0] TC[1] -> RUNNING:0#0 log: 02/22 00:59:39.962:ts0(Coast04):p0(0):tc1: State=RUNNING:0#0 tc_progress: TS[0] TC[1] -> 16:5 Sec(s) remaining (Delay) log: 02/22 00:59:44.951:ts0(Coast04):p0(0):tc0: Executing Command 1 log: 02/22 00:59:44.951:ts0(Coast04):p0(0):tc0: Reached SyncPoint 0 tc_state: TS[0] TC[0] -> RUNNING:1#1:Sync_0 log: 02/22 00:59:44.951:ts0(Coast04):p0(0):tc0: State=RUNNING:1#1:Sync_0 log: 02/22 00:59:44.961:TAS: ts0::tc0 Reached SyncPoint: 0 log: 02/22 00:59:44.962:TAS: Shared test cases not in sync yet log: 02/22 00:59:44.962:ts0(Coast04):p0(0):tc1: Executing Command 1 log: 02/22 00:59:44.962:ts0(Coast04):p0(0):tc1: Reached SyncPoint 0 tc_state: TS[0] TC[1] -> RUNNING:1#1:Sync_0 log: 02/22 00:59:44.962:ts0(Coast04):p0(0):tc1: State=RUNNING:1#1:Sync_0 log: 02/22 00:59:44.965:TAS: ts0::tc1 Reached SyncPoint: 0 log: 02/22 00:59:44.965:TAS: All TCs are in sync log: 02/22 00:59:44.968:ts0(Coast04):p0(0):tc0: Resuming Sequencer from SyncPoint 0 log: 02/22 00:59:44.968:ts0(Coast04):p0(0):tc0: Executing Command 2 log: 02/22 00:59:44.968:ts0(Coast04):p0(0):tc0: Delaying 20 seconds tc_state: TS[0] TC[0] -> RUNNING:2#2 log: 02/22 00:59:44.968:ts0(Coast04):p0(0):tc0: State=RUNNING:2#2 tc_progress: TS[0] TC[0] -> 4:20 Sec(s) remaining (Delay) log: 02/22 00:59:44.968:ts0(Coast04):p0(0):tc1: Resuming Sequencer from SyncPoint 0 log: 02/22 00:59:44.968:ts0(Coast04):p0(0):tc1: Executing Command 2 log: 02/22 00:59:44.968:ts0(Coast04):p0(0):tc1: Delaying 20 seconds tc_state: TS[0] TC[1] -> RUNNING:2#2

log: 02/22 00:59:44.951:ts0(Coast04):p0(0):tc0: Reached SyncPoint 0 tc_state: TS[0] TC[0] -> RUNNING:1#1:Sync_0

log: 02/22 00:59:44.951:ts0(Coast04):p0(0):tc0: State=RUNNING:1#1:Sync_0

Command Sequencer Auto-StopCommand Sequencer Auto-Stop

Auto-Stop Sequencer command:

01/10 02:26:04.121:ts0(Coast04):p0(0):tc0: Getting the next command...

01/10 02:26:04.121:ts0(Coast04):p0(0):tc0: The end of Command Sequencer

01/10 02:26:04.121:ts0(Coast04):p0(0):tc0: State=RUNNING

01/10 02:26:04.121:ts0(Coast04):p0(0):tc0: Auto-stopping test case

01/10 02:26:04.121:ts0(Coast04):p0(0):tc0: Stop Activity Timer ...

01/10 02:26:04.121:ts0(Coast04):p0(0):tc0: Stopping Traffic...

01/10 02:26:04.121:ts0(Coast04):p0(0):tc0: Stop Session Control Timers

01/10 02:26:04.121:ts0(Coast04):p0(0):tc0: All Sessions Disconnected...

01/10 02:26:05.121:ts0(Coast04):p0(0):tc0: Disconnect complete.

01/10 02:26:05.121:ts0(Coast04):p0(0):tc0: Stop Complete...

01/10 02:26:05.121:ts0(Coast04):p0(0):tc0: State=STOPPED

 

01/10 02:26:04.121:ts0(Coast04):p0(0):tc0: State=RUNNING

01/10 02:26:04.121:ts0(Coast04):p0(0):tc0: Auto-stopping test case

RUNNING:INDEX#COUNT

0#0, 1#1,...n#n. Indicates the exact step in the Sequencer (being executed by the TS).

  • INDEX shows you which command in the list is being executed
  • COUNT indicates the iteration of the sequencer command step loop (where applicable).

For example, If you have loops in your sequence, INDEX and COUNT will eventually differ, this is because command #5 may be being executed for the 10th time.

NOTE: Test Session Run Log displays 0-based index.  For example, RUNNING:5#5 indicates Running 6th step/row in the GUI (that is 5 in the column labeled # is Running).

 

 

NOTES:

  • The Test Log is not saved to the TAS if a test session is stopped with a forced abort.

  • The Test Log should be saved to the TAS even if the test session fails before reaching the Started state. A "Results to be named" entry should exist in the Test Log in this case, and will display the file name. If the Test Log could not be saved, the error will be present in the Real-Time Logs.

  • The Test Log message buffer can hold up to 100 messages. If the buffer reaches 100 messages, the informational messages are dropped and an error is logged indicating how many messages were dropped. Warning and error messages are never dropped when the buffer is flushed.

Test Server CPU Utilization Reporting in the Logs:
 
  • The TAS parses and analyzes CPU % reported by each TS-Process. If you reserve Processes, you are guaranteed that only your test is running on a TS-Process. If not, you may be sharing a TS-Process with multiple tests / users. When CPU thresholds are reached, the TAS will send Run Log messages to each related test session.
  • The TAS will check 2 thresholds of 80% and 95%.
  • The TAS will use the 3-interval average for each CPU, that is, the TS reports CPU% about every 10s, the TAS will calculate an AVG value of the last 3 reported values and use that in determining the threshold.
  • If any CPU reports a 3 interval average higher than 80%, it will cause the TAS to report a message to any test sessions running on the given TS-Process. Further messages will only be reported to test sessions if the # of CPUs reporting above 80% or 95% changes.    

  • A TS-Process may have 1 to many CPUs in use, one for Control and various for User or other processing.

  • If any CPU is above 95%, the TAS will report the # of CPUs over 95%

  • If any CPU is above 80%, but none above 95%, the TAS will report # of CPUs over 80%.

  • With each report it will include the AVG% across all CPUs and the MAX% for any CPU. So for example if one CPU is 96% and 3 are 85%, the TAS will report  1 CPU is 95%  avg%=87 max%=96. Once all CPUs are under 80%, the TAS will report: CPUs nominal  avg%=X max%=Y

  • Once all CPUs are below 80%, the TAS will report a message indicating the CPUs are nominal, it will not report again until a threshold is broken.

 

Message Examples:

warning: 01/07 12:27:51.739:TS(coast43[0]): P[0]: 2 CPUs 3-interval-avg over 95%, all-used-cpus-3-interval-avg%=82 last-reported-%: Control=2 User=100 User=100 User=92 User=92
warning: 01/07 12:27:52.436:TS(coast42[0]): P[0]: 2 CPUs 3-interval-avg over 95%, all-used-cpus-3-interval-avg%=83 last-reported-%: Control=1 User=100 User=100 User=90 User=92

After clearing:
warning: 01/07 12:33:31.882:TS(coast43[0]): P[0]: CPUs nominal 3-interval-avg%=52

3-interval-avg or all-used-cpus-3-interval-avg is calculated by taking the last 3 reported values for a CPU and dividing by 3. Then adding up the avg's for each in-use-CPU and dividing by the number of in-use-CPUs. In-use-CPU is a CPU that reports a % > 0, last-reported-% is just the most recent CPU % reported by the TS.  This is also an average CPU for past X-seconds as calculated by the TS.

Learn about: