Criterion Editor

You can view/add/edit a criterion on the View/Edit Criterion window by clicking the Add or Edit button from the Pass/Fail tab. A Criterion defines a single condition to check. You can specify when to check the condition and what to indicate when the condition is true.

The following are the main components of the View/Edit Criterion window:

Condition
Result Type
Dependencies (Start, Stop, Expire, Reset)

The information/parameters displayed on the View/Edit Criterion depends on the options you select from the various dropdown lists.

Conditions

Each Criterion is based on one the following Conditions and the parameters displayed on the bottom pane of the View/Edit Criterion window depends on the Condition you select.

Test State or Step — Compare a test session state or step to a user specified value.
Measurement — Compare a measurement to a user specified value. (Note - Subtotal measurements are not supported)
Compare Measurements — Compare Two Measurements. (Note - Subtotal measurements are not supported)
Test Case State — Indicates the Test State or automation control step to compare.
TS CPU % - Additional details can be found in TS CPU % Measurements document. TS CPU Percentage Measurements.
Run Log - determines status based on run log messages.

NOTE:The Operator for comparison is displayed for each criterion condition. The operator can be one of the following: >, >=, ==, !=, <, or <=

Test State or Step

When you select Test State or Step as the Criterion Condition, the following parameters are available for selection:

Test State or Step — The test state or automation control step to compare to. The list of test states includes: INIT, STARTING, STARTED, RUNNING, STOPPING, CLEANUP, WAIT_FOR_TRANSFERS, FINAL, COMPLETE, and COMPLETE_ERROR.

NOTE: When using automation control steps there will also be at least one choice for each step.

Operator — Select the Criterion Operator.

The operator can be one of the following: >, >=, ==, !=, <, or <=

Measurement

When you select Measurement as the Criterion Condition, the following parameters are available for selection:

View — Select the report view, i.e. either Summary or an individual test case.
Tab — Select the tab that contains the measurement to use as the criterion. The ALL tab contains the global Elapsed Time and Test Iteration measurements.
Measurement — Select the measurement to evaluate.
Operator — Select the Criterion Operator.

The operator can be one of the following: >, >=, ==, !=, <, or <=

Value — Enter the value that will be compared with the measurement.

NOTES:

If the Measurement is of String type, the TAS will perform a String comparison. Otherwise, if the Measurement is not of type String, it will attempt an Integer or Float comparison.
For a string measurement compare, both the Value (user entry) and the Measurement will be trimmed of leading spaces before comparison.

You can compare two measurements by selecting Compare Two Measurements as the Criterion Condition.

Summary Measurements are checked on 1-second intervals. This is to ensure that summary measurements include all values from all reporting test cases. Individual test case measurements are checked as the changes are reported to the TAS. When comparing two measurements, if either measurement is a summary, they are both checked on 1-second interval.

Make sure you understand how a measurement is reported before setting up Criteria. Some measurements are best checked only after a test case has finished (e.g. State = CLEANUP), some are safe to check throughout the test, and some should only be checked when another measurement reaches a certain value.

Test Case State

When you select Test Case State as the Criterion Condition, the following parameters are available for selection:

Test Case Instance — Select the instance of the Test Case.
Test Case State — Select the state to compare with the test case. The list of test states includes: UNINITIALIZED, INITIALIZING, INITIALIZED, STARTING, STARTED, RUNNING, STOPPING, STOPPED, and CLEANUP.
Operator — Select the operator for the comparison.

The operator can be one of the following: >, >=, ==, !=, <, or <=

NOTE: The Test Case State STARTING is < RUNNING and CLEANUP is > UNINITIALIZED.

See the Test Case State Machine topic to understand for how each state or step is related to each other for comparison purposes:

TS CPU %

When you select TS CPU % as the Criterion Condition, the following parameters are available for selection:

Test Server — Select the Test Server.
Process-Index — Select the Process Index. To a apply to TS-Process CPU %m select specific TS -process index (relative to test session). For TSs without multiple processes, pick p0 to be able to select Non-Control Cores and Control Core.: @TS-Level, p0, p1, p2, p3, p4, p5, p6, p7.
Measurement — Select the measurement - AVG, Control AVG , Non-Control AVG.
Operator — Select the operator for the comparison.

The operator can be one of the following: >, >=, ==, !=, <, or <=

Value — Enter the value that will be compared with the measurement.
Per-Interval — Select Per-Interval.

Run Log

When you select Run Log as the Criterion Condition, the following parameters are available for selection:

Source — Select ANY or TAS, TS or TC. The source is based on our normal About Test Logs Information.
Levels — Select the message levels - : Informational , Warning, Error.
Match — Select the Match options : StartsWith, Matches, Contains, EndsWith, !Matches - Enter up to 64 ASCII characters or leave blank to match all. Match Option StartsWith with empty text will search for occurrence of a log level (ex. Warning). For example, with empty text search, you can trigger based on just an Error from a particular Source or trigger when any WARNING or ERROR is reported. Note : An empty Match value will match any log message, regardless of the operator (StartWith, EndsWith, Matches, etc…)

Source examples:

TAS: TAS: MESSAGE
03/31 13:41:01.257:TAS:ts1(CI092) eth2 capturing started

TAS is the source, ts1(C1092) is just part of the message.

TS: <test server index (test server name)>:P<test server process index (test process index>: MESSAGE
04/02 15:54:16.387:ts0(Coast70):p3(3): ERROR: message is missing

TC: <test server index (test server name)>:P<test server process index (test process index>: TC<test case index within the test server>
04/02 15:54:16.387:ts0(Coast70):p3(3)::tc11: ERROR: Requested connect rate (3000.00) is too high

Examples:

1. To indicate a failure for any Error Log configure it like this : Never stops, never resets

2. For any error or warning from a specific Test Case (IP Application Node in this example), configure it like this :

It will expire and reset on each iteration of the test if you use a test loop, reporting a failure count for each loop.

^ Back to Top

Result Type

Each Criterion has a Result Type that determines which state should be indicated if the condition is true. The Result Types are PASS, FAIL and LOG, where the true states are PASSED, FAILED, and OCCURRED respectively.

See also Result Type Machine

For Example:

A PASS criterion with a measurement condition of Sessions Established >= 500, would indicate PASSED once the Sessions Established reach 500. If the criterion expires before reaching 500, the state would be set to FAILED.

^ Back to Top

Dependencies

Dependencies are the Start, Stop, Expire, Reset settings that point to another Criterion.

Each Criterion is associated with other Criteria or pre-defined test session events to determine when to start checking the condition, stop checking the condition, expire, or reset.

IMPORTANT: Start and Stop are independent of Expire and Reset:

Start and Stop only control when a Criteria’s Condition is checked. They indirectly affect when a Criterion will succeed.
Expire and Reset directly affect when a Criterion will expire and be reset even when its condition is not being checked.

The following pre-defined events are available as options from the Start, Stop, Expire, and Reset dropdown list:

Start of Test — Occurs immediately after the test is started
End of Test — Occurs when the test reaches COMPLETE or COMPLETE_ERROR. This is the latest point at which Criterion can be evaluated.
Start of Iteration — Occurs each time the overall test state reaches INITIALIZING, except for the first time.
End of Iteration — Occurs each time the overall test state reaches FINAL.
Never — Only applies to Stop and Reset, which is the same as End of Test, since stopping or resetting at this point would be of no use.

Start, Stop

You can control when the condition is being checked with Start and Stop. A basic criterion would include: Start = Start of Test and Stop = Never. Hence, evaluating the condition from the start of the test until the criterion expires or shows as true. However, you could also set up such that the criterion starts checking its condition based on other Criterion.

For Example:

You could set Start to another criterion that succeeds when a certain Test Case reaches the RUNNING state, and Stop to another criterion that succeeds when the same test case reaches the STOPPING state. This would ignore any values that occurred while the certain Test Case wasn’t running.

Expire, Reset

You can pick when a Criterion expires or resets. A reset would set the criterion back to the PENDING state. By default, criteria are set to expire at the End of Iteration and to reset at the Start of Iteration, so that multiple iteration tests are treated separately.