DTA Administration


The table below describes the fields in the DTA Administration window that lists all the Device Test Agent (DTA) that are registered / connected with the TAS. The DTA Application running on Android and iOS UEs talking directly to the Landslide TAS. The DTA Application will utilize the UE Node Handset Test Case uses. 

DTA Test Cases currently supported are:  (UE Node App)

 

To enable the DTA Application you need a new license, that includes the necessary license key which will provide either unlimited tests or some limited # of tests 10K, 1M. In addition, the number of DTA devices registered with the TAS is also licensed.

It is also expected to be used with VisionWorks, as DTA results are converted to TEST.PROCS_UE files.

In order to deploy this application, a Google Firebase Account must be setup and related information from the Firebase Account must be placed into a firebase.conf file on the TAS. Once properly licensed, Firebase is setup and the firebase.conf file is added, when you start the TAS you will see this confirmation of Firebase communications in the TAS output: Initializing DTA services and connecting to Firebase

Status: Indicates if FireBase is connected or not.

And when you open the GUI you will see the DTA Admin toolbar and menu options:

DTA (Device Test Agent)  via Admin -> Device Test Agents Administration

 

Device Test Agent (DTA)

Field Description

Recycle DTA

Select Recycle to send a command for the DTA to re-register with the TAS, without requiring user intervention on the phone.

Logout DTA

Select Logout to send a command for the DTA to logout and require the user to login again.  I.E. use this command if have other phone you want to register and you have reached the maximum DTAs.

Cancel All Pending Tests

 

Select to Cancel all pending tests on the selected DTAs. When tests are cancelled via the Cancel button, the state of the test will change to Canceling.

Retrieve DTA Logs

Select Retrieve Logs to retrieve the DTA Application Debug logs and place them in Results Website.

Reboot MDM

Supported DTAs

 

Select to reboot the MDM supported DTAs. The MDM flag must be set to YES.

Only available for DTAs with MDM (Mobile Device Management) capability.

Update DTA App for MDM

Supported DTAs

 

Select to update the DTA app for MDM supported DTAs. The MDM flag must be set to YES.

Only available for DTAs with MDM (Mobile Device Management) capability.

Wipe Data for MDM

Supported DTAs

 

Select to wipe data for MDM supported DTAs. The MDM flag must be set to YES.

Only available for DTAs with MDM (Mobile Device Management) capability.

Lock DTA App for MDM

Supported DTAs

 

Select to lock the DTA app for MDM supported DTAs. The MDM flag must be set to YES.

Only available for DTAs with MDM (Mobile Device Management) capability.

Unlock DTA App for MDM

Supported DTAs

 

Select to unlock the DTA app for MDM supported DTAs. The MDM flag must be set to YES.

Only available for DTAs with MDM (Mobile Device Management) capability.

Enable WIFI for MDM

Supported DTAs

 

Select to enable WIFI for MDM supported DTAs. The MDM flag must be set to YES.

Only available for DTAs with MDM (Mobile Device Management) capability.

Disable WIFI for MDM

Supported DTAs

 

Select to disable WIFI for MDM supported DTAs. The MDM flag must be set to YES.

Only available for DTAs with MDM (Mobile Device Management) capability.

Add DTA Test Session

Click the Add button to add a New DTA Test Session. A new window opens to create/execute a DTA Test Session. You can configure one or more DTA Test Cases:

 

Select Run Test Cases in Parallel When Possible to send a flag to DTA to allow tests to run at same time if they support it, otherwise each Test Case will execute one at a time, next one started when current one completes.

Enter the Log Level to send a value to DTA to indicate what the logging level should be for the test execution. 0 (Indicated no Logs OFF) , up to 10. Turning on Logs will cause DTA to produce a “Run Log” for each DTA Test Case.

Select Report Logs to send a flag to DTA to indicate to send the “Run Log” along with the results to the TAS. If this is not checked, the “Run Log” will be kept on the phone, retrievable using the “Retrieve Logs” on DTA Admin window.

Enter Expiration (min)  (Range: 1 to 30 minutes) - how long DTA will keep tests pending in the background. Used by a sleeping / background DTA to know if a test should be executed when it is activated. Any tests that are in the pending queue that have expired will not be executed.

Tags - Click to enable VisionWorks Tags for a test session. See additional details in topics : About the Test Case Settings Window and Customizing Report Options.

Intent of Test  - Click to select the intent of test for a test session. See additional details in topics : About the Test Case Settings Window and Customizing Report Options

 

The Test Cases list (left panel / list) is where you can add / removed DTA Test Cases from the DTA Test Session. The test cases will be started in the order in which they are listed, in a round robin fashion. You can save a DTA test case individually and open it in another DTA Test Session at a later time. Clinking on one of the test case will open the editor on the right hand side of the screen. Each Test Case editor should match what you see in the UE Node Handset test case.

See UE Node App topic for additional details on the supported Test Types:

 

  1. basic_data - UE Node APP - Data Flow Test
  2. dns - UE Node APP - DNS Test
  3. http - UE Node APP - HTTP Test
  4. iperf - UE Node APP - iPerf Test  (Additional data fields in DTA App : "Omit (s)" timer to accommodate a TCP slow start, Select Port Range End to enable different port for each test execution by applying an incremented port, select Maximum Throughput to full bandwidth of a data pipe between the client and server.
  5. latency/traceroute - UE Node APP - Latency/Traceroute Test (Ping Host, Ping DNS - Additional data fields in Max Hops when Ping DNS is enabled - value 1 to 30)
  6. mobile_info - UE Node APP - Mobile Info Test
  7. ping - UE Node APP - Ping Test (Hostnames, Ping Interval, Ping Count, etc)
  8. speedtest - UE Node APP - Speedtest Test
  9. video - UE Node APP - Video Test
  10. voice - UE Node APP - Voice Test

 

Scheduling Test Sessions is now supported via the Test Scheduler. DTA Tests can be recurring and can be paused, but they do not support the Stop or Abort options.

The Target DTAs (bottom panel / table) allows you to add or remove DTAs to include in the DTA Test Session.

 

See sample of Real-Time Logs below.

NOTEs:   
  • A DTA can run different tests, that from the TAS’s perspective we call “DTA Test Cases”.  From DTA Perspective these are just “tests”, and the DTA will run each of them independently, each given a unique Test ID (IMEI+RUN_ID) or just RUN ID (integer 1-9999). Within our UE Node test case these tests are just test options inside the test case, so that a single UE Node test case might execute multiple DTA Tests, and you can have multiple UE Node Test Cases in a Test Session. For DTAs, each test within the DTA Application is treated as their own separate DTA Test Case within Landslide.  The DTA Test Case acts more like a Test Session, it is 100% independent of other DTA Test Cases, has it’s own unique ID and results.  The relationships between things is different than Landslide Test Server testing.
  • A DTA Test Session is not like a regular Landslide Test Session. It serves only to define, organize, sequence and start DTA Test Cases on one or more DTAs.  Once the individual DTA Test Cases are all queued or executing on the DTA, the DTA Test Session is “complete”. The DTA Test Session is not used for viewing results, watching live measurements or interacting with running tests. What you can do is, “Stop” the DTA Test Session from starting DTA Test Cases that have not started yet. For example, if you setup a DTA Test Session to run 5 DTA Test Cases on 100 DTAs, the TAS will round-robin all phones starting the test case, like this:   DTA1-TC1, DTA2-TC1… DTAn-TC1, DTA1-TC2, DTA2-TC2, ….. DTAn-TCn. If you see something going wrong, you can stop the DTA Test Session before it reaches DTAn-TCn and it will stop sending start commands. The startup sequence will attempt to start all of one DTA Test Case on all the phones at the same time, then repeat this again for each DTA Test Case.

 

A DTA test session log file will be generated and can be found in Test Results - See Viewing Test Results.

The log files will follow this naming convention : Year-Month-Day_Time_DTA-TEST-SESSION_NameofTest.log.txt

Example name of log file : 20-06-02_08.24.54_DTA-TEST-SESSION_DTA_Multi_Test.log.txt and the contents which will include relevant information regarding your test.

Sample file :

2020-06-02T08:21:05.354-05: Starting test session: DTA_Multi_Test with ID=325
2020-06-02T08:21:05.354-05: -Starting test case: ping
2020-06-02T08:21:05.354-05: --Starting DTA ID=1
2020-06-02T08:21:05.355-05:   DTA(0001-123456789022222): Test start sent for ping
2020-06-02T08:21:05.355-05:  Confirming tests started for ping
2020-06-02T08:21:06.355-05: -Test case ping startup completed
2020-06-02T08:21:06.355-05: -Starting test case: mobile_info
2020-06-02T08:21:06.355-05: --Starting DTA ID=1
2020-06-02T08:21:06.356-05:   DTA(0001-123456789022222): Test start sent for mobile_info
2020-06-02T08:21:06.356-05:  Confirming tests started for mobile_info
2020-06-02T08:21:07.356-05: -Test case mobile_info startup completed
2020-06-02T08:21:07.356-05: -Starting test case: dns
2020-06-02T08:21:07.356-05: --Starting DTA ID=1
2020-06-02T08:21:07.357-05:   DTA(0001-123456789022222): Test start sent for dns
2020-06-02T08:21:07.357-05:  Confirming tests started for dns
2020-06-02T08:21:08.357-05: -Test case dns startup completed
2020-06-02T08:21:08.357-05: -Starting test case: http
2020-06-02T08:21:08.357-05: --Starting DTA ID=1
2020-06-02T08:21:08.358-05:   DTA(0001-123456789022222): Test start sent for http
2020-06-02T08:21:08.358-05:  Confirming tests started for http
2020-06-02T08:21:09.358-05: -Test case http startup completed
2020-06-02T08:21:09.359-05: -Starting test case: latency
2020-06-02T08:21:09.359-05: --Starting DTA ID=1
2020-06-02T08:21:09.359-05:   DTA(0001-123456789022222): Test start sent for latency
2020-06-02T08:21:09.359-05:  Confirming tests started for latency
2020-06-02T08:21:10.359-05: -Test case latency startup completed
2020-06-02T08:21:10.360-05: -Starting test case: speedtest
2020-06-02T08:21:10.360-05: --Starting DTA ID=1
2020-06-02T08:21:10.360-05:   DTA(0001-123456789022222): Test start sent for speedtest
2020-06-02T08:21:10.360-05:  Confirming tests started for speedtest
2020-06-02T08:21:11.361-05: -Test case speedtest startup completed
2020-06-02T08:21:11.361-05: -Starting test case: iperf
2020-06-02T08:21:11.361-05: --Starting DTA ID=1
2020-06-02T08:21:11.361-05:   DTA(0001-123456789022222): Test start sent for iperf
2020-06-02T08:21:11.361-05:  Confirming tests started for iperf
2020-06-02T08:21:12.362-05: -Test case iperf startup completed
2020-06-02T08:21:12.362-05: Test Session startup completed

 

Open DTA Test Session

 

Select to open DTA Test Session opens a window to select a saved DTA Test Session to open.

Selected UE

Attribute / Value

Clicking on a single DTA will show more details in the bottom table.  Most of this information is sourced from the DTA Status message sent approximately every 30s, or else from other actions and messages.

ID

The ID will be a number from 1 to 10000.  The TAS can handle up to 10,000 IMEIs, but only supports 300 registered/live DTAs at any given time. A system can support up to 10K different phones being used at different times, no more than 300 at once.

IMEI

DTAs are uniquely identified by their IMEI, but the TAS will assign a more human readable ID to each IMEI, and this assignment will be persistent.

Phone Info

Details of Phone - based on what the phone reports to the TAS. If the phone supports 5G NR, you will see - 5G NR in this field.

Username

the Landslide username that was used to register the phone.

DTA User

Currently a free-form text field that each user on the DTA/phone fills out with their actual name, to provide another unique ID for the DTA, i.e. the actual persons name using the phone.

State

Similar to Test Server State, it tracks if DTA is READY, RUNNING or if status messages are missing, IDLE then OFFLINE. When tests are cancelled via the Cancel button, the state will change to CANCELING.

MDM

YES - if a DTA has MDM (Mobile Device Management) capability, otherwise set to NO.

Last Update

Indicates a label/action and timestamp of the last event that occurred on the DTA (helps with understanding the State).

Application

The name of the DTA Application.

Available Tests

Lists all the tests that the phone supports, which could be more/less than TAS can support (this table is showing information reported by the phone).

Running Tests

Lists all the running or queued up tests (DTA test cases).

Firebase ID

The unique Firebase ID for the phone used for communications to/from.

Selected DTA's Tests

The table lists the tests per selected DTA. The table lists:

RID - Run Id

Type - Type of test (e.g. dns, http, etc)

Username - User Name on Landslide

DTA User - User Name on DTA App

State - State of test (e.g. Running, Stopped, Queued, etc)

Last Update - Last update of test (e.g. Log, Status, Running, etc)

The TAS will prevent shutdown if DTA tests are running, it will require the force flag - Force Shutdown? The TAS is active, do you want to force the shutdown (Yes or No).

Real-Time Logs:

NOTEs:    
  • The TAS will monitor the license and if it is exhausted, the TAS will add the following Warning messages to the Real-Time Logs: "Test results received, but license is exhausted", "DTA tests could not be executed: License exhausted".
  • In addition, the TAS startup log will include the starting point of the remaining tests. The following messages will be sent to the log (see example below) :
    • Setting Results ThreadPool to 3
    • Results Storage Max Size =10g
    • Initializing DTA services and connecting to Firebase
    • Remaining licensed DTA Tests: 20
    • Loading the test cases
    • Total DTA Tests Handled: 0
  • When a DTA test result is reported to the TAS, the TAS will always log a Real-Time Log message about TEST.PROCS conversion, either that a file is produced or error occurred.
  • When a DTA test result is reported to the TAS but does not get converted into TEST.PROCS, the TAS will provide a ConversionLog file that could be helpful for troubleshooting.
  • When the DTA sends test log (log reporting is optional) to the TAS and they arrive late (after the results) or if the TAS just doesn’t know about the given TEST-ID , this will result in the file being named “test.log”, instead of by the test type/name, e.g. “speedtest.log”.   They will be given a prefix timestamp that matches current time, but also the same Client ID and RUN ID as the test it was associated with.

Real-Time Logs: Using DTA you will want to watch/search the REAL-Time Logs, as that is where all actions are logged.

DTA Test Session actions will include a prefix of  DTA(<ID>):, e.g. DTS(1221), where the ID is an incremental integer 1 to 9999, just to uniquely ID the DTA Test Session object that starts DTA Test Cases.. example search:

USER                  DATE-TIME                SEVERITY       TYPE         ACTION    MESSAGE

sms                   2019-12-10 07:30:01  Informational  Operations  DTA       DTS(1031): Starting androidlongtimetest

sms                   2019-12-10 07:30:01  Informational  Operations  DTA       DTS(1031): Confirming tests started for mobile_info

sms                   2019-12-10 07:30:01  Warning          Operations  DTA       DTS(1031): DTA(62-?): No longer registered, skipping start of mobile_info and any others

sms                   2019-12-10 07:30:02  Informational  Operations  DTA       DTS(1031): Test case mobile_info startup completed

sms                   2019-12-10 07:30:02  Informational  Operations  DTA       DTS(1031): Confirming tests started for video

sms                   2019-12-10 07:30:03  Informational  Operations  DTA       DTS(1031): Test case video startup completed

sms                   2019-12-10 07:30:03  Informational  Operations  DTA       DTS(1031): Confirming tests started for http

sms                   2019-12-10 07:30:04  Informational  Operations  DTA       DTS(1031): Test case http startup completed

sms                   2019-12-10 07:30:04  Informational  Operations  DTA       DTS(1031): Confirming tests started for dns

sms                   2019-12-10 07:30:05  Informational  Operations  DTA       DTS(1031): Test case dns startup completed

sms                   2019-12-10 07:30:05  Informational  Operations  DTA       DTS(1031): Confirming tests started for latency

sms                   2019-12-10 07:30:06  Informational  Operations  DTA       DTS(1031): Test case latency startup completed

sms                   2019-12-10 07:30:06  Informational  Operations  DTA       DTS(1031): Confirming tests started for ping

sms                   2019-12-10 07:30:07  Informational  Operations  DTA       DTS(1031): Test case ping startup completed

sms                   2019-12-10 07:30:07  Informational  Operations  DTA       DTS(1031): Confirming tests started for speedtest

sms                   2019-12-10 07:30:08  Informational  Operations  DTA       DTS(1031): Test case speedtest startup completed

sms                   2019-12-10 07:30:08  Informational  Operations  DTA       DTS(1031): Confirming tests started for dns

sms                   2019-12-10 07:30:09  Informational  Operations  DTA       DTS(1031): Test case dns startup completed

sms                   2019-12-10 07:30:09  Informational  Operations  DTA       DTS(1031): Test Session startup completed

 

DTAs themselves are IDed in log messages with prefix DTA(<ID>-<IMEI>), e.g. DTA(0025-356169094417360)

<system>             2019-12-09 14:30:04  Informational  Operations  DTA       DTA(0032-355257099884386): Test Start Confirmed ID:355257099884386+92

sms                      2019-12-09 14:30:04  Informational  Operations  DTA       DTA(0032-355257099884386): Test start sent for dns

sms                      2019-12-09 14:30:05  Informational  Operations  DTA       DTA(0032-355257099884386)-Test(93/latency): Test Added, queued

<system>             2019-12-09 14:30:05  Informational  Operations  DTA       DTA(0032-355257099884386): Test Start Confirmed ID:355257099884386+93

sms                      2019-12-09 14:30:05  Informational  Operations  DTA       DTA(0032-355257099884386): Test start sent for latency

<system>             2019-12-09 14:30:06  Informational  Operations  DTA       DTA(0032-355257099884386): Test Start Confirmed ID:355257099884386+94

sms                      2019-12-09 14:30:06  Informational  Operations  DTA       DTA(0032-355257099884386)-Test(94/ping): Test Added, queued

sms                      2019-12-09 14:30:06  Informational  Operations  DTA       DTA(0032-355257099884386): Test start sent for ping

sms                      2019-12-09 14:30:07  Informational  Operations  DTA       DTA(0032-355257099884386)-Test(95/speedtest): Test Added, queued

<system>             2019-12-09 14:30:07  Informational  Operations  DTA       DTA(0032-355257099884386): Test Start Confirmed ID:355257099884386+95

sms                      2019-12-09 14:30:07  Informational  Operations  DTA       DTA(0032-355257099884386): Test start sent for speedtest

sms                      2019-12-09 14:30:08  Informational  Operations  DTA       DTA(0032-355257099884386)-Test(96/dns): Test Added, queued

<system>             2019-12-09 14:30:08  Informational  Operations  DTA       DTA(0032-355257099884386): Test Start Confirmed ID:355257099884386+96

sms                      2019-12-09 14:30:08  Informational  Operations  DTA       DTA(0032-355257099884386): Test start sent for dns

sms                      2019-12-09 14:30:19  Informational  Operations  DTA       DTA(0032-355257099884386)-Test(89/mobile_info): Results Saved: 19-12-09_13.30.19__0032-355257099884386__89__TAS__mobile_info.txt

sms                      2019-12-09 14:30:21  Informational  Operations  DTA       DTA(0032-355257099884386)-Test(90/video): Started, RID=90

sms                      2019-12-09 14:31:31  Informational  Operations  DTA       DTA(0032-355257099884386)-Test(90/video): Results Saved: 19-12-09_13.31.31__0032-355257099884386__90__TAS__video.txt

sms                      2019-12-09 14:31:32  Informational  Operations  DTA       DTA(0032-355257099884386)-Test(91/http): Started, RID=91

sms                      2019-12-09 14:31:44  Informational  Operations  DTA       DTA(0032-355257099884386)-Test(91/http): Results Saved: 19-12-09_13.31.44__0032-355257099884386__91__TAS__http.txt

 

DTA Test Cases that are executed are specifically are IDed with prefix DTA(<ID>-<IMEI>)-Test(<RUN_ID>/<TestCase>), e.g. DTA(0032-355257099884386)-Test(94/ping):

USER                  DATE-TIME                SEVERITY       TYPE        ACTION    MESSAGE

sms                   2019-12-09 14:30:06  Informational  Operations  DTA       DTA(0032-355257099884386)-Test(94/ping): Test Added, queued

sms                   2019-12-09 14:31:57  Informational  Operations  DTA       DTA(0032-355257099884386)-Test(94/ping): Started, RID=94

sms                   2019-12-09 14:32:07  Informational  Operations  DTA       DTA(0032-355257099884386)-Test(94/ping): Results Saved: 19-12-09_13.32.07__0032-355257099884386__94__TAS__ping.txt

The RUN ID is a semi-unique number 1-9999 created by the DTA for each DTA Test Case execution.  Over a significant period of time, the RUN-ID is a unique ID for a test execution on any DTA. When looking for test results you can use DTA ID and DTA Test RUN-ID in both REAL-TIME logs and Results.

Results use similar IDs/Prefixes.   The general prefix for DTA produced files:   <START_DATETIME>__<DTA_ID>-<DTA_IMEI>__<RUN_ID>__<DTA USER>__<TEST CASE>, this can include the .log (Run Log) and .txt (Results).  For the TEST_PROCS_UE produced by the TAS, it will be <START_DATETIME>__<DTA_ID>-<DTA_IMEI>__<RUN_ID>__TEST.PROCS_UE.<VW_TIMESTAMP>.csv.

Examples:

 

When executing tests from the TAS, the DTA Username is fixed at “TAS”.  Also the test results will go to the results page for the user that is logged into the TAS. When executing tests from the DTA/Phone itself, test results will go to the results page for the Landslide Username that the phone registered with and show the “DTA User” used during DTA Login.


Example:

 

These naming standards can be useful for querying the results website, i.e. for __<DTA_USERNAME>__, or __<RUN_ID>__ or __<DTA_ID>-, or __<TEST_CASE>.txt