Results Storage
Landslide stores all test result files on the TAS. Test result files are available for viewing and downloading via the Results Website - Results Website and Landslide Home Page Results Center (Beta, new for 24.3). The TAS provides an automatic cleanup of test results based the configuration of rsl_high and rsl_low threshold settings. When the result files take up rsl_high, the TAS will delete enough of the oldest files to bring the storage down to rsl_low. In addition to automatic cleanup, there is Results Administration available to manually delete files.
Overview
Configuration
Startup
Status
Logs
A Test Manager (hosting the TAS) comes with a single hard drive by default, but we also allow a second hard drive to be mounted and dedicated for the TAS system files and results files. Review your system hardware and installation guides before setting up custom rsl_high/rsl_low settings. The TAS will determine Results Storage hard drive based on what is mounted at /usr/sms/data/results. Regardless, a Landslide system administrator must do some analysis to properly configure their TAS for their own purposes. Consider how the TAS hard drive will be used:
Definitions:
- Total HD Space == How many bytes the disk partition has.
- rsl_high aka the Total HD Allocated to Results == How many bytes of the disk partition are expected to be used for Results Storage.
- Total System Files Space == How many bytes of disk partition are available for System Files ( Total HD Space – rsl_high )
- Remaining System Files Space == Total HD Space - ( rsl_high + <Disk space already used up for system files> ) OR Total System Files Space - <Disk space already used up by system files>
The limiting factor is the Total HD Space, everything subtracts from that. The Results Storage should be sized to allow enough space for the expected system files (Upgrades/Backups), and the system administrator should ensure that system files never take up more than that allocated space. There also should be a small buffer above rsl_high in case many tests are executed right before reaching rsl_high. Consider the case that many tests are started right as Results Storage is 1MB below rsl_high, whatever files the individual test create over rsl_high will not be reacted to until they are completed. rsl_high needs be low enough to make sure other system files + rsl_high + max-overage-from-one-test-run < total disk space.
The TAS will determine rsl_high and rsl_low based on the following: TAS Settings, License Keys or Hard Drive size.
If the TAS Settings include rsl_high and rsl_low, they will be applied.
Else if the license includes rsl_high and rsl_low, they will be applied.
Else the TAS will use ½ of the hard drive total space for rsl_high and rsl_low will be set to higher of 2GiB less than rsl_high or 40% of the hard drive.
By default a system is licensed for rsl_high=10000MiB and rsl_low=9000MiB so this will be the most common configuration.
However system administrators can override these default values using TAS Settings rsl_high and rsl_low.
rsl_low should be set no less than 2GiB below rsl_high. The difference between rsl_high and rsl_low is the working range for testing. The goal is to have a minimal working range to avoid having very long running cleanups, but enough to also not be executing cleanups very often. In general, as long as the working range is less than 2GiB, the cleanup will take under 2 minutes.
When the TAS starts, the Results Storage must be scanned and indexed. For systems with smaller hard drives under 1TB, this should be completed quick enough to not be noticed. But on systems with 500GiB and larger hard drives and rsl_high set higher than 400MiB, the initialization can take several minutes. If it takes longer than 5 minutes the TAS startup will be allowed to complete without waiting. But until the indexing completes, running tests will not be possible, with an error message: "Unable to start test, TAS is still initializing Results Storage"
The TAS startup log (tasoutput.txt) will also indicate some information about the Results Storage. When the TAS does start you will get the following information:
The TAS startup log (tasoutput.txt) will also indicate some information about the Results Storage. When the TAS does start you will get the following information:
Setting Results ThreadPool to 8
Results Storage Max Size=48.83GiB
Space allocated for other files=303697MiB
Initializing Result File Indexes
Platform: "C100"
…
Loading the root test cases
...
25% complete
50% complete
75% complete
Caching TCs
....................................................................................................Done
Initializing Other Components
Waiting for Initialization of Result File Indexes
...........................
TAS started version 24.2.0.1
If you attempt to start the TAS with rsl_high set >= the size of the HD Partition, it will fail to start with these messages:
CHECKING LINUX CONFIGURATION
Requesting License Checkout
TAS will go to Locked state after being disconnected from License Server for 30 minutes
Setting Results ThreadPool to 3
Results Storage Max Size=73.24GiB
--ERROR: Result Storage is set higher than total disk space of 66060MiB!!
--WARNING: The system should be upgraded with a .upg/.tsu to update OS level services.
Product Loaded: Landslide Version: TAS-99.9.0.2020-08-25.3drd-(internal)
Max TAS Memory: 3014MB
OS: Linux:2.6.31-14-server
Timezone: America/Chicago
Startup: Tue Aug 25 12:21:24 CDT 2020
Checking Tables:
Tables OK
Initializing SNMP trap sender
Results Storage Oldest File=2013-02-06+11:19:07
Results Storage Current Size=8.25GiB
TAS restarted same day, refreshing daily count based on hourly counts last hour@0
Initializing Test Servers
Default TS version not specified
Nagios Manager not installed,
TS Host agent registration disabled
Initializing other TAS services*Exception: com.sseworks.sp.server.framework.E while starting system: Hard disk partition is not big enough to support rsl_high setting
TAS startup failed
License Server Error: None
CHECKING LINUX CONFIGURATION
License Fingerprint: 95701E6OWM=f0eQ0RTMx38d1OMQ==756A1MS4=38d1OMQ==
When the TAS does start you will get this information:
Setting Results ThreadPool to 3
Results Storage Max Size=60.55GiB
Space allocated for other files=4060MiB
If the Indexes are not initialized within ~5min TAS is allowed to start but a message is printed:
Waiting for Initialization of Result File Indexes
....................................................................................................................................................................................
Result File Indexes have not be completed, tests will not be able to start until completed
TAS started version 99.9.0.2024-05-06.1pv
Results Storage can be monitored via System Status. The Results Storage progress bar displays the rsl_high, rsl_low and the current result file usage. However, the progress bar is not in a linear scale. The bottom half shows a range from 0 bytes to rsl_low and the top half shows the range from rsl_low to rsl_high. Once the usage is above rsl_low, the value will be linear from rsl_low to rsl_high. When the bar is blue the value displayed is a rough estimate of the results usage and will tend to be higher than the actual usage because it includes some read-only disk usage. Consider that once your TAS is running for a long time, you will never go below rsl_low. The tooltip will include a more precise reporting of the usage as well as the Oldest file saved indication. The Oldest file saved indication is set when known and refreshed at best every 30s. The time shown provides an indication of the historical results available, as that is the oldest file in Results Storage.
The progress bar provides a visual indication of how Results Storage is doing. The first/bottom ~1/3rd of the progress bar shows 0 to rsl_low bytes.
In our example, it goes from 0 to 39GiB. The next ~50%
In our example that goes from 818GIB,
Once the estimate of storage reaches a threshold of 2/3rds between rsl_low and rsl_high the bar turns orange.
The bar will turn red once it reaches rsl_high, but should only remain red for a short period of time.
The results cleanup should be started very soon after the bar is red.
Progress bar examples:
Reaching exactly rsl_max :
Going over rsl_max :
Results file cleanup is logged to the Real-Time Logs, with the Verb=Results. Searching the Real-Time Logs for Verb=Results (Log Search Criteria), to see when cleanups have occurred.
Example of Log Search with all Log Levels checked (Operations, Sys. Admin and System) and all Severity levels (Informational, Warning, Error and Fatal Error) and Verb = Results :
Example: showing start/stop of the cleanup -
