| [ Platform Documentation ] | [ Title ] | [ Contents ] | [ Previous ] | [ Next ] | [ Index ] |
- About LSF Resources
- How Resources are Classified
- How LSF Uses Resources
- Load Indices
- Static Resources
- Automatic Detection of Hardware Reconfiguration
[ Top ]
About LSF Resources
The LSF system uses built-in and configured resources to track job resource requirements and schedule jobs according to the resources available on individual hosts.
Viewing available resources
Use lsinfo to list the resources available in your cluster. The lsinfo
command lists all the resource names and their descriptions:
% lsinfo
RESOURCE_NAME TYPE ORDER DESCRIPTION
r15s Numeric Inc 15-second CPU run queue length
r1m Numeric Inc 1-minute CPU run queue length (alias:cpu)
r15m Numeric Inc 15-minute CPU run queue length
ut Numeric Inc 1-minute CPU utilization (0.0 to 1.0)
pg Numeric Inc Paging rate (pages/second)
io Numeric Inc Disk IO rate (Kbytes/second)
ls Numeric Inc Number of login sessions (alias: login)
it Numeric Dec Idle time (minutes) (alias: idle)
tmp Numeric Dec Disk space in /tmp (Mbytes)
swp Numeric Dec Available swap space (Mbytes) (alias:swap)
mem Numeric Dec Available memory (Mbytes)
ncpus Numeric Dec Number of CPUs
ndisks Numeric Dec Number of local disks
maxmem Numeric Dec Maximum memory (Mbytes)
maxswp Numeric Dec Maximum swap space (Mbytes)
maxtmp Numeric Dec Maximum /tmp space (Mbytes)
cpuf Numeric Dec CPU factor
rexpri Numeric N/A Remote execution priority
server Boolean N/A LSF server host
irix Boolean N/A IRIX UNIX
hpux Boolean N/A HP_UX
solaris Boolean N/A Sun Solaris
cserver Boolean N/A Compute server
fserver Boolean N/A File server
aix Boolean N/A AIX UNIX
type String N/A Host type
model String N/A Host model
status String N/A Host status
hname String N/A Host name
TYPE_NAME
HPPA
SGI6
ALPHA
SUNSOL
RS6K
NTX86
MODEL_NAME CPU_FACTOR
DEC3000 10.00
R10K 14.00
PENT200 6.00
IBM350 7.00
SunSparc 6.00
HP735 9.00
HP715 5.00
Use lshosts to get a list of the resources defined on a specific host:
% lshosts hostA
HOST_NAME type model cpuf ncpus maxmem maxswp server RESOURCES
hostA SOL732 Ultra2 20.2 2 256M 679M Yes ()
Viewing host load by resource
Use lshosts -s to view host load by shared resource:
% lshosts -s
RESOURCE VALUE LOCATION
tot_lic 5 host1 host2
tot_scratch 500 host1 host2
The above output indicates that 5 licenses are available, and that the shared scratch directory currently contains 500 MB of space.
The VALUE field indicates the amount of that resource. The LOCATION column
shows the hosts which share this resource. The lshosts -s command
displays static shared resources. The lsload -s command displays dynamic
shared resources.
[ Top ]
How Resources are Classified
Boolean resources
Boolean resources (for example, server to denote LSF server hosts) have a
value of one (1) if they are defined for a host, and zero (0) if they are not
defined for the host. Use Boolean resources to configure host attributes to be
used in selecting hosts to run jobs. For example:
- Machines may have different types and versions of operating systems.
- Machines may play different roles in the system, such as file server or compute server.
- Some machines may have special-purpose devices needed by some applications.
- Certain software packages or licenses may be available only on some of the machines.
Specify a Boolean resource in a resource requirement selection string of a job to select only hosts that can run the job. For example,
Some examples of Boolean resources:
Shared resources
Shared resources are configured resources that are not tied to a specific host, but are associated with the entire cluster, or a specific subset of hosts within the cluster. For example:
- Floating licenses for software packages
- Disk space on a file server which is mounted by several machines
- The physical network connecting the hosts
An application may use a shared resource by running on any host from which that resource is accessible. For example, in a cluster in which each host has a local disk but can also access a disk on a file server, the disk on the file server is a shared resource, and the local disk is a host-based resource. In contrast to host-based resources such as memory or swap space, using a shared resource from one machine affects the availability of that resource as seen by other machines. There will be one value for the entire cluster which measures the utilization of the shared resource, but each host-based resource is measured separately.
LSF does not contain any built-in shared resources. All shared resources must be configured by the LSF administrator. A shared resource may be configured to be dynamic or static. In the above example, the total space on the shared disk may be static while the amount of space currently free is dynamic. A site may also configure the shared resource to report numeric, string or Boolean values.
The following restrictions apply to the use of shared resources in LSF products.
-
A shared resource cannot be used as a load threshold in the
Hostssection of thelsf.cluster.cluster_name file. -
A shared resource cannot be used in the
loadSched/loadStopthresholds, or in the STOP_COND or RESUME_COND parameters in the queue definition in thelsb.queuesfile.
Viewing shared resources for hosts
Run bhosts -s to view shared resources for hosts. For example:
% bhosts -s
RESOURCE TOTAL RESERVED LOCATION
tot_lic 5 0.0 hostA hostB
tot_scratch 00 0.0 hostA hostB
avail_lic 2 3.0 hostA hostB
avail_scratch 100 400.0 hostA hostB
The TOTAL column displays the value of the resource. For dynamic resources,
the RESERVED column displays the amount that has been reserved by running
jobs.
[ Top ]
How LSF Uses Resources
Jobs submitted through the LSF system will have the resources they use monitored while they are running. This information is used to enforce resource usage limits and load thresholds as well as for fairshare scheduling.
LSF collects information such as:
- Total CPU time consumed by all processes in the job
- Total resident memory usage in KB of all currently running processes in a job
- Total virtual memory usage in KB of all currently running processes in a job
- Currently active process group ID in a job
- Currently active processes in a job
On UNIX, job-level resource usage is collected through a special process called PIM (Process Information Manager). PIM is managed internally by LSF.
Viewing job resource usage
The -l option of the bjobs command displays the current resource usage of
the job. The usage information is sampled by PIM every 30 seconds and
collected by sbatchd at a maximum frequency of every SBD_SLEEP_TIME
(configured in the lsb.params file) and sent to mbatchd. The update is done
only if the value for the CPU time, resident memory usage, or virtual memory
usage has changed by more than 10 percent from the previous update, or if a
new process or process group has been created.
Viewing load on a host
Use bhosts -l to check the load levels on the host, and adjust the suspending
conditions of the host or queue if necessary. The bhosts -l command gives
the most recent load values used for the scheduling of jobs. A dash (-) in the
output indicates that the particular threshold is not defined.
% bhosts -l hostB
HOST: hostB
STATUS CPUF JL/U MAX NJOBS RUN SSUSP USUSP RSV
ok 20.00 2 2 0 0 0 0 0
CURRENT LOAD USED FOR SCHEDULING:
r15s r1m r15m ut pg io ls t tmp swp mem
Total 0.3 0.8 0.9 61% 3.8 72 26 0 6M 253M 297M
Reserved 0.0 0.0 0.0 0% 0.0 0 0 0 0M 0M 0M
LOAD THRESHOLD USED FOR SCHEDULING:
r15s r1m r15m ut pg io ls it tmp swp me
m
loadSched - - - - - - - - - - -
loadStop - - - - - - - - - - -
[ Top ]
Load Indices
Load indices are built-in resources that measure the availability of dynamic, non-shared resources on hosts in the LSF cluster.
Load indices built into the LIM are updated at fixed time intervals.
External load indices are defined and configured by the LSF administrator. An External Load Information Manager (ELIM) program collects the values of site-defined external load indices and updates LIM when new values are received.
Load indices collected by LIM
.
Status
The status index is a string indicating the current status of the host. This
status applies to the LIM and RES.
The possible values for status are:
CPU run queue lengths (r15s, r1m, r15m)
The r15s, r1m and r15m load indices are the 15-second, 1-minute and 15-
minute average CPU run queue lengths. This is the average number of
processes ready to use the CPU during the given interval.
On UNIX, run queue length indices are not necessarily the same as the load
averages printed by the uptime(1) command; uptime load averages on some
platforms also include processes that are in short-term wait states (such as
paging or disk I/O).
On multiprocessor systems, more than one process can execute at a time. LSF scales the run queue value on multiprocessor systems to make the CPU load of uniprocessors and multiprocessors comparable. The scaled value is called the effective run queue length.
Use lsload -E to view the effective run queue length.
LSF also adjusts the CPU run queue based on the relative speeds of the processors (the CPU factor). The normalized run queue length is adjusted for both number of processors and CPU speed. The host with the lowest normalized run queue length will run a CPU-intensive job the fastest.
Use lsload -N to view the normalized CPU run queue lengths.
CPU utilization (ut)
The ut index measures CPU utilization, which is the percentage of time spent
running system and user code. A host with no process running has a ut value
of 0 percent; a host on which the CPU is completely loaded has a ut of 100
percent.
Paging rate (pg)
The pg index gives the virtual memory paging rate in pages per second. This
index is closely tied to the amount of available RAM memory and the total size
of the processes running on a host; if there is not enough RAM to satisfy all
processes, the paging rate will be high. Paging rate is a good measure of how
a machine will respond to interactive use; a machine that is paging heavily
feels very slow.
Login sessions (ls)
The ls index gives the number of users logged in. Each user is counted once,
no matter how many times they have logged into the host.
Interactive idle time (it)
On UNIX, the it index is the interactive idle time of the host, in minutes. Idle
time is measured from the last input or output on a directly attached terminal
or a network pseudo-terminal supporting a login session. This does not include
activity directly through the X server such as CAD applications or emacs
windows, except on Solaris and HP-UX systems.
On Windows NT, the it index is based on the time a screen saver has been
active on a particular host.
Temporary directories (tmp)
The tmp index is the space available in MB on the file system that contains the
temporary directory:
Swap space (swp)
The swp index gives the currently available virtual memory (swap space) in
MB. This represents the largest process that can be started on the host.
Memory (mem)
The mem index is an estimate of the real memory currently available to user
processes. This represents the approximate size of the largest process that
could be started on a host without causing the host to start paging.
LIM reports the amount of free memory available. LSF calculates free memory
as a sum of physical free memory, cached memory, buffered memory and an
adjustment value. The command vmstat also reports free memory but displays
these values separately. There may be a difference between the free memory
reported by LIM and the free memory reported by vmstat because of virtual
memory behavior variations among operating systems. You can write an ELIM
that overrides the free memory values returned by LIM.
I/O rate (io)
The io index measures I/O throughput to disks attached directly to this host,
in KB per second. It does not include I/O to disks that are mounted from other
hosts.
Viewing information about load indices
The lsinfo -l command displays all information available about load indices
in the system. You can also specify load indices on the command line to
display information about selected indices:
% lsinfo -l swp
RESOURCE_NAME: swp
DESCRIPTION: Available swap space (Mbytes) (alias: swap)
TYPE ORDER INTERVAL BUILTIN DYNAMIC RELEASE
Numeric Dec 60 Yes Yes NO
The lsload -l command displays the values of all load indices. External load
indices are configured by your LSF administrator:
% lsload
HOST_NAME status r15s r1m r15m ut pg ls it tmp swp mem
hostN ok 0.0 0.0 0.1 1% 0.0 1 224 43M 67M 3M
hostK -ok 0.0 0.0 0.0 3% 0.0 3 0 38M 40M 7M
hostF busy 0.1 0.1 0.3 7% *17 6 0 9M 23M 28M
hostG busy *6.2 6.9 9.5 85% 1.1 30 0 5M 400M 385M
hostV unavail
[ Top ]
Static Resources
Static resources are built-in resources that represent host information that does not change over time, such as the maximum RAM available to user processes or the number of processors in a machine. Most static resources are determined by the LIM at start-up time, or when LSF detects hardware configuration changes.
Static resources can be used to select appropriate hosts for particular jobs based on binary architecture, relative CPU speed, and system configuration.
The resources ncpus, maxmem, maxswp, and maxtmp are not static on UNIX
hosts that support dynamic hardware reconfiguration.
Static resources reported by LIM
CPU factor (cpuf)
The CPU factor is the speed of the host's CPU relative to other hosts in the cluster. If one processor is twice the speed of another, its CPU factor should be twice as large. The CPU factors are defined by the LSF administrator. For multiprocessor hosts, the CPU factor is the speed of a single processor; LSF automatically scales the host CPU load to account for additional processors.
Server
The server static resource is Boolean. It has the following values:
- 1 if the host is configured to run jobs from other hosts
- 0 if the host is an LSF client for submitting jobs to other hosts
[ Top ]
Automatic Detection of Hardware Reconfiguration
Some UNIX operating systems support dynamic hardware reconfiguration-- that is, the attaching or detaching of system boards in a live system without having to reboot the host.
Supported platforms
LSF is able to recognize changes in ncpus, maxmem, maxswp, maxtmp in the
following platforms:
Dynamic changes in ncpus
LSF is able to automatically detect a change in the number of processors in systems that support dynamic hardware reconfiguration.
The local LIM checks if there is a change in the number of processors at an
internal interval of 2 minutes. If it detects a change in the number of
processors, the local LIM also checks maxmem, maxswp, maxtmp. The local LIM
then sends this new information to the master LIM.
Dynamic changes in maxmem, maxswp, maxtmp
If you dynamically change maxmem, maxswp, or maxtmp without changing the
number of processors, you need to restart the local LIM with the command
lsadmin limrestart so that it can recognize the changes.
If you dynamically change the number of processors and any of maxmem,
maxswp, or maxtmp, the change will be automatically recognized by LSF. When
it detects a change in the number of processors, the local LIM also checks
maxmem, maxswp, maxtmp.
Viewing dynamic hardware changes
There may be a 2 minute delay before the changes are recognized by lsxxx
commands (for example, before lshosts displays the changes).
There may be at most a 2 + 10 minute delay before the changes are recognized
by bxxx commands (for example, before bhosts -l displays the changes).
This is because mbatchd contacts the master LIM at an internal interval of 10
minutes.
Configuration changes from a local cluster are communicated from the master
LIM to the remote cluster at an interval of 2 * CACHE_INTERVAL. The
parameter CACHE_INTERVAL is configured in lsf.cluster.cluster_name
and is by default 60 seconds.
This means that for changes to be recognized in a remote cluster there is a maximum delay of 2 minutes + 2*CACHE_INTERVAL.
How dynamic hardware changes affect LSF
LSF uses ncpus, maxmem, maxswp, maxtmp to make scheduling and load
decisions.
When processors are added or removed, LSF licensing is affected because LSF licenses are based on the number of processors.
If you put a processor offline:
- Per host or per-queue load thresholds may be exceeded sooner. This is because LSF uses the number of CPUS and relative CPU speeds to calculate effective run queue length.
-
The value of CPU run queue lengths (
r15s,r1m, andr15m) increases. - Jobs may also be suspended or not dispatched because of load thresholds.
-
Per-processor job slot limit (PJOB_LIMIT in
lsb.queues) may be exceeded sooner.
If you put a new processor online:
- Load thresholds may be reached later.
- The value of CPU run queue lengths (r15s, r1m, and r15m) is decreased.
- Jobs suspended due to load thresholds may be resumed.
Per-processor job slot limit (PJOB_LIMIT in lsb.queues) may be reached
later.
[ Top ]
| [ Platform Documentation ] | [ Title ] | [ Contents ] | [ Previous ] | [ Next ] | [ Index ] |
|
Date Modified: January 12, 2004
Platform Computing: www.platform.com Platform Support: support@platform.com Platform Information Development: doc@platform.com Copyright © 1994-2004 Platform Computing Corporation. All rights reserved. |