| [ Platform Documentation ] | [ Title ] | [ Contents ] | [ Previous ] | [ Next ] | [ Index ] |
[ Top ]
About Preemptive Scheduling
Preemptive scheduling lets a pending high-priority job take resources away from a running job of lower priority. When 2 jobs compete for the same resource, LSF automatically suspends the low-priority job to make resources available to the high-priority job. The low-priority job is resumed as soon as possible.
Use preemptive scheduling if you have long-running low-priority jobs causing high-priority jobs to wait an unacceptably long time.
The following types of jobs cannot be preempted:
Preemptive and preemptable queues
Jobs in a preemptive queue can preempt jobs in any queue of lower priority, even if the low-priority queues are not specified as preemptable.
Jobs in a preemptable queue can be preempted by jobs from any queue of a higher priority, even if the high-priority queues are not specified as preemptive.
Preemptive and preemptable jobs
Preemptive jobs are pending in a high-priority queue and require the specified resource. Their queue must be able to preempt the low-priority queue.
Preemptable jobs are running in a low-priority queue and are holding the specified resource. Their queue must be able to be preempted by the high- priority queue.
[ Top ]
How Preemptive Scheduling Works
Preemptive scheduling occurs when two jobs compete for the same resource. If a high-priority job is pending, LSF can suspend a lower priority job that is running, and then start the high-priority job using the job slot that becomes available. For this to happen, the high-priority job must be pending in a preemptive queue, or the low-priority job must belong to a preemptable queue.
By default, when multiple preemptable jobs exist (low-priority jobs holding the required resource), LSF preempts a job from the least-loaded host.
The preempted job is resumed as soon as more resources become available; it does not necessarily have to wait for the preempting job to finish.
Queues that can preempt others are more aggressive in scheduling jobs because a resource that is not available to a low-priority queue might be available (by preemption) to a high-priority queue.
By default, job slot limits are enforced based on the number of job slots taken by running and suspended jobs. With preemptive scheduling, the suspended jobs don't count against certain job slot limits. This means that when one job is suspended, another job can be started in its place.
LSF makes sure that the number of running jobs never exceeds the job slot limits. When LSF tries to place a preemptive job, LSF considers each job slot limit, but for certain job slot limits, LSF only counts the job slots used by running jobs that are not preemptable. Then, if starting the preemptive job would violate job slot limits, LSF suspends one or more low-priority jobs.
Job slot limits affected by preemptive scheduling
When you enable preemptive scheduling, you automatically affect the following job slot limits:
-
Total job slot limit for hosts, specified at the host level (SLOTS and HOSTS in
lsb.resources) -
Total job slot limit for individual users, specified at the user level (SLOTS and USERS in
lsb.resources); by default, suspended jobs still count against the limit for user groups
You can configure preemptive scheduling to affect following limits:
-
Total job slot limit for user groups, specified at the user level (SLOTS and USERS in
lsb.resources); if preemptive scheduling is enabled, suspended jobs never count against the limit for individual users -
Total number of jobs for users and user groups, specified at the host level (SLOTS, PER_USER=all, and HOSTS in
lsb.resources) -
Per-processor job slot limit for individual users, specified at the user level (SLOTS_PER_PROCESSOR, USERS, and PER_HOST=all in
lsb.resources) -
Per-processor job slot limit for user groups, specified at the user level (SLOTS_PER_PROCESSOR, USERS, and PER_HOST=all in
lsb.resources)
Job slot limits specified at the queue level are never affected by preemptive scheduling; they are always enforced for both running and suspended jobs.
Preemption of multiple job slots
If multiple resources are required, LSF can preempt multiple jobs, until sufficient resources are available. For example, one or more jobs might be preempted for a job that needs multiple job slots.
Preemption of parallel jobs
When a high-priority parallel job preempts multiple low-priority parallel jobs, sometimes LSF preempts more low-priority jobs than are necessary to release sufficient job slots to start the high-priority job.
Use the PREEMPT_FOR parameter in lsb.params to enable the optimized
preemption of parallel jobs, so LSF preempts fewer of the low-priority parallel
jobs.
See Optimized Preemption of Parallel Jobs for more information.
[ Top ]
Configuring Preemptive Scheduling
To configure preemptive scheduling, make at least one queue in the cluster preemptive (not the lowest-priority queue) or preemptable (not the highest- priority queue).
To make a queue preemptive or preemptable, set PREEMPTION in the queue
definition (lsb.queues) to PREEMPTIVE or PREEMPTABLE. A queue can be
both: its jobs can always preempt jobs in lower priority queues, and can always
be preempted by jobs from higher priority queues.
PREEMPTION = PREEMPTIVE[[queue_name[+pref_level]...]]PREEMPTABLE[[queue_name...]]
When you specify a list of queues, you must enclose the list in one set of square brackets.
PREEMPTIVE[[queue_name[+pref_level] ...]]
Defines a preemptive queue. Jobs in this queue can preempt jobs from specified lower priority queues only, or from all lower priority queues by default if the parameter is specified with no queue names.
If you specify a list of lower-priority preemptable queues, you must enclose
the list in one set of square brackets. To indicate an order of preference for the
lower-priority queues, put a plus sign (+) after the names of queues and a
preference level as a positive integer. See Configuring preemptable queue
preference for more information.
PREEMPTABLE [[queue_name ...]]
Defines a preemptable queue. Jobs in this queue can be preempted by jobs from specified higher-priority queues, or from all higher-priority queues by default (if the parameter is specified with no queue names), even if the higher- priority queues are not preemptive.
If you specify a list of higher-priority queues, you must enclose the list in one set of square brackets.
In these examples, assume Queue 1 has highest priority and Queue 4 the lowest.
-
If the following settings are in
lsb.queues:QUEUE_NAME=Queue1 PREEMPTION=PREEMPTIVE
Queue 1 is preemptive, so it can preempt jobs in all lower-priority queues (Queues 2, 3 and 4).
-
If the following settings are in
lsb.queues:QUEUE_NAME=Queue1 PREEMPTION=PREEMPTIVE[Queue3 Queue4]
Queue 1 is preemptive, but can only preempt jobs in Queues 3 and 4, not Queue 2.
-
If the following settings are in
lsb.queues:QUEUE_NAME=Queue3 PREEMPTION=PREEMPTIVE PREEMPTABLE
Queue 3 is preemptive and preemptable, so it can preempt jobs in all lower-priority queues (Queue 4), and its jobs can be preempted by all higher-priority queues (Queues 1 and 2).
Configuring additional job slot limits for preemptive scheduling
The following job slot limits are always affected by preemptive scheduling:
-
Total job slot limit for hosts, specified at the host level (SLOTS and HOSTS in
lsb.resources) -
Total job slot limit for individual users, specified at the user level (SLOTS and USERS in
lsb.resources); by default, suspended jobs still count against the limit for user groups
To configure additional job slot limits to be affected by preemptive scheduling,
set PREEMPT_FOR in lsb.params, and use one or more of the following
keywords to indicate that suspended jobs do not count against that job slot
limit:
-
GROUP_MAX--total job slot limit for user groups, specified at the user level (MAX_JOBS in
lsb.users); if preemptive scheduling is enabled, suspended jobs never count against the limit for individual users -
HOST_JLU--total number of jobs for users and user groups, specified at the host level (JL/U in
lsb.hosts) -
USER_JLP--user-processor job slot limit for individual users, specified at the user level (JL/P in
lsb.users) -
GROUP_JLP--per-processor job slot limit for user groups, specified at the user level (JL/P in
lsb.users)
Job slot limits specified at the queue level are never affected by preemptive scheduling.
Configuring preemptable queue preference
For preemptive queues, you can specify which preemptable queues are considered first for preemption by configuring queue preference in the queue with the highest priority.
To indicate the order of preference for the preemptable queues, put a plus sign
(+) after the names of the preemptable queues and a preference level as a
positive integer. Higher numbers indicate higher preferences for preempting a
job in that queue. If no queue preference is specified, it is assumed to be 0. If
there are multiple queues, LSF preempts jobs in the queue with the highest
preference; queues at the same level of preference are ordered by queue
priority.
When preemtable queue preference is enabled, LSF considers jobs from preferred queues first instead of choosing running jobs based on best hosts:
- Queues with a preference number are preferred over queues without a preference number
- Queues with a higher preference number are preferred over queues with lower preference number
- For queues without a preference number, the queue with lower-priority is preferred than the queue with higher priority
Begin Queue QUEUE_NAME = high_priority PRIORITY = 50 PREEMPTION = PREEMPTIVE[low_q1+2 low_q2 low_q3] End Queue
Jobs from low_q1 are preferred first for preemption before jobs from low_q2
and low_q3.
If preemptable queue preference and preemption on jobs with least run time are both enabled, the queue preference for the job is considered first, then the job run time.
Preempting jobs with the least run time
By default, when more than one preemptable job exists (low-priority jobs holding the required resource), LSF preempts a job from the least-loaded host.
You can configure LSF to consider the jobs with least run time first instead of choosing jobs based on least-loaded host. Run time is wall-clock time, not normalized run time.
If preemptable queue preference and preemption on jobs with least run time are both enabled, the queue preference for the job is considered first, then the job run time.
Set PREEMPT_FOR in lsb.params, and use the LEAST_RUN_TIME keyword
to indicate that jobs with the least run time are to be preempted before other
jobs.
Preventing preemption by run time
You can configure LSF to prevent preemption of a job that would finish within a specified time or that has been running for a specified time. Run time is wall- clock time, not normalized run time.
You must define a run limit for the job, either at job level by bsub -W option
or in the queue by configuring RUNLIMIT in lsb.queues.
NO_PREEMPT_RUN_TIME, NO_PREEMPT_FINISH_TIME (lsb.params)
Set NO_PREEMPT_RUN_TIME in lsb.params, and the jobs have been
running for the specified number of minutes or longer will not be preempted.
Set NO_PREEMPT_FINISH_TIME in lsb.params, and jobs that will finish
within the specified number of minutes will not be preempted.
[ Top ]
| [ Platform Documentation ] | [ Title ] | [ Contents ] | [ Previous ] | [ Next ] | [ Index ] |
|
Date Modified: January 12, 2004
Platform Computing: www.platform.com Platform Support: support@platform.com Platform Information Development: doc@platform.com Copyright © 1994-2004 Platform Computing Corporation. All rights reserved. |