| [ Platform Documentation ] | [ Title ] | [ Contents ] | [ Previous ] | [ Next ] | [ Index ] |
This document describes the use of external job submission and execution
controls called esub and eexec. These site-specific user-written executables
are used to validate, modify, and reject job submissions, pass data to and
modify job execution environments.
[ Top ]
Understanding External Executables
About esub and eexec
LSF provides the ability to validate, modify, or reject job submissions, modify
execution environments, and pass data from the submission host directly to the
execution host through the use of the esub and eexec executables. Both are
site-specific and user written and must be located in LSF_SERVERDIR.
To validate, modify, or reject a job, an esub needs to be written. See Using
esub
To modify the execution environment on the execution host, an eexec needs
to be written. See Working with eexec
To pass data directly to the execution host, an esub and eexec need to be
written. See Using esub and eexec to pass data to execution environments
Interactive remote execution
Interactive remote execution also runs esub and eexec if they are found in
LSF_SERVERDIR. For example, lsrun invokes esub, and RES runs eexec
before starting the task. esub is invoked at the time of the ls_connect(3) call,
and RES invokes eexec each time a remote task is executed. RES runs eexec
only at task startup time.
DCE credentials and AFS tokens
esub and eexec are also used for processing DCE credentials and AFS tokens.
See the following documents on the Platform Web site for more information:
[ Top ]
Using esub
About esub
An esub, short for external submission, is a user-written executable (binary
or script) that can be used to validate, modify, or reject jobs. The esub is put
into LSF_SERVERDIR (defined in lsf.conf) where LSF checks for its existence
when a job is submitted, restarted, and modified. If LSF finds an esub, it is run
by LSF. Whether the job is submitted, modified, or rejected depends on the
logic built into the esub.
Any messages that need to be provided to the user should be directed to the
standard error (stderr) stream and not the standard output (stdout) stream.
- Environment variables to bridge esub and LSF
- General esub logic
- Rejecting jobs
- Validating job submission parameters
- Modifying job submission parameters
- The bmod and brestart commands and esub
- How LSF supports multiple esub
- How master esub invokes application-specific esubs
- Configuring master esub and your application-specific esub
Environment variables to bridge esub and LSF
LSF provides the following environment variables in the esub execution
environment:
LSB_SUB_PARM_FILE
This variable points to a temporary file containing the job parameters that esub
reads when the job is submitted. The submission parameters are a set of name-
value pairs on separate lines in the format "option_name=value". The
following option names are supported:
Example submission parameter file
If a user submits the following job:
% bsub -q normal -x -P my_project -R "r1m rusage[dummy=1]" -n 90 sleep 10
The contents of the LSB_SUB_PARM_FILE will be:
LSB_SUB_QUEUE="normal" LSB_SUB_EXCLUSIVE=Y LSB_SUB_RES_REQ="r1m rusage[dummy=1]" LSB_SUB_PROJECT_NAME="my_project" LSB_SUB_COMMAND_LINE="sleep 10" LSB_SUB_NUM_PROCESSORS=90 LSB_SUB_MAX_NUM_PROCESSORS=90
LSB_SUB_ABORT_VALUE
This variable indicates the value esub should exit with if LSF is to reject the
job submission.
LSB_SUB_MODIFY_ENVFILE
The file in which esub should write any changes to the job environment
variables.
esub writes the variables to be modified to this file in the same format used in
LSB_SUB_PARM_FILE. The order of the variables does not matter.
After esub runs, LSF checks LSB_SUB_MODIFY_ENVFILE for changes and if
found, LSF will apply them to the job environment variables.
LSB_SUB_MODIFY_FILE
The file in which esub should write any submission parameter changes.
esub writes the job options to be modified to this file in the same format used
in LSB_SUB_PARM_FILE. The order of the options does not matter. After esub
runs, LSF checks LSB_SUB_MODIFY_FILE for changes and if found LSF will
apply them to the job.
LSB_SUB_ADDITIONAL cannot be changed in or added to LSB_SUB_MODIFY_FILE.
General esub logic
After esub runs, LSF checks:
-
Is the
esubexit value LSB_SUB_ABORT_VALUE? - Reject the job
- Go to step 5
- Does LSB_SUB_MODIFY_FILE or LSB_SUB_MODIFY_ENVFILE exist?
- Done
Rejecting jobs
Depending on your policies you may choose to reject a job. To do so, have
esub exit with LSB_SUB_ABORT_VALUE.
If esub rejects the job, it should not write to either LSB_SUB_MODIFY_FILE or
LSB_SUB_MODIFY_ENVFILE.
The following Bourne shell esub rejects all job submissions by exiting with
LSB_SUB_ABORT_VALUE:
#!/bin/sh # Redirect stderr to stdout so echo can be used for # error messages exec 1>&2 # Reject the submission echo "LSF is Rejecting your job submission..." exit $LSB_SUB_ABORT_VALUE
Validating job submission parameters
One use of validation is to support project-based accounting. The user can
request that the resources used by a job be charged to a particular project.
Projects are associated with a job at job submission time, so LSF will accept any
arbitrary string for a project name. In order to ensure that only valid projects
are entered and the user is eligible to charge to that project, an esub can be
written.
The following Bourne shell esub validates job submission parameters:
#!/bin/sh . $LSB_SUB_PARM_FILE # Redirect stderr to stdout so echo can be used for error messages exec 1>&2 # Check valid projects if [ $LSB_SUB_PROJECT_NAME != "proj1" -o $LSB_SUB_PROJECT_NAME != "proj2" ]; then echo "Incorrect project name specified" exit $LSB_SUB_ABORT_VALUE fi USER=`whoami` if [ $LSB_SUB_PROJECT_NAME = "proj1" ]; then # Only user1 and user2 can charge to proj1 if [$USER != "user1" -a $USER != "user2" ]; then echo "You are not allowed to charge to this project" exit $LSB_SUB_ABORT_VALUE fi fi
Modifying job submission parameters
esub can be used to modify submission parameters and the job environment
before the job is actually submitted.
The following example writes modifications to LSB_SUB_MODIFY_FILE for the following parameters:
In the example, user userA can only submit jobs to queue queueA. User userB
must use Bourne shell (/bin/sh), and user userC should never be able to
submit a job.
#!/bin/sh . $LSB_SUB_PARM_FILE # Redirect stderr to stdout so echo can be used for error messages exec 1>&2 USER=`whoami` # Ensure userA is using the right queue queueA if [ $USER="userA" -a $LSB_SUB_QUEUE != "queueA" ]; then echo "userA has submitted a job to an incorrect queue" echo "...submitting to queueA" echo 'LSB_SUB_QUEUE="queueA"' > $LSB_SUB_MODIFY_FILE fi # Ensure userB is using the right shell (/bin/sh) if [ $USER="userB" -a $SHELL != "/bin/sh" ]; then echo "userB has submitted a job using $SHELL" echo "...using /bin/sh instead" echo 'SHELL="/bin/sh"' > $LSB_SUB_MODIFY_ENVFILE fi # Deny userC the ability to submit a job if [ $USER="userC" ]; then echo "You are not permitted to submit a job." exit $LSB_SUB_ABORT_VALUE fi
The bmod and brestart commands and esub
You can use the bmod command to modify job submission parameters, and
brestart to restart checkpointed jobs. Like bsub, bmod and brestart also
call esub if it exists. bmod and brestart cannot make changes to the job
environment through esub. Environment changes only occur when esub is
called by the original job submission with bsub.
How LSF supports multiple esub
LSF provides a master esub (LSF_SERVERDIR/mesub) to handle the invocation
of individual esub executables and the job submission requirements of your
applications. Use the -a option of bsub to specify the application you are
running through LSF.
For example, to submit a FLUENT job:
bsub -a fluent bsub_options fluent_command
The method name fluent, uses the esub for FLUENT jobs
(LSF_SERVERDIR/esub.fluent), which sets the checkpointing method
LSB_ECHKPNT_METHOD="fluent" to use the echkpnt.fluent and
erestart.fluent.
LSB_ESUB_METHOD (lsf.conf)
To specify a mandatory esub method that applies to all job submissions, you
can configure LSB_ESUB_METHOD in lsf.conf.
LSB_ESUB_METHOD specifies the name of the esub method used in addition
to any methods specified in the bsub -a option.
For example, LSB_ESUB_METHOD="dce fluent" defines DCE as the
mandatory security system, and FLUENT as the mandatory application used on
all jobs.
How master esub invokes application-specific esubs
bsub invokes mesub at job submission, which calls:
-
Mandatory
esubprograms defined by LSB_ESUB_METHOD -
esubif it exists -
application-specific
esubprograms if thebsub -aoption is specified
In this example, esub.dce is defined as a mandatory esub, an esub already
exists in LSF_SERVERDIR, and the job is submitted as a FLUENT job to use
esub.fluent.
Configuring master esub and your application-specific esub
The master esub is installed as LSF_SERVERDIR/mesub. After installation:
-
Create your own application-specific
esub. -
Optional. Configure LSB_ESUB_METHOD in
lsf.confto specify a mandatoryesubfor all job submissions.
Use the following naming conventions:
-
On UNIX,
LSF_SERVERDIR/esub.applicationFor example,
esub.fluentfor FLUENT jobs -
On Windows,
LSF_SERVERDIR\esub.application.[exe |bat]For example,
esub.fluent.exe
Your existing esub does not need to follow this convention and does not need to be renamed. However, since mesub invokes any esub that follows this convention, you should move any backup copies of your esubs out of LSF_SERVERDIR or choose a name that does not follow the convention (for example, use esub_bak instead of esub.bak).
The name esub.user is reserved for backward compatibility. Do not use the name esub.user for your application-specific esub.
[ Top ]
Working with eexec
About eexec
The eexec program runs on the execution host at job start-up and completion
time and when checkpointing is initiated. It is run as the user after the job
environment variables have been set. The environment variable LS_EXEC_T is
set to START, END, and CHKPNT, respectively, to indicate when eexec is
invoked.
If you need to run eexec as a different user, such as root, you must properly
define LSF_EEXEC_USER in the file /etc/lsf.sudoers. See the Platform
LSF Reference for information about the lsf.sudoers file.
eexec is expected to finish running because the parent job process waits for
eexec to finish running before proceeding. The environment variable
LS_JOBPID stores the process ID of the process that invoked eexec. If eexec
is intended to monitor the execution of the job, eexec must fork a child and
then have the parent eexec process exit. The eexec child should periodically
test that the job process is still alive using the LS_JOBPID variable.
Using esub and eexec to pass data to execution environments
If esub needs to pass some data to eexec, it can write the data to its standard
output for eexec to read from its standard input (stdin). LSF effectively acts
as the pipe between esub and eexec (e.g., esub | eexec).
Standard output (stdout) from any esub is automatically sent to eexec.
Since eexec cannot handle more than one standard output stream, only one
esub can use standard output to generate data as standard input to eexec.
For example, the esub for AFS (esub.afs) sends its authentication tokens as
standard output to eexec. If you use AFS, no other esub can use standard
output.
[ Top ]
| [ Platform Documentation ] | [ Title ] | [ Contents ] | [ Previous ] | [ Next ] | [ Index ] |
|
Date Modified: January 12, 2004
Platform Computing: www.platform.com Platform Support: support@platform.com Platform Information Development: doc@platform.com Copyright © 1994-2004 Platform Computing Corporation. All rights reserved. |