8.5.3. Flow-of-Control
Describe how allocation requests are flowed to the scheduler, scheduler attachment upon startup of either side, where response to allocation requests are returned and how they get back to the requestor, where session controls are received for allocation instantiation and how that is done, where session control is called to indicate all jobs complete in session.
Probably best to start with an overview of how things flow and then drill down into the respective steps.
Allocation requests have to be serviced by the scheduler. Since
a requestor could attach directly to the scheduler (e.g., in the
case of a tool submitting a session request) or to a daemon within
the RM (who must then relay it to its system controller for forwarding
to the scheduler), the scheduler must inform the RM of the allocation
via the PMIx_Session_control API - i.e., the RM cannot guarantee
its ability to intercept and process an allocation response to learn
of a session that needs to be instantiated.
A scheduler attaches to the DVM master as a tool that declares itself the
scheduler (PMIX_SERVER_SCHEDULER), naming its own namespace and rank.
The master accepts that only from a tool running as root, as the user the
DVM runs as, or as a user named by the prte_pmix_scheduler_uids MCA
parameter - a comma-separated list of user names or numeric ids, for a
scheduler that runs as a service user of its own. It forgets the scheduler
when it disconnects, after which the DVM answers session and allocation
requests itself again.
PRRTE’s role in the PMIx_Allocation_request flow is generally to
pass the request on to the scheduler, and transport the reply back
to the requestor.
When PRRTE passes a request on - to the scheduler, or to another DVM’s
data server - the scheduler’s PMIx tells it that PRRTE is asking, and
PRRTE says whom it asks for in PMIX_ORIGIN_USERID,
PMIX_ORIGIN_GRPID and the other PMIX_ORIGIN_* attributes, beside
PMIX_REQUESTOR. Whether to believe that is the receiving host’s
decision. PRRTE, receiving a relayed request itself, believes it from a
relay running as the same user it names, as root, or as the user the DVM
runs as. Any other relay’s request is taken as the relay’s own.
Every request for new or extended resources (PMIX_ALLOC_NEW,
PMIX_ALLOC_EXTEND) also says whom the allocation is for, as
PMIX_ALLOC_UID and PMIX_ALLOC_GID. A requester may name its own
user - root may name any user - and any group it belongs to; naming
anything else is refused with PMIX_ERR_NO_PERMISSIONS before the
request goes anywhere. Where the requester names neither, PRRTE adds its
own user and group. The scheduler is to grant the allocation for exactly
that user and group and return both in its answer; an answer that names
another, or none, is an error, and PRRTE hands what was granted back.
The allocation then belongs to that user and group - they are who may
later extend or release it.
PMIX_ALLOC_ACTIVATE is the exception: PRRTE answers it itself,
without involving any resource manager. It asks for a daemon on nodes
the requestor already holds - an allocated node the DVM is not
spanning, because a --host/--hostfile given at startup narrowed
which nodes got one, or because a released reservation handed it back
without one. Nothing is added to the allocation and no slot count
changes, so there is nothing to ask a scheduler for and nothing a
scheduler could refuse; for the same reason the request is served even
where the scheduler owns the allocation, which is where a request for
new resources cannot be. The nodes are named with PMIX_HOST and/or
PMIX_HOSTFILE, in the same syntax the --activate command line
option takes, and the request is answered when it is granted - the
daemons themselves are reported by the PMIX_DVM_IS_READY event that
completes any DVM size change.
Under Slurm, a client’s PMIX_ALLOC_EXTEND is answered once queued,
with PMIX_OPERATION_IN_PROGRESS and the PMIX_ALLOC_ID.
PMIX_DVM_IS_READY or PMIX_ERR_DVM_MOD reports the grant. Until
then PMIX_ALLOC_REQ_CANCEL cancels it; after it, the cancel is
answered PMIX_ERR_EXISTS with the PMIX_ALLOC_ID, and the
allocation must be released. That PMIX_DVM_IS_READY, and
PMIX_QUERY_ALLOC_PROPERTIES, carry PMIX_ALLOC_GRANT_TIME: when
the scheduler granted it.
An allocation request can also arrive on a spawn, as
PMIX_SPAWN_ALLOC: the value is an array of pmix_info_t holding the
request’s directive (PMIX_ALLOC_REQ_DIRECTIVE) and the info a standalone
request would have carried. PRRTE serves that request first, exactly as it
would serve the standalone one, and only launches the job once it is granted
- pointing the job at what it was given, so it maps onto those resources
rather than onto everything else. The two failures are kept apart: an
allocation that is refused fails the spawn with PMIX_ERR_JOB_ALLOC_FAILED
and launches nothing, while a spawn that fails after the grant hands the
allocation back before returning its own error. This saves a caller the
“request, wait, read the id, spawn into it” sequence, and closes the window
in that sequence where resources are held for a job that does not exist.
PRRTE only creates a session object as a result of a call from the
scheduler via PMIx_Session_control. What it does with each directive
of that API - and where it deviates from the letter of the attribute
description, as it must for preemption - is documented in
Session Control.