FastX 5: Scheduling Sessions

FastX 5 modularizes all parts of starting and scheduling sessions into a new scheduler directory. From data collection via forms, to node selection using load balancing, to the actual start command to exec itself, everything in this process can be customized. And while the session is waiting to start ie a “pending session,” commands, can be executed allowing a user to view the status, check logs, or even cancel a the session even before it launched. Multiple schedulers can exist on a cluster allowing different apps to launch in any way the admin needs.

What is a Job Scheduler?

A job scheduler is a program that manages the process of starting tasks (jobs) on a given node in a cluster. In FastX, the task is typically starting the FastX session to do interactive work on a node.

The FastX job scheduler is a directory with a series of scripts used for data collection, node selection, execution of code to launch a session, and tools that are used to manage data during the session starting process.

Pending Sessions

There exists a delay between the time a user clicks the start button and when a FastX Session process actually starts. Under normal circumstances this may only be a few seconds. However when integrating commercial job schedulers (ie SLURM), this delay can be significant.

A pending session is a FastX session that has been scheduled but has yet to register itself with the cluster. Pending sessions will appear to the end user as a regular session, but with a limited set of actions.

The Schedule Directory

Job schedulers reside in the following locations

/usr/lib/fastx/5/integration/schedule # system default schedulers
/etc/fastx/schedule # custom schedulers and overrides

FastX ships with several default schedulers that cover a large majority of how admins want to connect.

The FastX Job Scheduler Directory Structure

A job scheduler is a directory that lives in one of the two directories stated above.

In this directory are a series of scripts that define the actions available to the scheduler

metadata.ini — stores any static configuration data that the scheduler scripts will need
preprocess — handles form processing, node filtering and selection, and data validation and transformation. Due to the complexity of this script, people rarely edit this file directly. Instead, tasks are broken down into sub scripts that an admin can override
- form.html — default form to use for custom data
- transform — modify the start data before passing it on
- node-filter — filter out nodes that do not match the scheduling criteria
- node-select — select the actual node to launch a session on
start — execute a series of commands to eventually start a FastX Session. /api/session/start
terminate — cancel a pending session /api/session/terminate
log — view log files on a pending session /api/session/log
status — periodically execute this command to make sure a pending session is still alive

Additional scripts can be added, but they are rarely useful

Overriding Scripts

Customizing scripts is at the heart of job scheduling. Every aspect of the start process can be overridden allowing fine grained integration into your cluster. Scripts are well defined and well documented. Their job is to read from stdin and generate a result printed to stdout. The output conforms to the result of the api call that it is overriding.

The start script is the most important script for job scheduling. This script tells the server how to actually start a session. Giving admins access to the start process enables them to inject special instructions allowing for unmatched customization.

The primary example for job scheduling is to integrate commercial Job Schedulers (most notably SLURM) into the start process of FastX. This means that SLURM can start the session through the FastX user interface with one click. Admins can create forms for data collection of the major SLURM sbatch options to pass to the start command.

Overriding the terminate script gives the user the ability to cancel the SLURM job through the user interface.

The Start Process

The start process is documented in the following diagram. A summary is listed below

The Client initiates a start api call to the Webserver
The Webserver forwards api calls to a Launcher that can process the scheduler the app is assigned
The Launcher processes and verifies all the app forms
- Run scripts: preprocess, transform, node-filter, node-select
The Launcher selects a node (StartLauncher) to actually launch the session
StartLauncher executes the start script that has the command to launch the session
- While waiting for the session to start user can execute commands status, log, terminate on StartLauncher to manage the pending session
The actual session is launched on ComputeNode
- Typically StartLauncher and ComputeNode are the same system
The session on ComputeNode connects back to StartLauncher and registers itself
At this point the job scheduling is done and no more scripts are used

Custom Forms

Custom forms are used to gather user defined data which can be sent to the job scheduler for better node selection. Forms are HTML based witht he ability to run javascript code if enabled.

Forms are sent to the client. The user fills them out and passes them back as part of the start data

The server can then process the form in a script to pass to the start script.
There are three methods for generating forms form.html, form_script metadata[‘forms’]

The preprocessor will choose which form to submit in the following order

metadata[‘forms’] — if forms are set in the metadata, use them. ignore the other methods form_script, and form.html
form_script — if form script exists, execute it. ignore form.html
form.html — dump the file as html.
no forms.

form.html

The simplest method is to add a file named form.html into your job scheduler directory. The preprocessor will send that static form to the user.

form_script

Forms can be dynamically generated. Placing a file named form_script into the scheduler directory will enable an admin to generate an HTML form based on the user data submitted.

metadata[‘forms’]

The most advanced method is for a user to add a list of forms to use in the metadata[‘forms’] section. The admin defines the files used in order


# forms to show to the client
# if the form is a string, interpret it as an html file
forms[]=form1.html
# if the form is a json object, file is required.  default type is html
forms[]={ "file": "form2.html" }
forms[]={ "file": "form3", "type": "html" }
# you can execute a custom script to generate html by setting type to script
forms[]={ "file": "form4",  "type": "script" }

Form Processing

When a client fills out a form, the data is added to a formData object with the parameter being the form file that was jsut processed.

start["formData"] = {
    "form1.html": { form_1_answers },
    "form2_script": { form_2_answers }

}

Once all form files have been processed, the scheduler can continue
Admins can short circuit form processing, by setting

formData[‘form_file’] = {}

Transform Script

Once all the forms have been submitted, the admin can create a script transform which can be used to validate the form data and transform it if needed. The result of the transform script is the data that will be sent to the start script.

Node Selection

Node selection is the process of selecting which server to run the start command on. It consists of the following steps

Filter out nodes that don’t meet the criteria
Selecting a node based on the nodes that fit the criteria

node-filter

The purpose of this file is to filter out the nodes that do not meet the criteria and to return those that do.

node-select

From the list of filtered nodes, pick one. That nodeID will be used to start the session