Plugins

Plain Renderer

Renderer is a type of mixin for checker class, which provide several _render_* methods. These utilities are designed for rendering of input files and resource dict.

The main entrance is _render_check. For checker class to make use of such mixin, the default implementation of check_checking and so on should follows the pattern below.

def check_checking(self, jobid):
    params = self.check_checking_main(jobid)
    return self._render_check(params)
# or for kickstart
def check_kickstart(self):
    return self._render_check(self.params)
# or for DS scheme
def check_finished(self, jobid):
    params = self.check_finished_main(jobid)
    return self._render_check(params=params, jobid=jobid, _type="check")

Insider _render_check, the renderer is responsible for: - generate new jobid from _render_newid (default random uuid), - generate input files given param from _render_input (default render template with vars insider {}), - generate resource dict from _render_resource (default add param and job_count=1 to resource).

Nohup

The nohup like plugins are subway.plugins.nohup.NHSSub and subway.plugins.nohup.NHSChk.

It is worth noting, the implementation doesn’t actually use nohup, instead it directly use subprocess.Popen in python builtin which is non-blocking by default.

Slurm

For slurm plugin, we provide two levels of abstractions.

The lower level ones are subway.plugins.slurm.SlurmSub and subway.plugins.slurm.SlurmChk.

The higher level ones are subway.plugins.sslurm.SSlurmSub and subway.plugins.sslurm.SSlurmChk. There are also DS scheme counterparts as subway.plugins.dslurm.DSlurmSub and subway.plugins.dslurm.DSlurmChk.

Lower-level utilities

For the lower level checker and submitter, they are very similar with plainsub and plainchk, with only several enhanced methods for timestamps generation and job finish bool.

Specifically, subway.plugins.slurm.SlurmChk only has slurm-specific methods for ending_time, finishing_time, is_finished, is_aborted.

For nomral user using slurm, we recommend the higher level abstraction classes which have better abstraction on job submitting and generations.

Architecture for sslurm

SSlurm classes are designed for SS scheme with slurm as job submission backend. SSlurm make more specific assumptions by mixin plain renderer, and leave less code for the user to implement.

The default behavior of sslurm will automatically render input and sbatch files from templates as well as tempates pathes are specified as (check_)slurm_template, (check_)input_template. All variables in the template files include param for corresponding job, conf of subway, jobid and checkid for assoc jobs in DS scheme.

To get attributes from these dict, we use ~ as separator in the template and brace all variables to be rendered within {}. For example, {conf~inputs_abs_path}. For list, we use key named as list_0 and so on. For example, {param~size~list_1}.

The only method, we must implement in sslurm checker, is check_checking_main. It takes jobid of checking state job, and return the params for possible new jobs. The is the core logic of how task trees grow, and such logic varies from projects which requires user customization.

More on dslurm

DSlurm classes are designed for DS scheme. One need also to implement at least check_finished_main beyond check_checking_main. The former method should read main output and generate param for associate check job. The latter should read check output and possibly combine with main output to generate new params for new jobs.