subway.plugins package

subway.plugins.debug module

debug for plain submitter and checker

class subway.plugins.debug.DebugChk(params=None, is_next=False, **kwargs)

Bases: subway.framework.PlainChk

__init__(params=None, is_next=False, **kwargs)

Parameters

params – list of dict

_abc_impl = <_abc_data object>

check_checking(jobid)

DIY: must. Check job whose state is checking. Jobs are sent to this method only when is_checked() returns True. Generate possible new jobs and inputs based on results from main task and possibly check task. The very core of checker class.

Parameters

jobid – str.

Returns

List[Tuple[str, Dict]]. [(jobid, resource dict), …]

class subway.plugins.debug.DebugSub(**kwargs)

Bases: subway.framework.PlainSub

_abc_impl = <_abc_data object>

submit_pending(inputpath)

DIY: must. The very core of submitter class. It describes how do you define submitting a job.

Parameters

jobid – str.

Returns

subway.plugins.dslurm module

Provide DSlurmSub and DSlurmChk class, with consideration on check scripts into slurm system

class subway.plugins.dslurm.DSlurmChk(params=None, _from='conf', **kwargs)

Bases: subway.plugins.sslurm.SSlurmChk

Slurm checker for DS scheme.

_abc_impl = <_abc_data object>

check_finished(jobid)

DIY: depends. Check job whose state is finished. Namely, the main task has been computed sucessfully, we need to check output of main task to further generate check job and associated check input. On the other hand, if there is no need for check job to be controlled by submitter (i.e. one simply check the main output in checker and generate new jobs), one can omit this method, since the default implementation is used for such case (no check job scenario).

Parameters

jobid – str.

Returns

List[Tuple[str, Dict]]. The length of the list must be 0 (no check job) or 1 (namely check job must be one to one correspondence with main job).

abstract check_finished_main(jobid)

Parameters

jobid – str.

Returns

List[Tuple[str, Dict[str, Any]]]. The length of the list must be 0 or 1. [(checkid, check_resource)]

class subway.plugins.dslurm.DSlurmSub(**kwargs)

Bases: subway.plugins.sslurm.SSlurmSub

Slurm submitter for DS scheme.

_abc_impl = <_abc_data object>

submit_aborted(jobid)

DIY: depends. If you have to deal with DS scheme as in Double vs. Single Submitter together with fault tolerance, then you must implement this method.

Parameters

jobid – str.

Returns

submit_finished(jobid)

DIY: depends. If you have to deal with DS scheme as in Double vs. Single Submitter, then you must implement this method.

Parameters

jobid – str.

Returns

subway.plugins.nohup module

puerly linux nohup interface for job submission

exception subway.plugins.nohup.NHException(message, code=10)

Bases: subway.exceptions.SubwayException

class subway.plugins.nohup.NHSChk(params=None, **kwargs)

Bases: subway.framework.PlainChk, subway.plugins.renderer.PlainRenderer

checker using non-blocking subprocess.Popen

_abc_impl = <_abc_data object>

check_checking(jobid)

DIY: must. Check job whose state is checking. Jobs are sent to this method only when is_checked() returns True. Generate possible new jobs and inputs based on results from main task and possibly check task. The very core of checker class.

Parameters

jobid – str.

Returns

List[Tuple[str, Dict]]. [(jobid, resource dict), …]

abstract check_checking_main(jobid)

DIY:MUST.

Parameters

jobid – str.

Returns

List[Union[Dict, List]]. list of param for new jobs

check_kickstart()

DIY: strongly recommended. Generate jobs with inputs at the beginning. The default impl does nothing. If the user doesn’t define this method in subclass, then the user must add inputs files and possible items in empty history.json by hand.

Returns

List[Tuple[str, Dict]]. [(jobid, resource dict)]

is_aborted(jobid)

Default to False. One can hack history in check_finished and manually change the state to aborted.

Parameters

jobid – str.

Returns

bool.

is_checked(jobid)

DIY: depends. Whether a checking task is finished.

Parameters

jobid – str.

Returns

bool.

is_failed(jobid)

DIY: depends. Whether a resolving task is failed.

Parameters

jobid – str.

Returns

bool.

is_finished(jobid)

Using Popen.poll() to detect whether job is finished. For restarted subway run which lose popen object in the memory, use ps pid instead.

Parameters

jobid – str.

Returns

is_frustrated(jobid)

DIY: depends. Whether a checking task is failed.

Parameters

jobid – str.

Returns

bool.

is_resolved(jobid)

DIY: depends. Whether a resolving task is finished.

Parameters

jobid – str.

Returns

bool.

class subway.plugins.nohup.NHSSub(**kwargs)

Bases: subway.framework.PlainSub

submitter using non-blocking subprocess.Popen

_abc_impl = <_abc_data object>

submit_pending(jobid)

DIY: not recommend. Already well written, submit with command from nohup_command in config.json, or from command argument when initializing the submitter.

Parameters

jobid – str.

Returns

None.

subway.plugins.renderer module

Mixin for checker for higher level render and workflow

class subway.plugins.renderer.PlainRenderer

Bases: object

_render_check(params, jobid=None, _type='main', prefix='')

DIY: not recommend. main entrance to rendered mixin, shall be called in checker class, check_* methods.

Parameters

• params – List[Union[Dict, List]].

• jobid – Optional[str]. Default None for new job render, and jobid is give for assoc job render.

• _type – Optional[str]. Default “main”. Current options also include “check”.

• prefix – Optional[str]. Default "".

Returns

List[Tuple[str, Dict[str, Any]]]. List of pairs with jobid and resource dict for new jobs or assoc job.

_render_input(jobid, checkid='', param=None, prefix='')

DIY: depends. If you try to customize this method, include super()._render_input is strongly recommended. Generate input files based on jobid and param. The default impl can render param to input.template. But this is not general enough for all user case, the user can simply rewrite this method.

Parameters

• jobid – str.

• param – Unoin[Dict, List, Tuple].

Returns

None.

_render_newid()

DIY: depends. generate new job id. In most cases, the default implementation by uuid4 is good enough.

Returns

str. jobid

_render_resource(jobid, checkid='', param=None, prefix='')

DIY: depends Default impl adds param and job_count=1 to resource dict. Even if you customize this method, we recommend you add super()._render_resource in your method.

Parameters

• jobid – str.

• checkid – Optional[str], default “”.

• param – Optional[Union[List, Dict]].

• prefix – Optiona[str], default “”.

Returns

Dict, resource dict.

subway.plugins.slurm module

basic slurm plugin with some improvement on bool and time relevant methods.

class subway.plugins.slurm.SlurmChk(params=None, **kwargs)

Bases: subway.framework.PlainChk

Low level checker for slurm with only some is_* and *_time methods redefined. General users are recommended to use subway.plugins.sslurm.SSlurmChk instead.

_abc_impl = <_abc_data object>

ending_time(jobid)

For DS scheme, return finish time of check slurm job. For SS scheme, simply return now.

Parameters

jobid – str.

Returns

float, timestamp.

finishing_time(jobid)

get finish time from sacct.

Parameters

jobid – str.

Returns

float, timestamp.

is_aborted(jobid)

Determine job states based on sacct of main job.

Parameters

jobid – str.

Returns

bool.

is_checked(jobid)

For DS scheme, judge based on slurm job state of check job. For SS scheme, always return True.

Parameters

jobid – str.

Returns

bool.

is_failed(jobid)

DIY: depends. Whether a resolving task is failed.

Parameters

jobid – str.

Returns

bool.

is_finished(jobid)

Determine job states based on sacct of main job.

Parameters

jobid – str.

Returns

bool.

is_frustrated(jobid)

For DS scheme, judge based on slurm job state of check job. For SS scheme, always return False.

Parameters

jobid – str.

Returns

bool.

is_resolved(jobid)

DIY: depends. Whether a resolving task is finished.

Parameters

jobid – str.

Returns

bool.

class subway.plugins.slurm.SlurmSub(**kwargs)

Bases: subway.framework.PlainSub

Exactly the same as subway.framework.PlainSub.

_abc_impl = <_abc_data object>

subway.plugins.sslurm module

More specific slurm plugin for checker and submitter, Extra S is for specific or single (indicates that check job not go through slurm)

class subway.plugins.sslurm.SSlurmChk(params=None, _from='conf', **kwargs)

Bases: subway.plugins.slurm.SlurmChk, subway.plugins.renderer.PlainRenderer

Slurm submitter with SS scheme and PlainRender mixin.

__init__(params=None, _from='conf', **kwargs)

Parameters

• params – List[Dict, List].

• _from – str. conf, template.

• kwargs –

_abc_impl = <_abc_data object>

_render_commands(jobid, checkid='', param=None, prefix='')

_render_input(jobid, checkid='', param=None, prefix='')

DIY: depends. Generate input files based on jobid and param. The default impl can render param to input.template and similar to sbatch.template. But this is not general enough for all user case, the user can simply rewrite this method.

Parameters

• jobid – str.

• param – Unoin[Dict, List, Tuple].

Returns

None.

_render_sbatch(jobid, checkid='', param=None, prefix='')

render sbatch script from template file or config.json. called from _render_input

Parameters

• jobid – str.

• checkid – Optiona[str].

• param – Union[List, Dict].

• prefix – Optional[str].

Returns

None.

_substitue_opts(opts, jobid, checkid='', param=None)

inner function to render slurm commands from templates.

Parameters

opts – List[str].

Returns

List[str].

check_checking(jobid)

Call check_checking_main to get params and then call _render_check.

Parameters

jobid – str.

Returns

abstract check_checking_main(jobid)

DIY: must. Given jobid of job in state of checking, return params list for new jobs to be run.

Parameters

jobid – str.

Returns

List[Tuple[str, Dict[str, Any]]].

check_kickstart()

Directly call _render_check(self.params).

Returns

class subway.plugins.sslurm.SSlurmSub(**kwargs)

Bases: subway.plugins.slurm.SlurmSub

Slurm checker with SS scheme.

_abc_impl = <_abc_data object>

submit_pending(jobid)

DIY: not recommend. Default impl is submitting jobs via sbatch, it is in general good enough.

Parameters

jobid – str.

Returns

None.