subway.components package

subway.components.genfiles module

smarter file rendered can be utilized in _render_input of SSlurmChk

subway.components.genfiles.generate_file(data, output_path, output_format=None, output_config=None, output_template=None)[source]
subway.components.genfiles.jsonify(data, _outer_most=False, _outer=True, func=None)[source]

subway.components.slurmoo module

Lower level abstraction of slurm from CLI interaction.

exception subway.components.slurmoo.SlurmException(message, code=90)[source]

Bases: subway.exceptions.SubwayException

__init__(message, code=90)[source]
Parameters

  • message – str.

  • code – int. 10 general, 11 jobid unmatch, 12 only valid for general without id 13 no such atrribute in history of conf

class subway.components.slurmoo.SlurmJob(jobname=None, jobid=None, sacct='sacct')[source]

Bases: object

Abstraction for job submitted to slurm.

__init__(jobname=None, jobid=None, sacct='sacct')[source]
Parameters

  • jobname – Optional[str].

  • jobid – Optional[str]. One must specify at least one between jobid and jobname.

  • sacct – Optional[str]. Binary for sacct in shell, default is “sacct”.

_get_jobid(jobname)[source]
_get_jobinfo(jobid)[source]

get job relavant info from sacct by jobid

Parameters

jobid – str.

Returns

jobinfo: Dict[str, str], {'User': 'linuxuser', 'JobID': '4500', 'JobName': 'uuid', 'Partition': 'general', 'State': 'COMPLETED', 'Timelimit': '365-00:00+', 'Start': '2020-02-23T10:05:55', 'End': '2020-02-23T10:06:15', 'Elapsed': '00:00:20', 'NNodes': '1', 'NCPUS': '2'}

get_jobid(jobname, tries=10, interval=0.8)[source]

get jobid from jobname via sacct query

Parameters

  • jobname – str.

  • tries – Optional[int]. Default 10. It is worthing noting that query soon after job submitted would meet empty line, so repetitive try is necessary.

  • interval – Optional[float]. Default 0.8. Seconds between two tries.

Returns

str. jobid.

Raises

SlurmException – when failed to get jobid after tries tries.

get_jobinfo(jobid, tries=10, interval=0.8)[source]

get job info from jobid via sacct query

Parameters

  • jobid – str, jobid in slurm system (not jobid in subway which is jobname in slurm!)

  • tries – Optional[int]. Default 10. It is worthing noting that query soon after job submitted would meet ilegal line, so repetitive try is necessary.

  • interval – Optional[float]. Default 0.8. Seconds between two tries.

Returns

Dict. Slurm job info.

class subway.components.slurmoo.SlurmTask(sbatch='sbatch', scancel='scancel', shebang='#!/bin/bash', sbatch_path=None, sbatch_options=None, sbatch_commands=None)[source]

Bases: object

Abstraction for slurm job to be submitted.

__init__(sbatch='sbatch', scancel='scancel', shebang='#!/bin/bash', sbatch_path=None, sbatch_options=None, sbatch_commands=None)[source]
Parameters

  • sbatch – string, binary for sbatch

  • sbatch – string, binary for scancel

  • shebang – string, the #! line

  • sbatch_path – string, sbatch script path

  • sbatch_options – list of strings, such as “–job-name=uuid”

  • sbatch_commands – list of strings, main command, such as “python test.py”

_render_sbatch()[source]

generate sbatch script for the task.

Returns

None.

cancel()[source]

cancel the job

Returns

subprocess.CompletedProcess, from scancel.

delete(include_output=False)[source]

Delete the sbatch script and slurm output (experimental)

Parameters

include_output – Optional[bool]. Default False, whether try to delete slurm output

Returns

None.

jobid()[source]

Get jobid from stdout of sbatch.

Returns

str. jobid

Raises

SlurmException – No legal stdout from sbatch command to capture the jobid.

slurm_outpath()[source]
stdouterr()[source]

get stdout and stderr for sbatch command

Returns

Tuple[str, str], string for stdout and stderr.

submit()[source]

submit jobs to slurm by sbatch in shell.

Returns

None.

exception subway.components.slurmoo.SlurmValueError(message, code=91)[source]

Bases: subway.components.slurmoo.SlurmException

__init__(message, code=91)[source]
Parameters

  • message – str.

  • code – int. 10 general, 11 jobid unmatch, 12 only valid for general without id 13 no such atrribute in history of conf

subway.components.slurmoo._slurm_time_trans(timestr, _format='%Y-%m-%dT%H:%M:%S')[source]

Fault tolerant time string translate to time obj and time stamps

Parameters

  • timestr – str.

  • _format – Optional[str]. Default as “%Y-%m-%dT%H:%M:%S” used in sacct output

Returns

Tuple[Datetime, float]. Datetime obj and timestamp.