SmartSim API

SmartSim API#

Experiment#

`Experiment.__init__`(name[, exp_path, launcher])	Initialize an Experiment instance
`Experiment.start`(*args[, block, summary, ...])	Start passed instances using Experiment launcher
`Experiment.stop`(*args)	Stop specific instances launched by this `Experiment`
`Experiment.create_ensemble`(name[, params, ...])	Create an `Ensemble` of `Model` instances
`Experiment.create_model`(name, run_settings)	Create a general purpose `Model`
`Experiment.create_database`([port, db_nodes, ...])	Initialize an Orchestrator database
`Experiment.create_run_settings`(exe[, ...])	Create a `RunSettings` instance.
`Experiment.create_batch_settings`([nodes, ...])	Create a `BatchSettings` instance
`Experiment.generate`(*args[, tag, overwrite, ...])	Generate the file structure for an `Experiment`
`Experiment.poll`([interval, verbose, ...])	Monitor jobs through logging to stdout.
`Experiment.finished`(entity)	Query if a job has completed.
`Experiment.get_status`(*args)	Query the status of launched instances
`Experiment.reconnect_orchestrator`(checkpoint)	Reconnect to a running `Orchestrator`
`Experiment.summary`([style])	Return a summary of the `Experiment`

class Experiment(name: str, exp_path: str | None = None, launcher: str = 'local')[source]#

Bases: object

Experiments are the Python user interface for SmartSim.

Experiment is a factory class that creates stages of a workflow and manages their execution.

The instances created by an Experiment represent executable code that is either user-specified, like the Model instance created by Experiment.create_model, or pre-configured, like the Orchestrator instance created by Experiment.create_database.

Experiment methods that accept a variable list of arguments, such as Experiment.start or Experiment.stop, accept any number of the instances created by the Experiment.

In general, the Experiment class is designed to be initialized once and utilized throughout runtime.

Initialize an Experiment instance

With the default settings, the Experiment will use the local launcher, which will start all Experiment created instances on the localhost.

Example of initializing an Experiment with the local launcher

exp = Experiment(name="my_exp", launcher="local")

SmartSim supports multiple launchers which also can be specified based on the type of system you are running on.

exp = Experiment(name="my_exp", launcher="slurm")

If you wish your driver script and Experiment to be run across multiple system with different schedulers (workload managers) you can also use the auto argument to have the Experiment guess which launcher to use based on system installed binaries and libraries

exp = Experiment(name="my_exp", launcher="auto")

The Experiment path will default to the current working directory and if the Experiment.generate method is called, a directory with the Experiment name will be created to house the output from the Experiment.

Parameters:

name (str) – name for the Experiment
exp_path (str, optional) – path to location of Experiment directory if generated
launcher (str, optional) – type of launcher being used, options are “slurm”, “pbs”, “lsf”, or “local”. If set to “auto”, an attempt will be made to find an available launcher on the system. Defaults to “local”

append_to_db_identifier_list(db_identifier: str) → None[source]#: Check if db_identifier already exists when calling create_database

create_batch_settings(nodes: int = 1, time: str = '', queue: str = '', account: str = '', batch_args: Dict[str, str] | None = None, **kwargs: Any) → BatchSettings[source]#

Create a BatchSettings instance

Batch settings parameterize batch workloads. The result of this function can be passed to the Ensemble initialization.

the batch_args parameter can be used to pass in a dictionary of additional batch command arguments that aren’t supported through the smartsim interface

# i.e. for Slurm
batch_args = {
    "distribution": "block"
    "exclusive": None
}
bs = exp.create_batch_settings(nodes=3,
                               time="10:00:00",
                               batch_args=batch_args)
bs.set_account("default")

Parameters:

nodes (int, optional) – number of nodes for batch job, defaults to 1
time (str, optional) – length of batch job, defaults to “”
queue (str, optional) – queue or partition (if slurm), defaults to “”
account (str, optional) – user account name for batch system, defaults to “”
batch_args (dict[str, str], optional) – additional batch arguments, defaults to None

Returns:

a newly created BatchSettings instance

Return type:

BatchSettings

Raises:

SmartSimError – if batch creation fails

create_database(port: int = 6379, db_nodes: int = 1, batch: bool = False, hosts: str | List[str] | None = None, run_command: str = 'auto', interface: str = 'ipogif0', account: str | None = None, time: str | None = None, queue: str | None = None, single_cmd: bool = True, db_identifier: str = 'orchestrator', **kwargs: Any) → Orchestrator[source]#

Initialize an Orchestrator database

The Orchestrator database is a key-value store based on Redis that can be launched together with other Experiment created instances for online data storage.

When launched, Orchestrator can be used to communicate data between Fortran, Python, C, and C++ applications.

Machine Learning models in Pytorch, Tensorflow, and ONNX (i.e. scikit-learn) can also be stored within the Orchestrator database where they can be called remotely and executed on CPU or GPU where the database is hosted.

To enable a SmartSim Model to communicate with the database the workload must utilize the SmartRedis clients. For more information on the database, and SmartRedis clients see the documentation at www.craylabs.org

Parameters:

port (int, optional) – TCP/IP port, defaults to 6379
db_nodes (int, optional) – number of database shards, defaults to 1
batch (bool, optional) – run as a batch workload, defaults to False
hosts (list[str], optional) – specify hosts to launch on, defaults to None
run_command (str, optional) – specify launch binary or detect automatically, defaults to “auto”
interface (str, optional) – Network interface, defaults to “ipogif0”
account (str, optional) – account to run batch on, defaults to None
time (str, optional) – walltime for batch ‘HH:MM:SS’ format, defaults to None
queue (str, optional) – queue to run the batch on, defaults to None
single_cmd (bool, optional) – run all shards with one (MPMD) command, defaults to True
db_identifier (str, optional) – an identifier to distinguish this orchestrator in multiple-database experiments, defaults to “orchestrator”

Raises:

SmartSimError – if detection of launcher or of run command fails
SmartSimError – if user indicated an incompatible run command for the launcher

Returns:

Orchestrator

Return type:

Orchestrator or derived class

create_ensemble(name: str, params: Dict[str, Any] | None = None, batch_settings: BatchSettings | None = None, run_settings: RunSettings | None = None, replicas: int | None = None, perm_strategy: str = 'all_perm', **kwargs: Any) → Ensemble[source]#

Create an Ensemble of Model instances

Ensembles can be launched sequentially or as a batch if using a non-local launcher. e.g. slurm

Ensembles require one of the following combinations of arguments

run_settings and params

run_settings and replicas

batch_settings

batch_settings, run_settings, and params

batch_settings, run_settings, and replicas

If given solely batch settings, an empty ensemble will be created that models can be added to manually through Ensemble.add_model(). The entire ensemble will launch as one batch.

Provided batch and run settings, either params or replicas must be passed and the entire ensemble will launch as a single batch.

Provided solely run settings, either params or replicas must be passed and the ensemble members will each launch sequentially.

The kwargs argument can be used to pass custom input parameters to the permutation strategy.

Parameters:

name (str) – name of the ensemble
params (dict[str, Any]) – parameters to expand into Model members
batch_settings (BatchSettings) – describes settings for Ensemble as batch workload
run_settings (RunSettings) – describes how each Model should be executed
replicas (int) – number of replicas to create
perm_strategy (str, optional) – strategy for expanding params into Model instances from params argument options are “all_perm”, “step”, “random” or a callable function. Default is “all_perm”.

Raises:

SmartSimError – if initialization fails

Returns:

Ensemble instance

Return type:

Ensemble

create_model(name: str, run_settings: RunSettings, params: Dict[str, Any] | None = None, path: str | None = None, enable_key_prefixing: bool = False, batch_settings: BatchSettings | None = None) → Model[source]#

Create a general purpose Model

The Model class is the most general encapsulation of executable code in SmartSim. Model instances are named references to pieces of a workflow that can be parameterized, and executed.

Model instances can be launched sequentially, as a batch job, or as a group by adding them into an Ensemble.

All models require a reference to run settings to specify which executable to launch as well provide options for how to launch the executable with the underlying WLM. Furthermore, batch a reference to a batch settings can be added to launch the model as a batch job through Experiment.start. If a model with a reference to a set of batch settings is added to a larger entity with its own set of batch settings (for e.g. an Ensemble) the batch settings of the larger entity will take precedence and the batch setting of the model will be strategically ignored.

Parameters supplied in the params argument can be written into configuration files supplied at runtime to the model through Model.attach_generator_files. params can also be turned into executable arguments by calling Model.params_to_args

By default, Model instances will be executed in the current working directory if no path argument is supplied. If a Model instance is passed to Experiment.generate, a directory within the Experiment directory will be created to house the input and output files from the model.

Example initialization of a Model instance

from smartsim import Experiment
run_settings = exp.create_run_settings("python", "run_pytorch_model.py")
model = exp.create_model("pytorch_model", run_settings)

# adding parameters to a model
run_settings = exp.create_run_settings("python", "run_pytorch_model.py")
train_params = {
    "batch": 32,
    "epoch": 10,
    "lr": 0.001
}
model = exp.create_model("pytorch_model", run_settings, params=train_params)
model.attach_generator_files(to_configure="./train.cfg")
exp.generate(model)

New in 0.4.0, Model instances can be colocated with an Orchestrator database shard through Model.colocate_db. This will launch a single Orchestrator instance on each compute host used by the (possibly distributed) application. This is useful for performant online inference or processing at runtime.

New in 0.4.2, Model instances can now be colocated with an Orchestrator database over either TCP or UDS using the Model.colocate_db_tcp or Model.colocate_db_uds method respectively. The original Model.colocate_db method is now deprecated, but remains as an alias for Model.colocate_db_tcp for backward compatibility.

Parameters:

name (str) – name of the model
run_settings (RunSettings) – defines how Model should be run
params (dict, optional) – model parameters for writing into configuration files
path (str, optional) – path to where the model should be executed at runtime
enable_key_prefixing (bool, optional) – If True, data sent to the Orchestrator using SmartRedis from this Model will be prefixed with the Model name. Default is True.
batch_settings (BatchSettings | None) – Settings to run model individually as a batch job, defaults to None

Raises:

SmartSimError – if initialization fails

Returns:

the created Model

Return type:

Model

Create a RunSettings instance.

run_command=”auto” will attempt to automatically match a run command on the system with a RunSettings class in SmartSim. If found, the class corresponding to that run_command will be created and returned.

If the local launcher is being used, auto detection will be turned off.

If a recognized run command is passed, the RunSettings instance will be a child class such as SrunSettings

If not supported by smartsim, the base RunSettings class will be created and returned with the specified run_command and run_args will be evaluated literally.

Run Commands with implemented helper classes:

aprun (ALPS)
srun (SLURM)
mpirun (OpenMPI)
jsrun (LSF)

Parameters:

run_command (str) – command to run the executable
exe (str) – executable to run
exe_args (list[str], optional) – arguments to pass to the executable
run_args (dict[str, t.Union[int, str, float, None]], optional) – arguments to pass to the run_command
env_vars (dict[str, str], optional) – environment variables to pass to the executable
container (Container, optional) – if execution environment is containerized

Returns:

the created RunSettings

Return type:

RunSettings

disable_telemetry() → None[source]#: Experiments will stop producing telemetry for all entities run through Experiment.start

Warning

This method is currently implemented so that ALL Experiment instances will stop producing telemetry data. In the future it is planned to have this method work on a “per instance” basis!

enable_telemetry() → None[source]#: Experiments will start producing telemetry for all entities run through Experiment.start

Warning

This method is currently implemented so that ALL Experiment instances will begin producing telemetry data. In the future it is planned to have this method work on a “per instance” basis!

finished(entity: SmartSimEntity) → bool[source]#

Query if a job has completed.

An instance of Model or Ensemble can be passed as an argument.

Passing Orchestrator will return an error as a database deployment is never finished until stopped by the user.

Parameters:: entity (Model | Ensemble) – object launched by this Experiment
Returns:: True if job has completed, False otherwise
Return type:: bool
Raises:: SmartSimError – if entity has not been launched by this Experiment

generate(*args: Any, tag: str | None = None, overwrite: bool = False, verbose: bool = False) → None[source]#

Generate the file structure for an Experiment

Experiment.generate creates directories for each instance passed to organize Experiments that launch many instances.

If files or directories are attached to Model objects using Model.attach_generator_files(), those files or directories will be symlinked, copied, or configured and written into the created directory for that instance.

Instances of Model, Ensemble and Orchestrator can all be passed as arguments to the generate method.

Parameters:

tag (str, optional) – tag used in to_configure generator files
overwrite (bool, optional) – overwrite existing folders and contents, defaults to False
verbose (bool) – log parameter settings to std out

get_status(*args: Any) → List[str][source]#

Query the status of launched instances

Return a smartsim.status string representing the status of the launched instance.

exp.get_status(model)

As with an Experiment method, multiple instance of varying types can be passed to and all statuses will be returned at once.

statuses = exp.get_status(model, ensemble, orchestrator)
complete = [s == smartsim.status.STATUS_COMPLETED for s in statuses]
assert all(complete)

Returns:: status of the instances passed as arguments
Return type:: list[str]
Raises:: SmartSimError – if status retrieval fails

poll(interval: int = 10, verbose: bool = True, kill_on_interrupt: bool = True) → None[source]#

Monitor jobs through logging to stdout.

This method should only be used if jobs were launched with Experiment.start(block=False)

The internal specified will control how often the logging is performed, not how often the polling occurs. By default, internal polling is set to every second for local launcher jobs and every 10 seconds for all other launchers.

If internal polling needs to be slower or faster based on system or site standards, set the SMARTSIM_JM_INTERNAL environment variable to control the internal polling interval for SmartSim.

For more verbose logging output, the SMARTSIM_LOG_LEVEL environment variable can be set to debug

If kill_on_interrupt=True, then all jobs launched by this experiment are guaranteed to be killed when ^C (SIGINT) signal is received. If kill_on_interrupt=False, then it is not guaranteed that all jobs launched by this experiment will be killed, and the zombie processes will need to be manually killed.

Parameters:

interval (int, optional) – frequency (in seconds) of logging to stdout, defaults to 10 seconds
verbose (bool, optional) – set verbosity, defaults to True
kill_on_interrupt (bool, optional) – flag for killing jobs when SIGINT is received

Raises:

SmartSimError –

reconnect_orchestrator(checkpoint: str) → Orchestrator[source]#

Reconnect to a running Orchestrator

This method can be used to connect to a Orchestrator deployment that was launched by a previous Experiment. This can be helpful in the case where separate runs of an Experiment wish to use the same Orchestrator instance currently running on a system.

Parameters:: checkpoint (str) – the smartsim_db.dat file created when an Orchestrator is launched

start(*args: Any, block: bool = True, summary: bool = False, kill_on_interrupt: bool = True) → None[source]#

Start passed instances using Experiment launcher

Any instance Model, Ensemble or Orchestrator instance created by the Experiment can be passed as an argument to the start method.

exp = Experiment(name="my_exp", launcher="slurm")
settings = exp.create_run_settings(exe="./path/to/binary")
model = exp.create_model("my_model", settings)
exp.start(model)

Multiple instance can also be passed to the start method at once no matter which type of instance they are. These will all be launched together.

exp.start(model_1, model_2, db, ensemble, block=True)
# alternatively
stage_1 = [model_1, model_2, db, ensemble]
exp.start(*stage_1, block=True)

If block==True the Experiment will poll the launched instances at runtime until all non-database jobs have completed. Database jobs must be killed by the user by passing them to Experiment.stop. This allows for multiple stages of a workflow to produce to and consume from the same Orchestrator database.

Parameters:

block (bool, optional) – block execution until all non-database jobs are finished, defaults to True
summary (bool, optional) – print a launch summary prior to launch, defaults to False
kill_on_interrupt (bool, optional) – flag for killing jobs when ^C (SIGINT) signal is received.

stop(*args: Any) → None[source]#

Stop specific instances launched by this Experiment

Instances of Model, Ensemble and Orchestrator can all be passed as arguments to the stop method.

Whichever launcher was specified at Experiment initialization will be used to stop the instance. For example, which using the slurm launcher, this equates to running scancel on the instance.

Example

exp.stop(model)
# multiple
exp.stop(model_1, model_2, db, ensemble)

Raises:

TypeError – if wrong type
SmartSimError – if stop request fails

summary(style: str = 'github') → str[source]#

Return a summary of the Experiment

The summary will show each instance that has been launched and completed in this Experiment

Parameters:: style (str, optional) – the style in which the summary table is formatted, for a full list of styles see: astanin/python-tabulate, defaults to “github”
Returns:: tabulate string of Experiment history
Return type:: str

Settings#

Settings are provided to Model and Ensemble objects to provide parameters for how a job should be executed. Some are specifically meant for certain launchers like SbatchSettings is solely meant for system using Slurm as a workload manager. MpirunSettings for OpenMPI based jobs is supported by Slurm and PBSPro.

Types of Settings:

`RunSettings`(exe[, exe_args, run_command, ...])	Run parameters for a `Model`
`SrunSettings`(exe[, exe_args, run_args, ...])	Initialize run parameters for a slurm job with `srun`
`AprunSettings`(exe[, exe_args, run_args, ...])	Settings to run job with `aprun` command
`MpirunSettings`(exe[, exe_args, run_args, ...])	Settings to run job with `mpirun` command (MPI-standard)
`MpiexecSettings`(exe[, exe_args, run_args, ...])	Settings to run job with `mpiexec` command (MPI-standard)
`OrterunSettings`(exe[, exe_args, run_args, ...])	Settings to run job with `orterun` command (MPI-standard)
`JsrunSettings`(exe[, exe_args, run_args, ...])	Settings to run job with `jsrun` command
`SbatchSettings`([nodes, time, account, ...])	Specify run parameters for a Slurm batch job
`QsubBatchSettings`([nodes, ncpus, time, ...])	Specify `qsub` batch parameters for a job
`BsubBatchSettings`([nodes, time, project, ...])	Specify `bsub` batch parameters for a job

Settings objects can accept a container object that defines a container runtime, image, and arguments to use for the workload. Below is a list of supported container runtimes.

Types of Containers:

Singularity(*args, **kwargs)

Singularity (apptainer) container type.

RunSettings#

When running SmartSim on laptops and single node workstations, the base RunSettings object is used to parameterize jobs. RunSettings include a run_command parameter for local launches that utilize a parallel launch binary like mpirun, mpiexec, and others.

`RunSettings.add_exe_args`(args)	Add executable arguments to executable
`RunSettings.update_env`(env_vars)	Update the job environment variables

Run parameters for a Model

The base RunSettings class should only be used with the local launcher on single node, workstations, or laptops.

If no run_command is specified, the executable will be launched locally.

run_args passed as a dict will be interpreted literally for local RunSettings and added directly to the run_command e.g. run_args = {“-np”: 2} will be “-np 2”

Example initialization

rs = RunSettings("echo", "hello", "mpirun", run_args={"-np": "2"})

Parameters:

exe (str) – executable to run
exe_args (str | list[str], optional) – executable arguments, defaults to None
run_command (str, optional) – launch binary (e.g. “srun”), defaults to empty str
run_args (dict[str, str], optional) – arguments for run command (e.g. -np for mpiexec), defaults to None
env_vars (dict[str, str], optional) – environment vars to launch job with, defaults to None
container (Container, optional) – container type for workload (e.g. “singularity”), defaults to None

add_exe_args(args: str | List[str]) → None[source]#

Add executable arguments to executable

Parameters:: args (str | list[str]) – executable arguments
Raises:: TypeError – if exe args are not strings

property env_vars: Dict[str, str | None]#

property exe_args: str | List[str]#

format_env_vars() → List[str][source]#

Build environment variable string

Returns:: formatted list of strings to export variables
Return type:: list[str]

format_run_args() → List[str][source]#

Return formatted run arguments

For RunSettings, the run arguments are passed literally with no formatting.

Returns:: list run arguments for these settings
Return type:: list[str]

make_mpmd(settings: RunSettings) → None[source]#

Make job an MPMD job

Parameters:: settings (RunSettings) – RunSettings instance

reserved_run_args: set[str] = {}#

property run_args: Dict[str, int | str | float | None]#

property run_command: str | None#

Return the launch binary used to launch the executable

Attempt to expand the path to the executable if possible

Returns:: launch binary e.g. mpiexec
Type:: str | None

set(arg: str, value: str | None = None, condition: bool = True) → None[source]#

Allows users to set individual run arguments.

A method that allows users to set run arguments after object instantiation. Does basic formatting such as stripping leading dashes. If the argument has been set previously, this method will log warning but ultimately comply.

Conditional expressions may be passed to the conditional parameter. If the expression evaluates to True, the argument will be set. In not an info message is logged and no further operation is performed.

Basic Usage

rs = RunSettings("python")
rs.set("an-arg", "a-val")
rs.set("a-flag")
rs.format_run_args()  # returns ["an-arg", "a-val", "a-flag", "None"]

Slurm Example with Conditional Setting

import socket

rs = SrunSettings("echo", "hello")
rs.set_tasks(1)
rs.set("exclusive")

# Only set this argument if condition param evals True
# Otherwise log and NOP
rs.set("partition", "debug",
       condition=socket.gethostname()=="testing-system")

rs.format_run_args()
# returns ["exclusive", "None", "partition", "debug"] iff
  socket.gethostname()=="testing-system"
# otherwise returns ["exclusive", "None"]

Parameters:

arg (str) – name of the argument
value (str | None) – value of the argument
conditon – set the argument if condition evaluates to True

set_binding(binding: str) → None[source]#

Set binding

Parameters:: binding (str) – Binding

set_broadcast(dest_path: str | None = None) → None[source]#

Copy executable file to allocated compute nodes

Parameters:: dest_path (str | None) – Path to copy an executable file

set_cpu_bindings(bindings: int | List[int]) → None[source]#

Set the cores to which MPI processes are bound

Parameters:: bindings (list[int] | int) – List specifing the cores to which MPI processes are bound

set_cpus_per_task(cpus_per_task: int) → None[source]#

Set the number of cpus per task

Parameters:: cpus_per_task (int) – number of cpus per task

set_excluded_hosts(host_list: str | List[str]) → None[source]#

Specify a list of hosts to exclude for launching this job

Parameters:: host_list (str | list[str]) – hosts to exclude

set_hostlist(host_list: str | List[str]) → None[source]#

Specify the hostlist for this job

Parameters:: host_list (str | list[str]) – hosts to launch on

set_hostlist_from_file(file_path: str) → None[source]#

Use the contents of a file to specify the hostlist for this job

Parameters:: file_path (str) – Path to the hostlist file

set_memory_per_node(memory_per_node: int) → None[source]#

Set the amount of memory required per node in megabytes

Parameters:: memory_per_node (int) – Number of megabytes per node

set_mpmd_preamble(preamble_lines: List[str]) → None[source]#

Set preamble to a file to make a job MPMD

Parameters:: preamble_lines (list[str]) – lines to put at the beginning of a file.

set_nodes(nodes: int) → None[source]#

Set the number of nodes

Parameters:: nodes (int) – number of nodes to run with

set_quiet_launch(quiet: bool) → None[source]#

Set the job to run in quiet mode

Parameters:: quiet (bool) – Whether the job should be run quietly

set_task_map(task_mapping: str) → None[source]#

Set a task mapping

Parameters:: task_mapping (str) – task mapping

set_tasks(tasks: int) → None[source]#

Set the number of tasks to launch

Parameters:: tasks (int) – number of tasks to launch

set_tasks_per_node(tasks_per_node: int) → None[source]#

Set the number of tasks per node

Parameters:: tasks_per_node (int) – number of tasks to launch per node

set_time(hours: int = 0, minutes: int = 0, seconds: int = 0) → None[source]#

Automatically format and set wall time

Parameters:

hours (int) – number of hours to run job
minutes (int) – number of minutes to run job
seconds (int) – number of seconds to run job

set_verbose_launch(verbose: bool) → None[source]#

Set the job to run in verbose mode

Parameters:: verbose (bool) – Whether the job should be run verbosely

set_walltime(walltime: str) → None[source]#

Set the formatted walltime

Parameters:: walltime (str) – Time in format required by launcher``

update_env(env_vars: Dict[str, str | int | float | bool]) → None[source]#

Update the job environment variables

To fully inherit the current user environment, add the workload-manager-specific flag to the launch command through the add_exe_args() method. For example, --export=ALL for slurm, or -V for PBS/aprun.

Parameters:: env_vars (dict[str, Union[str, int, float, bool]]) – environment variables to update or add
Raises:: TypeError – if env_vars values cannot be coerced to strings

SrunSettings#

SrunSettings can be used for running on existing allocations, running jobs in interactive allocations, and for adding srun steps to a batch.

`SrunSettings.set_nodes`(nodes)	Set the number of nodes
`SrunSettings.set_tasks`(tasks)	Set the number of tasks for this job
`SrunSettings.set_tasks_per_node`(tasks_per_node)	Set the number of tasks for this job
`SrunSettings.set_walltime`(walltime)	Set the walltime of the job
`SrunSettings.set_hostlist`(host_list)	Specify the hostlist for this job
`SrunSettings.set_excluded_hosts`(host_list)	Specify a list of hosts to exclude for launching this job
`SrunSettings.set_cpus_per_task`(cpus_per_task)	Set the number of cpus to use per task
`SrunSettings.add_exe_args`(args)	Add executable arguments to executable
`SrunSettings.format_run_args`()	Return a list of slurm formatted run arguments
`SrunSettings.format_env_vars`()	Build bash compatible environment variable string for Slurm
`SrunSettings.update_env`(env_vars)	Update the job environment variables

Initialize run parameters for a slurm job with srun

SrunSettings should only be used on Slurm based systems.

If an allocation is specified, the instance receiving these run parameters will launch on that allocation.

Parameters:

exe (str) – executable to run
exe_args (list[str] | str, optional) – executable arguments, defaults to None
run_args (dict[str, t.Union[int, str, float, None]], optional) – srun arguments without dashes, defaults to None
env_vars (dict[str, str], optional) – environment variables for job, defaults to None
alloc (str, optional) – allocation ID if running on existing alloc, defaults to None

add_exe_args(args: str | List[str]) → None#

Add executable arguments to executable

Parameters:: args (str | list[str]) – executable arguments
Raises:: TypeError – if exe args are not strings

check_env_vars() → None[source]#

Warn a user trying to set a variable which is set in the environment

Given Slurm’s env var precedence, trying to export a variable which is already present in the environment will not work.

colocated_db_settings: t.Optional[t.Dict[str, t.Union[bool, int, str, None, t.List[str], t.Iterable[t.Union[int, t.Iterable[int]]], t.List[DBModel], t.List[DBScript], t.Dict[str, t.Union[int, None]], t.Dict[str, str]]]]#

property env_vars: Dict[str, str | None]#

property exe_args: str | List[str]#

format_comma_sep_env_vars() → Tuple[str, List[str]][source]#

Build environment variable string for Slurm

Slurm takes exports in comma separated lists the list starts with all as to not disturb the rest of the environment for more information on this, see the slurm documentation for srun

Returns:: the formatted string of environment variables
Return type:: tuple[str, list[str]]

format_env_vars() → List[str][source]#

Build bash compatible environment variable string for Slurm

Returns:: the formatted string of environment variables
Return type:: list[str]

format_run_args() → List[str][source]#

Return a list of slurm formatted run arguments

Returns:: list of slurm arguments for these settings
Return type:: list[str]

make_mpmd(settings: RunSettings) → None[source]#

Make a mpmd workload by combining two srun commands

This connects the two settings to be executed with a single Model instance

Parameters:: settings (SrunSettings) – SrunSettings instance

reserved_run_args: set[str] = {'D', 'chdir'}#

property run_args: Dict[str, int | str | float | None]#

property run_command: str | None#

Return the launch binary used to launch the executable

Attempt to expand the path to the executable if possible

Returns:: launch binary e.g. mpiexec
Type:: str | None

set(arg: str, value: str | None = None, condition: bool = True) → None#

Allows users to set individual run arguments.

Basic Usage

rs = RunSettings("python")
rs.set("an-arg", "a-val")
rs.set("a-flag")
rs.format_run_args()  # returns ["an-arg", "a-val", "a-flag", "None"]

Slurm Example with Conditional Setting

import socket

rs = SrunSettings("echo", "hello")
rs.set_tasks(1)
rs.set("exclusive")

# Only set this argument if condition param evals True
# Otherwise log and NOP
rs.set("partition", "debug",
       condition=socket.gethostname()=="testing-system")

rs.format_run_args()
# returns ["exclusive", "None", "partition", "debug"] iff
  socket.gethostname()=="testing-system"
# otherwise returns ["exclusive", "None"]

Parameters:

arg (str) – name of the argument
value (str | None) – value of the argument
conditon – set the argument if condition evaluates to True

set_binding(binding: str) → None#

Set binding

Parameters:: binding (str) – Binding

set_broadcast(dest_path: str | None = None) → None[source]#

Copy executable file to allocated compute nodes

This sets --bcast

Parameters:: dest_path (str | None) – Path to copy an executable file

set_cpu_bindings(bindings: int | List[int]) → None[source]#

Bind by setting CPU masks on tasks

This sets --cpu-bind using the map_cpu:<list> option

Parameters:: bindings (list[int] | int) – List specifing the cores to which MPI processes are bound

set_cpus_per_task(cpus_per_task: int) → None[source]#

Set the number of cpus to use per task

This sets --cpus-per-task

Parameters:: num_cpus (int) – number of cpus to use per task

set_excluded_hosts(host_list: str | List[str]) → None[source]#

Specify a list of hosts to exclude for launching this job

Parameters:: host_list (list[str]) – hosts to exclude
Raises:: TypeError –

set_het_group(het_group: Iterable[int]) → None[source]#

Set the heterogeneous group for this job

this sets –het-group

Parameters:: het_group (int or iterable of ints) – list of heterogeneous groups

set_hostlist(host_list: str | List[str]) → None[source]#

Specify the hostlist for this job

This sets --nodelist

Parameters:: host_list (str | list[str]) – hosts to launch on
Raises:: TypeError – if not str or list of str

set_hostlist_from_file(file_path: str) → None[source]#

Use the contents of a file to set the node list

This sets --nodefile

Parameters:: file_path (str) – Path to the hostlist file

set_memory_per_node(memory_per_node: int) → None[source]#

Specify the real memory required per node

This sets --mem in megabytes

Parameters:: memory_per_node (int) – Amount of memory per node in megabytes

set_mpmd_preamble(preamble_lines: List[str]) → None#

Set preamble to a file to make a job MPMD

Parameters:: preamble_lines (list[str]) – lines to put at the beginning of a file.

set_nodes(nodes: int) → None[source]#

Set the number of nodes

Effectively this is setting: srun --nodes <num_nodes>

Parameters:: nodes (int) – number of nodes to run with

set_quiet_launch(quiet: bool) → None[source]#

Set the job to run in quiet mode

This sets --quiet

Parameters:: quiet (bool) – Whether the job should be run quietly

set_task_map(task_mapping: str) → None#

Set a task mapping

Parameters:: task_mapping (str) – task mapping

set_tasks(tasks: int) → None[source]#

Set the number of tasks for this job

This sets --ntasks

Parameters:: tasks (int) – number of tasks

set_tasks_per_node(tasks_per_node: int) → None[source]#

Set the number of tasks for this job

This sets --ntasks-per-node

Parameters:: tasks_per_node (int) – number of tasks per node

set_time(hours: int = 0, minutes: int = 0, seconds: int = 0) → None#

Automatically format and set wall time

Parameters:

hours (int) – number of hours to run job
minutes (int) – number of minutes to run job
seconds (int) – number of seconds to run job

set_verbose_launch(verbose: bool) → None[source]#

Set the job to run in verbose mode

This sets --verbose

Parameters:: verbose (bool) – Whether the job should be run verbosely

set_walltime(walltime: str) → None[source]#

Set the walltime of the job

format = “HH:MM:SS”

Parameters:: walltime (str) – wall time

update_env(env_vars: Dict[str, str | int | float | bool]) → None#

Update the job environment variables

Parameters:: env_vars (dict[str, Union[str, int, float, bool]]) – environment variables to update or add
Raises:: TypeError – if env_vars values cannot be coerced to strings

AprunSettings#

AprunSettings can be used on any system that supports the Cray ALPS layer. SmartSim supports using AprunSettings on PBSPro WLM systems.

AprunSettings can be used in interactive session (on allocation) and within batch launches (e.g., QsubBatchSettings)

`AprunSettings.set_cpus_per_task`(cpus_per_task)	Set the number of cpus to use per task
`AprunSettings.set_hostlist`(host_list)	Specify the hostlist for this job
`AprunSettings.set_tasks`(tasks)	Set the number of tasks for this job
`AprunSettings.set_tasks_per_node`(tasks_per_node)	Set the number of tasks for this job
`AprunSettings.make_mpmd`(settings)	Make job an MPMD job
`AprunSettings.add_exe_args`(args)	Add executable arguments to executable
`AprunSettings.format_run_args`()	Return a list of ALPS formatted run arguments
`AprunSettings.format_env_vars`()	Format the environment variables for aprun
`AprunSettings.update_env`(env_vars)	Update the job environment variables

Settings to run job with aprun command

AprunSettings can be used for the pbs launcher.

Parameters:

exe (str) – executable
exe_args (str | list[str], optional) – executable arguments, defaults to None
run_args (dict[str, t.Union[int, str, float, None]], optional) – arguments for run command, defaults to None
env_vars (dict[str, str], optional) – environment vars to launch job with, defaults to None

add_exe_args(args: str | List[str]) → None#

Add executable arguments to executable

Parameters:: args (str | list[str]) – executable arguments
Raises:: TypeError – if exe args are not strings

colocated_db_settings: t.Optional[t.Dict[str, t.Union[bool, int, str, None, t.List[str], t.Iterable[t.Union[int, t.Iterable[int]]], t.List[DBModel], t.List[DBScript], t.Dict[str, t.Union[int, None]], t.Dict[str, str]]]]#

property env_vars: Dict[str, str | None]#

property exe_args: str | List[str]#

format_env_vars() → List[str][source]#

Format the environment variables for aprun

Returns:: list of env vars
Return type:: list[str]

format_run_args() → List[str][source]#

Return a list of ALPS formatted run arguments

Returns:: list of ALPS arguments for these settings
Return type:: list[str]

make_mpmd(settings: RunSettings) → None[source]#

Make job an MPMD job

This method combines two AprunSettings into a single MPMD command joined with ‘:’

Parameters:: settings (AprunSettings) – AprunSettings instance

reserved_run_args: set[str] = {}#

property run_args: Dict[str, int | str | float | None]#

property run_command: str | None#

Return the launch binary used to launch the executable

Attempt to expand the path to the executable if possible

Returns:: launch binary e.g. mpiexec
Type:: str | None

set(arg: str, value: str | None = None, condition: bool = True) → None#

Allows users to set individual run arguments.

Basic Usage

rs = RunSettings("python")
rs.set("an-arg", "a-val")
rs.set("a-flag")
rs.format_run_args()  # returns ["an-arg", "a-val", "a-flag", "None"]

Slurm Example with Conditional Setting

import socket

rs = SrunSettings("echo", "hello")
rs.set_tasks(1)
rs.set("exclusive")

# Only set this argument if condition param evals True
# Otherwise log and NOP
rs.set("partition", "debug",
       condition=socket.gethostname()=="testing-system")

rs.format_run_args()
# returns ["exclusive", "None", "partition", "debug"] iff
  socket.gethostname()=="testing-system"
# otherwise returns ["exclusive", "None"]

Parameters:

arg (str) – name of the argument
value (str | None) – value of the argument
conditon – set the argument if condition evaluates to True

set_binding(binding: str) → None#

Set binding

Parameters:: binding (str) – Binding

set_broadcast(dest_path: str | None = None) → None#

Copy executable file to allocated compute nodes

Parameters:: dest_path (str | None) – Path to copy an executable file

set_cpu_bindings(bindings: int | List[int]) → None[source]#

Specifies the cores to which MPI processes are bound

This sets --cpu-binding

Parameters:: bindings (list[int] | int) – List of cpu numbers

set_cpus_per_task(cpus_per_task: int) → None[source]#

Set the number of cpus to use per task

This sets --cpus-per-pe

Parameters:: cpus_per_task (int) – number of cpus to use per task

set_excluded_hosts(host_list: str | List[str]) → None[source]#

Specify a list of hosts to exclude for launching this job

Parameters:: host_list (str | list[str]) – hosts to exclude
Raises:: TypeError – if not str or list of str

set_hostlist(host_list: str | List[str]) → None[source]#

Specify the hostlist for this job

Parameters:: host_list (str | list[str]) – hosts to launch on
Raises:: TypeError – if not str or list of str

set_hostlist_from_file(file_path: str) → None[source]#

Use the contents of a file to set the node list

This sets --node-list-file

Parameters:: file_path (str) – Path to the hostlist file

set_memory_per_node(memory_per_node: int) → None[source]#

Specify the real memory required per node

This sets --memory-per-pe in megabytes

Parameters:: memory_per_node (int) – Per PE memory limit in megabytes

set_mpmd_preamble(preamble_lines: List[str]) → None#

Set preamble to a file to make a job MPMD

Parameters:: preamble_lines (list[str]) – lines to put at the beginning of a file.

set_nodes(nodes: int) → None#

Set the number of nodes

Parameters:: nodes (int) – number of nodes to run with

set_quiet_launch(quiet: bool) → None[source]#

Set the job to run in quiet mode

This sets --quiet

Parameters:: quiet (bool) – Whether the job should be run quietly

set_task_map(task_mapping: str) → None#

Set a task mapping

Parameters:: task_mapping (str) – task mapping

set_tasks(tasks: int) → None[source]#

Set the number of tasks for this job

This sets --pes

Parameters:: tasks (int) – number of tasks

set_tasks_per_node(tasks_per_node: int) → None[source]#

Set the number of tasks for this job

This sets --pes-per-node

Parameters:: tasks_per_node (int) – number of tasks per node

set_time(hours: int = 0, minutes: int = 0, seconds: int = 0) → None#

Automatically format and set wall time

Parameters:

hours (int) – number of hours to run job
minutes (int) – number of minutes to run job
seconds (int) – number of seconds to run job

set_verbose_launch(verbose: bool) → None[source]#

Set the job to run in verbose mode

This sets --debug arg to the highest level

Parameters:: verbose (bool) – Whether the job should be run verbosely

set_walltime(walltime: str) → None[source]#

Set the walltime of the job

Walltime is given in total number of seconds

Parameters:: walltime (str) – wall time

update_env(env_vars: Dict[str, str | int | float | bool]) → None#

Update the job environment variables

Parameters:: env_vars (dict[str, Union[str, int, float, bool]]) – environment variables to update or add
Raises:: TypeError – if env_vars values cannot be coerced to strings

JsrunSettings#

JsrunSettings can be used on any system that supports the IBM LSF launcher.

JsrunSettings can be used in interactive session (on allocation) and within batch launches (i.e. BsubBatchSettings)

`JsrunSettings.set_num_rs`(num_rs)	Set the number of resource sets to use
`JsrunSettings.set_cpus_per_rs`(cpus_per_rs)	Set the number of cpus to use per resource set
`JsrunSettings.set_gpus_per_rs`(gpus_per_rs)	Set the number of gpus to use per resource set
`JsrunSettings.set_rs_per_host`(rs_per_host)	Set the number of resource sets to use per host
`JsrunSettings.set_tasks`(tasks)	Set the number of tasks for this job
`JsrunSettings.set_tasks_per_rs`(tasks_per_rs)	Set the number of tasks per resource set
`JsrunSettings.set_binding`(binding)	Set binding
`JsrunSettings.make_mpmd`(settings)	Make step an MPMD (or SPMD) job.
`JsrunSettings.set_mpmd_preamble`(preamble_lines)	Set preamble used in ERF file.
`JsrunSettings.update_env`(env_vars)	Update the job environment variables
`JsrunSettings.set_erf_sets`(erf_sets)	Set resource sets used for ERF (SPMD or MPMD) steps.
`JsrunSettings.format_env_vars`()	Format environment variables.
`JsrunSettings.format_run_args`()	Return a list of LSF formatted run arguments

Settings to run job with jsrun command

JsrunSettings should only be used on LSF-based systems.

Parameters:

exe (str) – executable
exe_args (str | list[str], optional) – executable arguments, defaults to None
run_args (dict[str, t.Union[int, str, float, None]], optional) – arguments for run command, defaults to None
env_vars (dict[str, str], optional) – environment vars to launch job with, defaults to None

add_exe_args(args: str | List[str]) → None#

Add executable arguments to executable

Parameters:: args (str | list[str]) – executable arguments
Raises:: TypeError – if exe args are not strings

colocated_db_settings: t.Optional[t.Dict[str, t.Union[bool, int, str, None, t.List[str], t.Iterable[t.Union[int, t.Iterable[int]]], t.List[DBModel], t.List[DBScript], t.Dict[str, t.Union[int, None]], t.Dict[str, str]]]]#

property env_vars: Dict[str, str | None]#

property exe_args: str | List[str]#

format_env_vars() → List[str][source]#

Format environment variables. Each variable needs to be passed with --env. If a variable is set to None, its value is propagated from the current environment.

Returns:: formatted list of strings to export variables
Return type:: list[str]

format_run_args() → List[str][source]#

Return a list of LSF formatted run arguments

Returns:: list of LSF arguments for these settings
Return type:: list[str]

make_mpmd(settings: RunSettings) → None[source]#

Make step an MPMD (or SPMD) job.

This method will activate job execution through an ERF file.

Optionally, this method adds an instance of JsrunSettings to the list of settings to be launched in the same ERF file.

Parameters:: settings (JsrunSettings, optional) – JsrunSettings instance

reserved_run_args: set[str] = {'chdir', 'h'}#

property run_args: Dict[str, int | str | float | None]#

property run_command: str | None#

Return the launch binary used to launch the executable

Attempt to expand the path to the executable if possible

Returns:: launch binary e.g. mpiexec
Type:: str | None

set(arg: str, value: str | None = None, condition: bool = True) → None#

Allows users to set individual run arguments.

Basic Usage

rs = RunSettings("python")
rs.set("an-arg", "a-val")
rs.set("a-flag")
rs.format_run_args()  # returns ["an-arg", "a-val", "a-flag", "None"]

Slurm Example with Conditional Setting

import socket

rs = SrunSettings("echo", "hello")
rs.set_tasks(1)
rs.set("exclusive")

# Only set this argument if condition param evals True
# Otherwise log and NOP
rs.set("partition", "debug",
       condition=socket.gethostname()=="testing-system")

rs.format_run_args()
# returns ["exclusive", "None", "partition", "debug"] iff
  socket.gethostname()=="testing-system"
# otherwise returns ["exclusive", "None"]

Parameters:

arg (str) – name of the argument
value (str | None) – value of the argument
conditon – set the argument if condition evaluates to True

set_binding(binding: str) → None[source]#

Set binding

This sets --bind

Parameters:: binding (str) – Binding, e.g. packed:21

set_broadcast(dest_path: str | None = None) → None#

Copy executable file to allocated compute nodes

Parameters:: dest_path (str | None) – Path to copy an executable file

set_cpu_bindings(bindings: int | List[int]) → None#

Set the cores to which MPI processes are bound

Parameters:: bindings (list[int] | int) – List specifing the cores to which MPI processes are bound

set_cpus_per_rs(cpus_per_rs: int) → None[source]#

Set the number of cpus to use per resource set

This sets --cpu_per_rs

Parameters:: cpus_per_rs (int or str) – number of cpus to use per resource set or ALL_CPUS

set_cpus_per_task(cpus_per_task: int) → None[source]#

Set the number of cpus per tasks.

This function is an alias for set_cpus_per_rs.

Parameters:: cpus_per_task (int) – number of cpus per resource set

set_erf_sets(erf_sets: Dict[str, str]) → None[source]#

Set resource sets used for ERF (SPMD or MPMD) steps.

erf_sets is a dictionary used to fill the ERF line representing these settings, e.g. {“host”: “1”, “cpu”: “{0:21}, {21:21}”, “gpu”: “*”} can be used to specify rank (or rank_count), hosts, cpus, gpus, and memory. The key rank is used to give specific ranks, as in {“rank”: “1, 2, 5”}, while the key rank_count is used to specify the count only, as in {“rank_count”: “3”}. If both are specified, only rank is used.

Parameters:: hosts (dict[str,str]) – dictionary of resources

set_excluded_hosts(host_list: str | List[str]) → None#

Specify a list of hosts to exclude for launching this job

Parameters:: host_list (str | list[str]) – hosts to exclude

set_gpus_per_rs(gpus_per_rs: int) → None[source]#

Set the number of gpus to use per resource set

This sets --gpu_per_rs

Parameters:: gpus_per_rs (int or str) – number of gpus to use per resource set or ALL_GPUS

set_hostlist(host_list: str | List[str]) → None#

Specify the hostlist for this job

Parameters:: host_list (str | list[str]) – hosts to launch on

set_hostlist_from_file(file_path: str) → None#

Use the contents of a file to specify the hostlist for this job

Parameters:: file_path (str) – Path to the hostlist file

set_individual_output(suffix: str | None = None) → None[source]#

Set individual std output.

This sets --stdio_mode individual and inserts the suffix into the output name. The resulting output name will be self.name + suffix + .out.

Parameters:: suffix (str, optional) – Optional suffix to add to output file names, it can contain %j, %h, %p, or %t, as specified by jsrun options.

set_memory_per_node(memory_per_node: int) → None[source]#

Specify the number of megabytes of memory to assign to a resource set

Alias for set_memory_per_rs.

Parameters:: memory_per_node (int) – Number of megabytes per rs

set_memory_per_rs(memory_per_rs: int) → None[source]#

Specify the number of megabytes of memory to assign to a resource set

This sets --memory_per_rs

Parameters:: memory_per_rs (int) – Number of megabytes per rs

set_mpmd_preamble(preamble_lines: List[str]) → None[source]#

Set preamble used in ERF file. Typical lines include oversubscribe-cpu : allow or overlapping-rs : allow. Can be used to set launch_distribution. If it is not present, it will be inferred from the settings, or set to packed by default.

Parameters:: preamble_lines (list[str]) – lines to put at the beginning of the ERF file.

set_nodes(nodes: int) → None#

Set the number of nodes

Parameters:: nodes (int) – number of nodes to run with

set_num_rs(num_rs: str | int) → None[source]#

Set the number of resource sets to use

This sets --nrs.

Parameters:: num_rs (int or str) – Number of resource sets or ALL_HOSTS

set_quiet_launch(quiet: bool) → None#

Set the job to run in quiet mode

Parameters:: quiet (bool) – Whether the job should be run quietly

set_rs_per_host(rs_per_host: int) → None[source]#

Set the number of resource sets to use per host

This sets --rs_per_host

Parameters:: rs_per_host (int) – number of resource sets to use per host

set_task_map(task_mapping: str) → None#

Set a task mapping

Parameters:: task_mapping (str) – task mapping

set_tasks(tasks: int) → None[source]#

Set the number of tasks for this job

This sets --np

Parameters:: tasks (int) – number of tasks

set_tasks_per_node(tasks_per_node: int) → None[source]#

Set the number of tasks per resource set.

This function is an alias for set_tasks_per_rs.

Parameters:: tasks_per_node (int) – number of tasks per resource set

set_tasks_per_rs(tasks_per_rs: int) → None[source]#

Set the number of tasks per resource set

This sets --tasks_per_rs

Parameters:: tasks_per_rs (int) – number of tasks per resource set

set_time(hours: int = 0, minutes: int = 0, seconds: int = 0) → None#

Automatically format and set wall time

Parameters:

hours (int) – number of hours to run job
minutes (int) – number of minutes to run job
seconds (int) – number of seconds to run job

set_verbose_launch(verbose: bool) → None#

Set the job to run in verbose mode

Parameters:: verbose (bool) – Whether the job should be run verbosely

set_walltime(walltime: str) → None#

Set the formatted walltime

Parameters:: walltime (str) – Time in format required by launcher``

update_env(env_vars: Dict[str, str | int | float | bool]) → None#

Update the job environment variables

Parameters:: env_vars (dict[str, Union[str, int, float, bool]]) – environment variables to update or add
Raises:: TypeError – if env_vars values cannot be coerced to strings

MpirunSettings#

MpirunSettings are for launching with OpenMPI. MpirunSettings are supported on Slurm and PBSpro.

`MpirunSettings.set_cpus_per_task`(cpus_per_task)	Set the number of tasks for this job
`MpirunSettings.set_hostlist`(host_list)	Set the hostlist for the `mpirun` command
`MpirunSettings.set_tasks`(tasks)	Set the number of tasks for this job
`MpirunSettings.set_task_map`(task_mapping)	Set `mpirun` task mapping
`MpirunSettings.make_mpmd`(settings)	Make a mpmd workload by combining two `mpirun` commands
`MpirunSettings.add_exe_args`(args)	Add executable arguments to executable
`MpirunSettings.format_run_args`()	Return a list of MPI-standard formatted run arguments
`MpirunSettings.format_env_vars`()	Format the environment variables for mpirun
`MpirunSettings.update_env`(env_vars)	Update the job environment variables

Settings to run job with mpirun command (MPI-standard)

Note that environment variables can be passed with a None value to signify that they should be exported from the current environment

Any arguments passed in the run_args dict will be converted into mpirun arguments and prefixed with --. Values of None can be provided for arguments that do not have values.

Parameters:

exe (str) – executable
exe_args (str | list[str], optional) – executable arguments, defaults to None
run_args (dict[str, t.Union[int, str, float, None]], optional) – arguments for run command, defaults to None
env_vars (dict[str, str], optional) – environment vars to launch job with, defaults to None

add_exe_args(args: str | List[str]) → None#

Add executable arguments to executable

Parameters:: args (str | list[str]) – executable arguments
Raises:: TypeError – if exe args are not strings

colocated_db_settings: t.Optional[t.Dict[str, t.Union[bool, int, str, None, t.List[str], t.Iterable[t.Union[int, t.Iterable[int]]], t.List[DBModel], t.List[DBScript], t.Dict[str, t.Union[int, None]], t.Dict[str, str]]]]#

property env_vars: Dict[str, str | None]#

property exe_args: str | List[str]#

format_env_vars() → List[str]#

Format the environment variables for mpirun

Returns:: list of env vars
Return type:: list[str]

format_run_args() → List[str]#

Return a list of MPI-standard formatted run arguments

Returns:: list of MPI-standard arguments for these settings
Return type:: list[str]

make_mpmd(settings: RunSettings) → None#

Make a mpmd workload by combining two mpirun commands

This connects the two settings to be executed with a single Model instance

Parameters:: settings (MpirunSettings) – MpirunSettings instance

reserved_run_args: set[str] = {'wd', 'wdir'}#

property run_args: Dict[str, int | str | float | None]#

property run_command: str | None#

Return the launch binary used to launch the executable

Attempt to expand the path to the executable if possible

Returns:: launch binary e.g. mpiexec
Type:: str | None

set(arg: str, value: str | None = None, condition: bool = True) → None#

Allows users to set individual run arguments.

Basic Usage

rs = RunSettings("python")
rs.set("an-arg", "a-val")
rs.set("a-flag")
rs.format_run_args()  # returns ["an-arg", "a-val", "a-flag", "None"]

Slurm Example with Conditional Setting

import socket

rs = SrunSettings("echo", "hello")
rs.set_tasks(1)
rs.set("exclusive")

# Only set this argument if condition param evals True
# Otherwise log and NOP
rs.set("partition", "debug",
       condition=socket.gethostname()=="testing-system")

rs.format_run_args()
# returns ["exclusive", "None", "partition", "debug"] iff
  socket.gethostname()=="testing-system"
# otherwise returns ["exclusive", "None"]

Parameters:

arg (str) – name of the argument
value (str | None) – value of the argument
conditon – set the argument if condition evaluates to True

set_binding(binding: str) → None#

Set binding

Parameters:: binding (str) – Binding

set_broadcast(dest_path: str | None = None) → None#

Copy the specified executable(s) to remote machines

This sets --preload-binary

Parameters:: dest_path (str | None) – Destination path (Ignored)

set_cpu_binding_type(bind_type: str) → None#

Specifies the cores to which MPI processes are bound

This sets --bind-to for MPI compliant implementations

Parameters:: bind_type (str) – binding type

set_cpu_bindings(bindings: int | List[int]) → None#

Set the cores to which MPI processes are bound

Parameters:: bindings (list[int] | int) – List specifing the cores to which MPI processes are bound

set_cpus_per_task(cpus_per_task: int) → None#

Set the number of tasks for this job

This sets --cpus-per-proc for MPI compliant implementations

note: this option has been deprecated in openMPI 4.0+ and will soon be replaced.

Parameters:: cpus_per_task (int) – number of tasks

set_excluded_hosts(host_list: str | List[str]) → None#

Specify a list of hosts to exclude for launching this job

Parameters:: host_list (str | list[str]) – hosts to exclude

set_hostlist(host_list: str | List[str]) → None#

Set the hostlist for the mpirun command

This sets --host

Parameters:: host_list (str | list[str]) – list of host names
Raises:: TypeError – if not str or list of str

set_hostlist_from_file(file_path: str) → None#

Use the contents of a file to set the hostlist

This sets --hostfile

Parameters:: file_path (str) – Path to the hostlist file

set_memory_per_node(memory_per_node: int) → None#

Set the amount of memory required per node in megabytes

Parameters:: memory_per_node (int) – Number of megabytes per node

set_mpmd_preamble(preamble_lines: List[str]) → None#

Set preamble to a file to make a job MPMD

Parameters:: preamble_lines (list[str]) – lines to put at the beginning of a file.

set_nodes(nodes: int) → None#

Set the number of nodes

Parameters:: nodes (int) – number of nodes to run with

set_quiet_launch(quiet: bool) → None#

Set the job to run in quiet mode

This sets --quiet

Parameters:: quiet (bool) – Whether the job should be run quietly

set_task_map(task_mapping: str) → None#

Set mpirun task mapping

this sets --map-by <mapping>

For examples, see the man page for mpirun

Parameters:: task_mapping (str) – task mapping

set_tasks(tasks: int) → None#

Set the number of tasks for this job

This sets -n for MPI compliant implementations

Parameters:: tasks (int) – number of tasks

set_tasks_per_node(tasks_per_node: int) → None#

Set the number of tasks per node

Parameters:: tasks_per_node (int) – number of tasks to launch per node

set_time(hours: int = 0, minutes: int = 0, seconds: int = 0) → None#

Automatically format and set wall time

Parameters:

hours (int) – number of hours to run job
minutes (int) – number of minutes to run job
seconds (int) – number of seconds to run job

set_verbose_launch(verbose: bool) → None#

Set the job to run in verbose mode

This sets --verbose

Parameters:: verbose (bool) – Whether the job should be run verbosely

set_walltime(walltime: str) → None#

Set the maximum number of seconds that a job will run

This sets --timeout

Parameters:: walltime (str) – number like string of seconds that a job will run in secs

update_env(env_vars: Dict[str, str | int | float | bool]) → None#

Update the job environment variables

Parameters:: env_vars (dict[str, Union[str, int, float, bool]]) – environment variables to update or add
Raises:: TypeError – if env_vars values cannot be coerced to strings

MpiexecSettings#

MpiexecSettings are for launching with OpenMPI’s mpiexec. MpirunSettings are supported on Slurm and PBSpro.

`MpiexecSettings.set_cpus_per_task`(cpus_per_task)	Set the number of tasks for this job
`MpiexecSettings.set_hostlist`(host_list)	Set the hostlist for the `mpirun` command
`MpiexecSettings.set_tasks`(tasks)	Set the number of tasks for this job
`MpiexecSettings.set_task_map`(task_mapping)	Set `mpirun` task mapping
`MpiexecSettings.make_mpmd`(settings)	Make a mpmd workload by combining two `mpirun` commands
`MpiexecSettings.add_exe_args`(args)	Add executable arguments to executable
`MpiexecSettings.format_run_args`()	Return a list of MPI-standard formatted run arguments
`MpiexecSettings.format_env_vars`()	Format the environment variables for mpirun
`MpiexecSettings.update_env`(env_vars)	Update the job environment variables

Settings to run job with mpiexec command (MPI-standard)

Note that environment variables can be passed with a None value to signify that they should be exported from the current environment

Any arguments passed in the run_args dict will be converted into mpiexec arguments and prefixed with --. Values of None can be provided for arguments that do not have values.

Parameters:

exe (str) – executable
exe_args (str | list[str], optional) – executable arguments, defaults to None
run_args (dict[str, t.Union[int, str, float, None]], optional) – arguments for run command, defaults to None
env_vars (dict[str, str], optional) – environment vars to launch job with, defaults to None

add_exe_args(args: str | List[str]) → None#

Add executable arguments to executable

Parameters:: args (str | list[str]) – executable arguments
Raises:: TypeError – if exe args are not strings

colocated_db_settings: t.Optional[t.Dict[str, t.Union[bool, int, str, None, t.List[str], t.Iterable[t.Union[int, t.Iterable[int]]], t.List[DBModel], t.List[DBScript], t.Dict[str, t.Union[int, None]], t.Dict[str, str]]]]#

property env_vars: Dict[str, str | None]#

property exe_args: str | List[str]#

format_env_vars() → List[str]#

Format the environment variables for mpirun

Returns:: list of env vars
Return type:: list[str]

format_run_args() → List[str]#

Return a list of MPI-standard formatted run arguments

Returns:: list of MPI-standard arguments for these settings
Return type:: list[str]

make_mpmd(settings: RunSettings) → None#

Make a mpmd workload by combining two mpirun commands

This connects the two settings to be executed with a single Model instance

Parameters:: settings (MpirunSettings) – MpirunSettings instance

reserved_run_args: set[str] = {'wd', 'wdir'}#

property run_args: Dict[str, int | str | float | None]#

property run_command: str | None#

Return the launch binary used to launch the executable

Attempt to expand the path to the executable if possible

Returns:: launch binary e.g. mpiexec
Type:: str | None

set(arg: str, value: str | None = None, condition: bool = True) → None#

Allows users to set individual run arguments.

Basic Usage

rs = RunSettings("python")
rs.set("an-arg", "a-val")
rs.set("a-flag")
rs.format_run_args()  # returns ["an-arg", "a-val", "a-flag", "None"]

Slurm Example with Conditional Setting

import socket

rs = SrunSettings("echo", "hello")
rs.set_tasks(1)
rs.set("exclusive")

# Only set this argument if condition param evals True
# Otherwise log and NOP
rs.set("partition", "debug",
       condition=socket.gethostname()=="testing-system")

rs.format_run_args()
# returns ["exclusive", "None", "partition", "debug"] iff
  socket.gethostname()=="testing-system"
# otherwise returns ["exclusive", "None"]

Parameters:

arg (str) – name of the argument
value (str | None) – value of the argument
conditon – set the argument if condition evaluates to True

set_binding(binding: str) → None#

Set binding

Parameters:: binding (str) – Binding

set_broadcast(dest_path: str | None = None) → None#

Copy the specified executable(s) to remote machines

This sets --preload-binary

Parameters:: dest_path (str | None) – Destination path (Ignored)

set_cpu_binding_type(bind_type: str) → None#

Specifies the cores to which MPI processes are bound

This sets --bind-to for MPI compliant implementations

Parameters:: bind_type (str) – binding type

set_cpu_bindings(bindings: int | List[int]) → None#

Set the cores to which MPI processes are bound

Parameters:: bindings (list[int] | int) – List specifing the cores to which MPI processes are bound

set_cpus_per_task(cpus_per_task: int) → None#

Set the number of tasks for this job

This sets --cpus-per-proc for MPI compliant implementations

note: this option has been deprecated in openMPI 4.0+ and will soon be replaced.

Parameters:: cpus_per_task (int) – number of tasks

set_excluded_hosts(host_list: str | List[str]) → None#

Specify a list of hosts to exclude for launching this job

Parameters:: host_list (str | list[str]) – hosts to exclude

set_hostlist(host_list: str | List[str]) → None#

Set the hostlist for the mpirun command

This sets --host

Parameters:: host_list (str | list[str]) – list of host names
Raises:: TypeError – if not str or list of str

set_hostlist_from_file(file_path: str) → None#

Use the contents of a file to set the hostlist

This sets --hostfile

Parameters:: file_path (str) – Path to the hostlist file

set_memory_per_node(memory_per_node: int) → None#

Set the amount of memory required per node in megabytes

Parameters:: memory_per_node (int) – Number of megabytes per node

set_mpmd_preamble(preamble_lines: List[str]) → None#

Set preamble to a file to make a job MPMD

Parameters:: preamble_lines (list[str]) – lines to put at the beginning of a file.

set_nodes(nodes: int) → None#

Set the number of nodes

Parameters:: nodes (int) – number of nodes to run with

set_quiet_launch(quiet: bool) → None#

Set the job to run in quiet mode

This sets --quiet

Parameters:: quiet (bool) – Whether the job should be run quietly

set_task_map(task_mapping: str) → None#

Set mpirun task mapping

this sets --map-by <mapping>

For examples, see the man page for mpirun

Parameters:: task_mapping (str) – task mapping

set_tasks(tasks: int) → None#

Set the number of tasks for this job

This sets -n for MPI compliant implementations

Parameters:: tasks (int) – number of tasks

set_tasks_per_node(tasks_per_node: int) → None#

Set the number of tasks per node

Parameters:: tasks_per_node (int) – number of tasks to launch per node

set_time(hours: int = 0, minutes: int = 0, seconds: int = 0) → None#

Automatically format and set wall time

Parameters:

hours (int) – number of hours to run job
minutes (int) – number of minutes to run job
seconds (int) – number of seconds to run job

set_verbose_launch(verbose: bool) → None#

Set the job to run in verbose mode

This sets --verbose

Parameters:: verbose (bool) – Whether the job should be run verbosely

set_walltime(walltime: str) → None#

Set the maximum number of seconds that a job will run

This sets --timeout

Parameters:: walltime (str) – number like string of seconds that a job will run in secs

update_env(env_vars: Dict[str, str | int | float | bool]) → None#

Update the job environment variables

Parameters:: env_vars (dict[str, Union[str, int, float, bool]]) – environment variables to update or add
Raises:: TypeError – if env_vars values cannot be coerced to strings

OrterunSettings#

OrterunSettings are for launching with OpenMPI’s orterun. OrterunSettings are supported on Slurm and PBSpro.

`OrterunSettings.set_cpus_per_task`(cpus_per_task)	Set the number of tasks for this job
`OrterunSettings.set_hostlist`(host_list)	Set the hostlist for the `mpirun` command
`OrterunSettings.set_tasks`(tasks)	Set the number of tasks for this job
`OrterunSettings.set_task_map`(task_mapping)	Set `mpirun` task mapping
`OrterunSettings.make_mpmd`(settings)	Make a mpmd workload by combining two `mpirun` commands
`OrterunSettings.add_exe_args`(args)	Add executable arguments to executable
`OrterunSettings.format_run_args`()	Return a list of MPI-standard formatted run arguments
`OrterunSettings.format_env_vars`()	Format the environment variables for mpirun
`OrterunSettings.update_env`(env_vars)	Update the job environment variables

Settings to run job with orterun command (MPI-standard)

Note that environment variables can be passed with a None value to signify that they should be exported from the current environment

Any arguments passed in the run_args dict will be converted into orterun arguments and prefixed with --. Values of None can be provided for arguments that do not have values.

Parameters:

exe (str) – executable
exe_args (str | list[str], optional) – executable arguments, defaults to None
run_args (dict[str, t.Union[int, str, float, None]], optional) – arguments for run command, defaults to None
env_vars (dict[str, str], optional) – environment vars to launch job with, defaults to None

add_exe_args(args: str | List[str]) → None#

Add executable arguments to executable

Parameters:: args (str | list[str]) – executable arguments
Raises:: TypeError – if exe args are not strings

colocated_db_settings: t.Optional[t.Dict[str, t.Union[bool, int, str, None, t.List[str], t.Iterable[t.Union[int, t.Iterable[int]]], t.List[DBModel], t.List[DBScript], t.Dict[str, t.Union[int, None]], t.Dict[str, str]]]]#

property env_vars: Dict[str, str | None]#

property exe_args: str | List[str]#

format_env_vars() → List[str]#

Format the environment variables for mpirun

Returns:: list of env vars
Return type:: list[str]

format_run_args() → List[str]#

Return a list of MPI-standard formatted run arguments

Returns:: list of MPI-standard arguments for these settings
Return type:: list[str]

make_mpmd(settings: RunSettings) → None#

Make a mpmd workload by combining two mpirun commands

This connects the two settings to be executed with a single Model instance

Parameters:: settings (MpirunSettings) – MpirunSettings instance

reserved_run_args: set[str] = {'wd', 'wdir'}#

property run_args: Dict[str, int | str | float | None]#

property run_command: str | None#

Return the launch binary used to launch the executable

Attempt to expand the path to the executable if possible

Returns:: launch binary e.g. mpiexec
Type:: str | None

set(arg: str, value: str | None = None, condition: bool = True) → None#

Allows users to set individual run arguments.

Basic Usage

rs = RunSettings("python")
rs.set("an-arg", "a-val")
rs.set("a-flag")
rs.format_run_args()  # returns ["an-arg", "a-val", "a-flag", "None"]

Slurm Example with Conditional Setting

import socket

rs = SrunSettings("echo", "hello")
rs.set_tasks(1)
rs.set("exclusive")

# Only set this argument if condition param evals True
# Otherwise log and NOP
rs.set("partition", "debug",
       condition=socket.gethostname()=="testing-system")

rs.format_run_args()
# returns ["exclusive", "None", "partition", "debug"] iff
  socket.gethostname()=="testing-system"
# otherwise returns ["exclusive", "None"]

Parameters:

arg (str) – name of the argument
value (str | None) – value of the argument
conditon – set the argument if condition evaluates to True

set_binding(binding: str) → None#

Set binding

Parameters:: binding (str) – Binding

set_broadcast(dest_path: str | None = None) → None#

Copy the specified executable(s) to remote machines

This sets --preload-binary

Parameters:: dest_path (str | None) – Destination path (Ignored)

set_cpu_binding_type(bind_type: str) → None#

Specifies the cores to which MPI processes are bound

This sets --bind-to for MPI compliant implementations

Parameters:: bind_type (str) – binding type

set_cpu_bindings(bindings: int | List[int]) → None#

Set the cores to which MPI processes are bound

Parameters:: bindings (list[int] | int) – List specifing the cores to which MPI processes are bound

set_cpus_per_task(cpus_per_task: int) → None#

Set the number of tasks for this job

This sets --cpus-per-proc for MPI compliant implementations

note: this option has been deprecated in openMPI 4.0+ and will soon be replaced.

Parameters:: cpus_per_task (int) – number of tasks

set_excluded_hosts(host_list: str | List[str]) → None#

Specify a list of hosts to exclude for launching this job

Parameters:: host_list (str | list[str]) – hosts to exclude

set_hostlist(host_list: str | List[str]) → None#

Set the hostlist for the mpirun command

This sets --host

Parameters:: host_list (str | list[str]) – list of host names
Raises:: TypeError – if not str or list of str

set_hostlist_from_file(file_path: str) → None#

Use the contents of a file to set the hostlist

This sets --hostfile

Parameters:: file_path (str) – Path to the hostlist file

set_memory_per_node(memory_per_node: int) → None#

Set the amount of memory required per node in megabytes

Parameters:: memory_per_node (int) – Number of megabytes per node

set_mpmd_preamble(preamble_lines: List[str]) → None#

Set preamble to a file to make a job MPMD

Parameters:: preamble_lines (list[str]) – lines to put at the beginning of a file.

set_nodes(nodes: int) → None#

Set the number of nodes

Parameters:: nodes (int) – number of nodes to run with

set_quiet_launch(quiet: bool) → None#

Set the job to run in quiet mode

This sets --quiet

Parameters:: quiet (bool) – Whether the job should be run quietly

set_task_map(task_mapping: str) → None#

Set mpirun task mapping

this sets --map-by <mapping>

For examples, see the man page for mpirun

Parameters:: task_mapping (str) – task mapping

set_tasks(tasks: int) → None#

Set the number of tasks for this job

This sets -n for MPI compliant implementations

Parameters:: tasks (int) – number of tasks

set_tasks_per_node(tasks_per_node: int) → None#

Set the number of tasks per node

Parameters:: tasks_per_node (int) – number of tasks to launch per node

set_time(hours: int = 0, minutes: int = 0, seconds: int = 0) → None#

Automatically format and set wall time

Parameters:

hours (int) – number of hours to run job
minutes (int) – number of minutes to run job
seconds (int) – number of seconds to run job

set_verbose_launch(verbose: bool) → None#

Set the job to run in verbose mode

This sets --verbose

Parameters:: verbose (bool) – Whether the job should be run verbosely

set_walltime(walltime: str) → None#

Set the maximum number of seconds that a job will run

This sets --timeout

Parameters:: walltime (str) – number like string of seconds that a job will run in secs

update_env(env_vars: Dict[str, str | int | float | bool]) → None#

Update the job environment variables

Parameters:: env_vars (dict[str, Union[str, int, float, bool]]) – environment variables to update or add
Raises:: TypeError – if env_vars values cannot be coerced to strings

SbatchSettings#

SbatchSettings are used for launching batches onto Slurm WLM systems.

`SbatchSettings.set_account`(account)	Set the account for this batch job
`SbatchSettings.set_batch_command`(command)	Set the command used to launch the batch e.g.
`SbatchSettings.set_nodes`(num_nodes)	Set the number of nodes for this batch job
`SbatchSettings.set_hostlist`(host_list)	Specify the hostlist for this job
`SbatchSettings.set_partition`(partition)	Set the partition for the batch job
`SbatchSettings.set_queue`(queue)	alias for set_partition
`SbatchSettings.set_walltime`(walltime)	Set the walltime of the job
`SbatchSettings.format_batch_args`()	Get the formatted batch arguments for a preview

class SbatchSettings(nodes: int | None = None, time: str = '', account: str | None = None, batch_args: Dict[str, str | None] | None = None, **kwargs: Any)[source]#

Specify run parameters for a Slurm batch job

Slurm sbatch arguments can be written into batch_args as a dictionary. e.g. {‘ntasks’: 1}

If the argument doesn’t have a parameter, put None as the value. e.g. {‘exclusive’: None}

Initialization values provided (nodes, time, account) will overwrite the same arguments in batch_args if present

Parameters:

nodes (int, optional) – number of nodes, defaults to None
time (str, optional) – walltime for job, e.g. “10:00:00” for 10 hours
account (str, optional) – account for job, defaults to None
batch_args (dict[str, str], optional) – extra batch arguments, defaults to None

add_preamble(lines: List[str]) → None#

Add lines to the batch file preamble. The lines are just written (unmodified) at the beginning of the batch file (after the WLM directives) and can be used to e.g. start virtual environments before running the executables.

Parameters:: line (str or list[str]) – lines to add to preamble.

property batch_args: Dict[str, str | None]#

property batch_cmd: str#

Return the batch command

Tests to see if we can expand the batch command path. If we can, then returns the expanded batch command. If we cannot, returns the batch command as is.

Returns:: batch command
Type:: str

format_batch_args() → List[str][source]#

Get the formatted batch arguments for a preview

Returns:: batch arguments for Sbatch
Return type:: list[str]

property preamble: Iterable[str]#: Return an iterable of preamble clauses to be prepended to the batch file

set_account(account: str) → None[source]#

Set the account for this batch job

Parameters:: account (str) – account id

set_batch_command(command: str) → None#

Set the command used to launch the batch e.g. sbatch

Parameters:: command (str) – batch command

set_cpus_per_task(cpus_per_task: int) → None[source]#

Set the number of cpus to use per task

This sets --cpus-per-task

Parameters:: num_cpus (int) – number of cpus to use per task

set_hostlist(host_list: str | List[str]) → None[source]#

Specify the hostlist for this job

Parameters:: host_list (str | list[str]) – hosts to launch on
Raises:: TypeError – if not str or list of str

set_nodes(num_nodes: int) → None[source]#

Set the number of nodes for this batch job

Parameters:: num_nodes (int) – number of nodes

set_partition(partition: str) → None[source]#

Set the partition for the batch job

Parameters:: partition (str) – partition name

set_queue(queue: str) → None[source]#

alias for set_partition

Sets the partition for the slurm batch job

Parameters:: queue (str) – the partition to run the batch job on

set_walltime(walltime: str) → None[source]#

Set the walltime of the job

format = “HH:MM:SS”

Parameters:: walltime (str) – wall time

QsubBatchSettings#

QsubBatchSettings are used to configure jobs that should be launched as a batch on PBSPro systems.

`QsubBatchSettings.set_account`(account)	Set the account for this batch job
`QsubBatchSettings.set_batch_command`(command)	Set the command used to launch the batch e.g.
`QsubBatchSettings.set_nodes`(num_nodes)	Set the number of nodes for this batch job
`QsubBatchSettings.set_ncpus`(num_cpus)	Set the number of cpus obtained in each node.
`QsubBatchSettings.set_queue`(queue)	Set the queue for the batch job
`QsubBatchSettings.set_resource`(...)	Set a resource value for the Qsub batch
`QsubBatchSettings.set_walltime`(walltime)	Set the walltime of the job
`QsubBatchSettings.format_batch_args`()	Get the formatted batch arguments for a preview

Specify qsub batch parameters for a job

nodes, and ncpus are used to create the select statement for PBS if a select statement is not included in the resources. If both are supplied the value for select statement supplied in resources will override.

Parameters:

nodes (int, optional) – number of nodes for batch, defaults to None
ncpus (int, optional) – number of cpus per node, defaults to None
time (str, optional) – walltime for batch job, defaults to None
queue (str, optional) – queue to run batch in, defaults to None
account (str, optional) – account for batch launch, defaults to None
resources (dict[str, str], optional) – overrides for resource arguments, defaults to None
batch_args (dict[str, str], optional) – overrides for PBS batch arguments, defaults to None

add_preamble(lines: List[str]) → None#

Parameters:: line (str or list[str]) – lines to add to preamble.

property batch_args: Dict[str, str | None]#

property batch_cmd: str#

Return the batch command

Tests to see if we can expand the batch command path. If we can, then returns the expanded batch command. If we cannot, returns the batch command as is.

Returns:: batch command
Type:: str

format_batch_args() → List[str][source]#

Get the formatted batch arguments for a preview

Returns:: batch arguments for Qsub
Return type:: list[str]
Raises:: ValueError – if options are supplied without values

property preamble: Iterable[str]#: Return an iterable of preamble clauses to be prepended to the batch file

property resources: Dict[str, str | int]#

set_account(account: str) → None[source]#

Set the account for this batch job

Parameters:: acct (str) – account id

set_batch_command(command: str) → None#

Set the command used to launch the batch e.g. sbatch

Parameters:: command (str) – batch command

set_hostlist(host_list: str | List[str]) → None[source]#

Specify the hostlist for this job

Parameters:: host_list (str | list[str]) – hosts to launch on
Raises:: TypeError – if not str or list of str

set_ncpus(num_cpus: int | str) → None[source]#

Set the number of cpus obtained in each node.

If a select argument is provided in QsubBatchSettings.resources, then this value will be overridden

Parameters:: num_cpus (int) – number of cpus per node in select

set_nodes(num_nodes: int) → None[source]#

Set the number of nodes for this batch job

In PBS, ‘select’ is the more primitive way of describing how many nodes to allocate for the job. ‘nodes’ is equivalent to ‘select’ with a ‘place’ statement. Assuming that only advanced users would use ‘set_resource’ instead, defining the number of nodes here is sets the ‘nodes’ resource.

Parameters:: num_nodes (int) – number of nodes

set_queue(queue: str) → None[source]#

Set the queue for the batch job

Parameters:: queue (str) – queue name

set_resource(resource_name: str, value: str | int) → None[source]#

Set a resource value for the Qsub batch

If a select statement is provided, the nodes and ncpus arguments will be overridden. Likewise for Walltime

Parameters:

resource_name (str) – name of resource, e.g. walltime
value (str) – value

set_walltime(walltime: str) → None[source]#

Set the walltime of the job

format = “HH:MM:SS”

If a walltime argument is provided in QsubBatchSettings.resources, then this value will be overridden

Parameters:: walltime (str) – wall time

BsubBatchSettings#

BsubBatchSettings are used to configure jobs that should be launched as a batch on LSF systems.

`BsubBatchSettings.set_walltime`(walltime)	Set the walltime
`BsubBatchSettings.set_smts`(smts)	Set SMTs
`BsubBatchSettings.set_project`(project)	Set the project
`BsubBatchSettings.set_nodes`(num_nodes)	Set the number of nodes for this batch job
`BsubBatchSettings.set_expert_mode_req`(...)	Set allocation for expert mode.
`BsubBatchSettings.set_hostlist`(host_list)	Specify the hostlist for this job
`BsubBatchSettings.set_tasks`(tasks)	Set the number of tasks for this job
`BsubBatchSettings.format_batch_args`()	Get the formatted batch arguments for a preview

Specify bsub batch parameters for a job

Parameters:

nodes (int, optional) – number of nodes for batch, defaults to None
time (str, optional) – walltime for batch job in format hh:mm, defaults to None
project (str, optional) – project for batch launch, defaults to None
batch_args (dict[str, str], optional) – overrides for LSF batch arguments, defaults to None
smts (int, optional) – SMTs, defaults to 0

add_preamble(lines: List[str]) → None#

Parameters:: line (str or list[str]) – lines to add to preamble.

property batch_args: Dict[str, str | None]#

property batch_cmd: str#

Return the batch command

Tests to see if we can expand the batch command path. If we can, then returns the expanded batch command. If we cannot, returns the batch command as is.

Returns:: batch command
Type:: str

format_batch_args() → List[str][source]#

Get the formatted batch arguments for a preview

Returns:: list of batch arguments for Qsub
Return type:: list[str]

property preamble: Iterable[str]#: Return an iterable of preamble clauses to be prepended to the batch file

set_account(account: str) → None[source]#

Set the project

this function is an alias for set_project.

Parameters:: account (str) – project name

set_batch_command(command: str) → None#

Set the command used to launch the batch e.g. sbatch

Parameters:: command (str) – batch command

set_expert_mode_req(res_req: str, slots: int) → None[source]#

Set allocation for expert mode. This will activate expert mode (-csm) and disregard all other allocation options.

This sets -csm -n slots -R res_req

set_hostlist(host_list: str | List[str]) → None[source]#

Specify the hostlist for this job

Parameters:: host_list (str | list[str]) – hosts to launch on
Raises:: TypeError – if not str or list of str

set_nodes(num_nodes: int) → None[source]#

Set the number of nodes for this batch job

This sets -nnodes.

Parameters:: nodes (int) – number of nodes

set_project(project: str) → None[source]#

Set the project

This sets -P.

Parameters:: time (str) – project name

set_queue(queue: str) → None[source]#

Set the queue for this job

Parameters:: queue (str) – The queue to submit the job on

set_smts(smts: int) → None[source]#

Set SMTs

This sets -alloc_flags. If the user sets SMT explicitly through -alloc_flags, then that takes precedence.

Parameters:: smts (int) – SMT (e.g on Summit: 1, 2, or 4)

set_tasks(tasks: int) → None[source]#

Set the number of tasks for this job

This sets -n

Parameters:: tasks (int) – number of tasks

set_walltime(walltime: str) → None[source]#

Set the walltime

This sets -W.

Parameters:: walltime (str) – Time in hh:mm format, e.g. “10:00” for 10 hours, if time is supplied in hh:mm:ss format, seconds will be ignored and walltime will be set as hh:mm

Singularity#

Singularity is a type of Container that can be passed to a RunSettings class or child class to enable running the workload in a container.

class Singularity(*args: Any, **kwargs: Any)[source]#

Singularity (apptainer) container type. To be passed into a RunSettings class initializer or Experiment.create_run_settings.

Note

Singularity integration is currently tested with Apptainer 1.0 with slurm and PBS workload managers only.

Also, note that user-defined bind paths (mount argument) may be disabled by a system administrator

Parameters:

image (str) – local or remote path to container image, e.g. docker://sylabsio/lolcow
args (str | list[str], optional) – arguments to ‘singularity exec’ command
mount (str | list[str] | dict[str, str], optional) – paths to mount (bind) from host machine into image.

Orchestrator#

class Orchestrator(port: int = 6379, interface: str | List[str] = 'lo', launcher: str = 'local', run_command: str = 'auto', db_nodes: int = 1, batch: bool = False, hosts: str | List[str] | None = None, account: str | None = None, time: str | None = None, alloc: str | None = None, single_cmd: bool = False, *, threads_per_queue: int | None = None, inter_op_threads: int | None = None, intra_op_threads: int | None = None, db_identifier: str = 'orchestrator', **kwargs: Any)[source]#

The Orchestrator is an in-memory database that can be launched alongside entities in SmartSim. Data can be transferred between entities by using one of the Python, C, C++ or Fortran clients within an entity.

Initialize an Orchestrator reference for local launch

Parameters:

port (int, optional) – TCP/IP port, defaults to 6379
interface (str, list[str], optional) – network interface(s), defaults to “lo”

Extra configurations for RedisAI

See https://oss.redislabs.com/redisai/configuration/

Parameters:

threads_per_queue (int, optional) – threads per GPU device
inter_op_threads (int, optional) – threads accross CPU operations
intra_op_threads (int, optional) – threads per CPU operation

property batch: bool#

property db_identifier: str#

Return the DB identifier, which is common to a DB and all of its nodes

Returns:: DB identifier
Return type:: str

property db_models: Iterable[smartsim.entity.DBModel]#: Return an immutable collection of attached models

property db_nodes: int#

Read only property for the number of nodes an Orchestrator is launched across. Notice that SmartSim currently assumes that each shard will be launched on its own node. Therefore this property is currently an alias to the num_shards attribute.

Returns:: Number of database nodes
Return type:: int

property db_scripts: Iterable[smartsim.entity.DBScript]#: Return an immutable collection of attached scripts

enable_checkpoints(frequency: int) → None[source]#

Sets the database’s save configuration to save the DB every ‘frequency’ seconds given that at least one write operation against the DB occurred in that time. E.g., if frequency is 900, then the database will save to disk after 900 seconds if there is at least 1 change to the dataset.

Parameters:: frequency (int) – the given number of seconds before the DB saves

get_address() → List[str][source]#

Return database addresses

Returns:: addresses
Return type:: list[str]
Raises:: SmartSimError – If database address cannot be found or is not active

property hosts: List[str]#

Return the hostnames of orchestrator instance hosts

Note that this will only be populated after the orchestrator has been launched by SmartSim.

Returns:: hostnames
Return type:: list[str]

is_active() → bool[source]#

Check if the database is active

Returns:: True if database is active, False otherwise
Return type:: bool

property num_shards: int#

Return the number of DB shards contained in the orchestrator. This might differ from the number of DBNode objects, as each DBNode may start more than one shard (e.g. with MPMD).

Returns:: num_shards
Return type:: int

remove_stale_files() → None[source]#: Can be used to remove database files of a previous launch

reset_hosts() → None[source]#: Clear hosts or reset them to last user choice

set_batch_arg(arg: str, value: str | None = None) → None[source]#

Set a batch argument the orchestrator should launch with

Some commonly used arguments such as –job-name are used by SmartSim and will not be allowed to be set.

Parameters:

arg (str) – batch argument to set e.g. “exclusive”
value (str | None) – batch param - set to None if no param value

Raises:

SmartSimError – if orchestrator not launching as batch

set_cpus(num_cpus: int) → None[source]#

Set the number of CPUs available to each database shard

This effectively will determine how many cpus can be used for compute threads, background threads, and network I/O.

Parameters:: num_cpus (int) – number of cpus to set

set_db_conf(key: str, value: str) → None[source]#

Set any valid configuration at runtime without the need to restart the database. All configuration parameters that are set are immediately loaded by the database and will take effect starting with the next command executed.

Parameters:

key (str) – the configuration parameter
value (str) – the database configuration parameter’s new value

set_eviction_strategy(strategy: str) → None[source]#

Sets how the database will select what to remove when ‘maxmemory’ is reached. The default is noeviction.

Parameters:

strategy (str) – The max memory policy to use e.g. “volatile-lru”, “allkeys-lru”, etc.

Raises:

SmartSimError – If ‘strategy’ is an invalid maxmemory policy
SmartSimError – If database is not active

set_hosts(host_list: List[str] | str) → None[source]#

Specify the hosts for the Orchestrator to launch on

Parameters:: host_list (str, list[str]) – list of host (compute node names)
Raises:: TypeError – if wrong type

set_max_clients(clients: int = 50000) → None[source]#

Sets the max number of connected clients at the same time. When the number of DB shards contained in the orchestrator is more than two, then every node will use two connections, one incoming and another outgoing.

Parameters:: clients (int, optional) – the maximum number of connected clients

set_max_memory(mem: str) → None[source]#

Sets the max memory configuration. By default there is no memory limit. Setting max memory to zero also results in no memory limit. Once a limit is surpassed, keys will be removed according to the eviction strategy. The specified memory size is case insensitive and supports the typical forms of: 1k => 1000 bytes 1kb => 1024 bytes 1m => 1000000 bytes 1mb => 1024*1024 bytes 1g => 1000000000 bytes 1gb => 1024*1024*1024 bytes

Parameters:

mem (str) – the desired max memory size e.g. 3gb

Raises:

SmartSimError – If ‘mem’ is an invalid memory value
SmartSimError – If database is not active

set_max_message_size(size: int = 1073741824) → None[source]#

Sets the database’s memory size limit for bulk requests, which are elements representing single strings. The default is 1 gigabyte. Message size must be greater than or equal to 1mb. The specified memory size should be an integer that represents the number of bytes. For example, to set the max message size to 1gb, use 1024*1024*1024.

Parameters:: size (int, optional) – maximum message size in bytes

set_path(new_path: str) → None#

set_run_arg(arg: str, value: str | None = None) → None[source]#

Set a run argument the orchestrator should launch each node with (it will be passed to jrun)

Some commonly used arguments are used by SmartSim and will not be allowed to be set. For example, “n”, “N”, etc.

Parameters:

arg (str) – run argument to set
value (str | None) – run parameter - set to None if no parameter value

set_walltime(walltime: str) → None[source]#

Set the batch walltime of the orchestrator

Note: This will only effect orchestrators launched as a batch

Parameters:: walltime (str) – amount of time e.g. 10 hours is 10:00:00
Raises:: SmartSimError – if orchestrator isn’t launching as batch

property type: str#: Return the name of the class

Model#

`Model.__init__`(name, params, path, run_settings)	Initialize a `Model`
`Model.attach_generator_files`([to_copy, ...])	Attach files to an entity for generation
`Model.colocate_db`(args, *kwargs)	An alias for `Model.colocate_db_tcp`
`Model.colocate_db_tcp`([port, ifname, ...])	Colocate an Orchestrator instance with this Model over TCP/IP.
`Model.colocate_db_uds`([unix_socket, ...])	Colocate an Orchestrator instance with this Model over UDS.
`Model.colocated`	Return True if this Model will run with a colocated Orchestrator
`Model.add_ml_model`(name, backend[, model, ...])	A TF, TF-lite, PT, or ONNX model to load into the DB at runtime
`Model.add_script`(name[, script, ...])	TorchScript to launch with this Model instance
`Model.add_function`(name[, function, device, ...])	TorchScript function to launch with this Model instance
`Model.params_to_args`()	Convert parameters to command line arguments and update run settings.
`Model.register_incoming_entity`(incoming_entity)	Register future communication between entities.
`Model.enable_key_prefixing`()	If called, the entity will prefix its keys with its own model name
`Model.disable_key_prefixing`()	If called, the entity will not prefix its keys with its own model name
`Model.query_key_prefixing`()	Inquire as to whether this entity will prefix its keys with its name

class Model(name: str, params: Dict[str, str], path: str, run_settings: RunSettings, params_as_args: List[str] | None = None, batch_settings: BatchSettings | None = None)[source]#

Bases: SmartSimEntity

Initialize a Model

Parameters:

name (str) – name of the model
params (dict) – model parameters for writing into configuration files or to be passed as command line arguments to executable.
path (str) – path to output, error, and configuration files
run_settings (RunSettings) – launcher settings specified in the experiment
params_as_args (list[str]) – list of parameters which have to be interpreted as command line arguments to be added to run_settings
batch_settings (BatchSettings | None) – Launcher settings for running the individual model as a batch job, defaults to None

add_function(name: str, function: str | None = None, device: Literal['CPU', 'GPU'] = 'CPU', devices_per_node: int = 1, first_device: int = 0) → None[source]#

TorchScript function to launch with this Model instance

Each script function to the model will be loaded into a non-converged orchestrator prior to the execution of this Model instance.

For converged orchestrators, the add_script() method should be used.

Device selection is either “GPU” or “CPU”. If many devices are present, a number can be passed for specification e.g. “GPU:1”.

Setting devices_per_node=N, with N greater than one will result in the model being stored in the first N devices of type device.

Parameters:

name (str) – key to store function under
function (str, optional) – TorchScript function code
device (str, optional) – device for script execution, defaults to “CPU”
devices_per_node (int) – The number of GPU devices available on the host. This parameter only applies to GPU devices and will be ignored if device is specified as CPU.
first_device (int) – The first GPU device to use on the host. This parameter only applies to GPU devices and will be ignored if device is specified as CPU.

add_ml_model(name: str, backend: str, model: bytes | None = None, model_path: str | None = None, device: Literal['CPU', 'GPU'] = 'CPU', devices_per_node: int = 1, first_device: int = 0, batch_size: int = 0, min_batch_size: int = 0, min_batch_timeout: int = 0, tag: str = '', inputs: List[str] | None = None, outputs: List[str] | None = None) → None[source]#

A TF, TF-lite, PT, or ONNX model to load into the DB at runtime

Each ML Model added will be loaded into an orchestrator (converged or not) prior to the execution of this Model instance

One of either model (in memory representation) or model_path (file) must be provided

Parameters:

name (str) – key to store model under
backend (str) – name of the backend (TORCH, TF, TFLITE, ONNX)
model (byte string, optional) – A model in memory (only supported for non-colocated orchestrators)
model_path (file path to model) – serialized model
device (str, optional) – name of device for execution, defaults to “CPU”
devices_per_node (int) – The number of GPU devices available on the host. This parameter only applies to GPU devices and will be ignored if device is specified as CPU.
first_device (int) – The first GPU device to use on the host. This parameter only applies to GPU devices and will be ignored if device is specified as CPU.
batch_size (int, optional) – batch size for execution, defaults to 0
min_batch_size (int, optional) – minimum batch size for model execution, defaults to 0
min_batch_timeout (int, optional) – time to wait for minimum batch size, defaults to 0
tag (str, optional) – additional tag for model information, defaults to “”
inputs (list[str], optional) – model inputs (TF only), defaults to None
outputs (list[str], optional) – model outupts (TF only), defaults to None

add_script(name: str, script: str | None = None, script_path: str | None = None, device: Literal['CPU', 'GPU'] = 'CPU', devices_per_node: int = 1, first_device: int = 0) → None[source]#

TorchScript to launch with this Model instance

Each script added to the model will be loaded into an orchestrator (converged or not) prior to the execution of this Model instance

Device selection is either “GPU” or “CPU”. If many devices are present, a number can be passed for specification e.g. “GPU:1”.

Setting devices_per_node=N, with N greater than one will result in the script being stored in the first N devices of type device; alternatively, setting first_device=M will result in the script being stored on nodes M through M + N - 1.

One of either script (in memory string representation) or script_path (file) must be provided

Parameters:

name (str) – key to store script under
script (str, optional) – TorchScript code (only supported for non-colocated orchestrators)
script_path (str, optional) – path to TorchScript code
device (str, optional) – device for script execution, defaults to “CPU”
devices_per_node (int) – The number of GPU devices available on the host. This parameter only applies to GPU devices and will be ignored if device is specified as CPU.
first_device (int) – The first GPU device to use on the host. This parameter only applies to GPU devices and will be ignored if device is specified as CPU.

attach_generator_files(to_copy: List[str] | None = None, to_symlink: List[str] | None = None, to_configure: List[str] | None = None) → None[source]#

Attach files to an entity for generation

Attach files needed for the entity that, upon generation, will be located in the path of the entity. Invoking this method after files have already been attached will overwrite the previous list of entity files.

During generation, files “to_copy” are copied into the path of the entity, and files “to_symlink” are symlinked into the path of the entity.

Files “to_configure” are text based model input files where parameters for the model are set. Note that only models support the “to_configure” field. These files must have fields tagged that correspond to the values the user would like to change. The tag is settable but defaults to a semicolon e.g. THERMO = ;10;

Parameters:

to_copy (list, optional) – files to copy, defaults to []
to_symlink (list, optional) – files to symlink, defaults to []
to_configure (list, optional) – input files with tagged parameters, defaults to []

property attached_files_table: str#

Return a list of attached files as a plain text table

Returns:: String version of table
Return type:: str

colocate_db(*args: Any, **kwargs: Any) → None[source]#: An alias for Model.colocate_db_tcp

colocate_db_tcp(port: int = 6379, ifname: t.Union[str, list[str]] = 'lo', db_cpus: int = 1, custom_pinning: t.Optional[t.Iterable[t.Union[int, t.Iterable[int]]]] = None, debug: bool = False, db_identifier: str = '', **kwargs: t.Any) → None[source]#

Colocate an Orchestrator instance with this Model over TCP/IP.

This method will initialize settings which add an unsharded database to this Model instance. Only this Model will be able to communicate with this colocated database by using the loopback TCP interface.

Extra parameters for the db can be passed through kwargs. This includes many performance, caching and inference settings.

ex. kwargs = {
    maxclients: 100000,
    threads_per_queue: 1,
    inter_op_threads: 1,
    intra_op_threads: 1,
    server_threads: 2 # keydb only
}

Generally these don’t need to be changed.

Parameters:

port (int, optional) – port to use for orchestrator database, defaults to 6379
ifname (str | list[str], optional) – interface to use for orchestrator, defaults to “lo”
db_cpus (int, optional) – number of cpus to use for orchestrator, defaults to 1
custom_pinning (iterable of ints or iterable of ints, optional) – CPUs to pin the orchestrator to. Passing an empty iterable disables pinning
debug (bool, optional) – launch Model with extra debug information about the colocated db
kwargs (dict, optional) – additional keyword arguments to pass to the orchestrator database

colocate_db_uds(unix_socket: str = '/tmp/redis.socket', socket_permissions: int = 755, db_cpus: int = 1, custom_pinning: Iterable[int | Iterable[int]] | None = None, debug: bool = False, db_identifier: str = '', **kwargs: Any) → None[source]#

Colocate an Orchestrator instance with this Model over UDS.

This method will initialize settings which add an unsharded database to this Model instance. Only this Model will be able to communicate with this colocated database by using Unix Domain sockets.

Extra parameters for the db can be passed through kwargs. This includes many performance, caching and inference settings.

example_kwargs = {
    "maxclients": 100000,
    "threads_per_queue": 1,
    "inter_op_threads": 1,
    "intra_op_threads": 1,
    "server_threads": 2 # keydb only
}

Generally these don’t need to be changed.

Parameters:

unix_socket (str, optional) – path to where the socket file will be created
socket_permissions (int, optional) – permissions for the socketfile
db_cpus (int, optional) – number of cpus to use for orchestrator, defaults to 1
custom_pinning (iterable of ints or iterable of ints, optional) – CPUs to pin the orchestrator to. Passing an empty iterable disables pinning
debug (bool, optional) – launch Model with extra debug information about the colocated db
kwargs (dict, optional) – additional keyword arguments to pass to the orchestrator database

property colocated: bool#: Return True if this Model will run with a colocated Orchestrator

property db_models: Iterable[DBModel]#: Return an immutable collection of attached models

property db_scripts: Iterable[DBScript]#: Return an immutable collection attached of scripts

disable_key_prefixing() → None[source]#: If called, the entity will not prefix its keys with its own model name

enable_key_prefixing() → None[source]#: If called, the entity will prefix its keys with its own model name

params_to_args() → None[source]#: Convert parameters to command line arguments and update run settings.

print_attached_files() → None[source]#: Print a table of the attached files on std out

query_key_prefixing() → bool[source]#: Inquire as to whether this entity will prefix its keys with its name

register_incoming_entity(incoming_entity: SmartSimEntity) → None[source]#

Registers the named data sources that this entity has access to by storing the key_prefix associated with that entity

Parameters:: incoming_entity (SmartSimEntity) – The entity that data will be received from
Raises:: SmartSimError – if incoming entity has already been registered

property type: str#: Return the name of the class

Ensemble#

`Ensemble.__init__`(name, params[, ...])	Initialize an Ensemble of Model instances.
`Ensemble.add_model`(model)	Add a model to this ensemble
`Ensemble.add_ml_model`(name, backend[, ...])	A TF, TF-lite, PT, or ONNX model to load into the DB at runtime
`Ensemble.add_script`(name[, script, ...])	TorchScript to launch with every entity belonging to this ensemble
`Ensemble.add_function`(name[, function, ...])	TorchScript function to launch with every entity belonging to this ensemble
`Ensemble.attach_generator_files`([to_copy, ...])	Attach files to each model within the ensemble for generation
`Ensemble.enable_key_prefixing`()	If called, each model within this ensemble will prefix its key with its own model name.
`Ensemble.models`	Helper property to cast self.entities to Model type for type correctness
`Ensemble.query_key_prefixing`()	Inquire as to whether each model within the ensemble will prefix their keys
`Ensemble.register_incoming_entity`(...)	Register future communication between entities.

class Ensemble(name: str, params: Dict[str, Any], params_as_args: List[str] | None = None, batch_settings: BatchSettings | None = None, run_settings: RunSettings | None = None, perm_strat: str = 'all_perm', **kwargs: Any)[source]#

Bases: EntityList[Model]

Ensemble is a group of Model instances that can be treated as a reference to a single instance.

Initialize an Ensemble of Model instances.

The kwargs argument can be used to pass custom input parameters to the permutation strategy.

Parameters:

name (str) – name of the ensemble
params (dict[str, Any]) – parameters to expand into Model members
params_as_args (list[str]) – list of params that should be used as command line arguments to the Model member executables and not written to generator files
batch_settings (BatchSettings, optional) – describes settings for Ensemble as batch workload
run_settings (RunSettings, optional) – describes how each Model should be executed
replicas (int, optional) – number of Model replicas to create - a keyword argument of kwargs
perm_strategy (str) – strategy for expanding params into Model instances from params argument options are “all_perm”, “step”, “random” or a callable function. Defaults to “all_perm”.

Returns:

Ensemble instance

Return type:

Ensemble

add_function(name: str, function: str | None = None, device: Literal['CPU', 'GPU'] = 'CPU', devices_per_node: int = 1, first_device: int = 0) → None[source]#

TorchScript function to launch with every entity belonging to this ensemble

Each script function to the model will be loaded into a non-converged orchestrator prior to the execution of every entity belonging to this ensemble.

For converged orchestrators, the add_script() method should be used.

Device selection is either “GPU” or “CPU”. If many devices are present, a number can be passed for specification e.g. “GPU:1”.

Parameters:

name (str) – key to store function under
function (str, optional) – TorchScript code
device (str, optional) – device for script execution, defaults to “CPU”
devices_per_node (int) – number of devices on each host
first_device (int) – first device to use on each host

A TF, TF-lite, PT, or ONNX model to load into the DB at runtime

Each ML Model added will be loaded into an orchestrator (converged or not) prior to the execution of every entity belonging to this ensemble

One of either model (in memory representation) or model_path (file) must be provided

Parameters:

name (str) – key to store model under
model (str | bytes | None) – model in memory
model_path (file path to model) – serialized model
backend (str) – name of the backend (TORCH, TF, TFLITE, ONNX)
device (str, optional) – name of device for execution, defaults to “CPU”
devices_per_node (int, optional) – number of GPUs per node in multiGPU nodes, defaults to 1
first_device (int, optional) – first device in multi-GPU nodes to use for execution, defaults to 0; ignored if devices_per_node is 1
batch_size (int, optional) – batch size for execution, defaults to 0
min_batch_size (int, optional) – minimum batch size for model execution, defaults to 0
min_batch_timeout (int, optional) – time to wait for minimum batch size, defaults to 0
tag (str, optional) – additional tag for model information, defaults to “”
inputs (list[str], optional) – model inputs (TF only), defaults to None
outputs (list[str], optional) – model outupts (TF only), defaults to None

add_model(model: Model) → None[source]#

Add a model to this ensemble

Parameters:

model (Model) – model instance to be added

Raises:

TypeError – if model is not an instance of Model
EntityExistsError – if model already exists in this ensemble

TorchScript to launch with every entity belonging to this ensemble

Each script added to the model will be loaded into an orchestrator (converged or not) prior to the execution of every entity belonging to this ensemble

Device selection is either “GPU” or “CPU”. If many devices are present, a number can be passed for specification e.g. “GPU:1”.

Setting devices_per_node=N, with N greater than one will result in the model being stored in the first N devices of type device.

One of either script (in memory string representation) or script_path (file) must be provided

Parameters:

name (str) – key to store script under
script (str, optional) – TorchScript code
script_path (str, optional) – path to TorchScript code
device (str, optional) – device for script execution, defaults to “CPU”
devices_per_node (int) – number of devices on each host
first_device (int) – first device to use on each host

attach_generator_files(to_copy: List[str] | None = None, to_symlink: List[str] | None = None, to_configure: List[str] | None = None) → None[source]#

Attach files to each model within the ensemble for generation

Attach files needed for the entity that, upon generation, will be located in the path of the entity.

During generation, files “to_copy” are copied into the path of the entity, and files “to_symlink” are symlinked into the path of the entity.

Parameters:

to_copy (list, optional) – files to copy, defaults to []
to_symlink (list, optional) – files to symlink, defaults to []
to_configure (list, optional) – input files with tagged parameters, defaults to []

property attached_files_table: str#

Return a plain-text table with information about files attached to models belonging to this ensemble.

Returns:: A table of all files attached to all models
Return type:: str

property db_models: Iterable[smartsim.entity.DBModel]#: Return an immutable collection of attached models

property db_scripts: Iterable[smartsim.entity.DBScript]#: Return an immutable collection of attached scripts

enable_key_prefixing() → None[source]#: If called, each model within this ensemble will prefix its key with its own model name.

property models: Iterable[Model]#: Helper property to cast self.entities to Model type for type correctness

print_attached_files() → None[source]#: Print table of attached files to std out

query_key_prefixing() → bool[source]#

Inquire as to whether each model within the ensemble will prefix their keys

Returns:: True if all models have key prefixing enabled, False otherwise
Return type:: bool

register_incoming_entity(incoming_entity: SmartSimEntity) → None[source]#

Registers the named data sources that this entity has access to by storing the key_prefix associated with that entity

Only python clients can have multiple incoming connections

Parameters:: incoming_entity (SmartSimEntity) – The entity that data will be received from

property type: str#: Return the name of the class

Machine Learning#

SmartSim includes built-in utilities for supporting TensorFlow, Keras, and Pytorch.

TensorFlow#

SmartSim includes built-in utilities for supporting TensorFlow and Keras in training and inference.

freeze_model(model: Model, output_dir: str, file_name: str) → Tuple[str, List[str], List[str]][source]#

Freeze a Keras or TensorFlow Graph

to use a Keras or TensorFlow model in SmartSim, the model must be frozen and the inputs and outputs provided to the smartredis.client.set_model_from_file() method.

This utiliy function provides everything users need to take a trained model and put it inside an orchestrator instance

Parameters:

model (tf.Module) – TensorFlow or Keras model
output_dir (str) – output dir to save model file to
file_name (str) – name of model file to create

Returns:

path to model file, model input layer names, model output layer names

Return type:

str, list[str], list[str]

serialize_model(model: Model) → Tuple[str, List[str], List[str]][source]#

Serialize a Keras or TensorFlow Graph

to use a Keras or TensorFlow model in SmartSim, the model must be frozen and the inputs and outputs provided to the smartredis.client.set_model() method.

This utiliy function provides everything users need to take a trained model and put it inside an orchestrator instance.

Parameters:: model (tf.Module) – TensorFlow or Keras model
Returns:: serialized model, model input layer names, model output layer names
Return type:: str, list[str], list[str]

class StaticDataGenerator(**kwargs: Any)[source]#

Bases: _TFDataGenerationCommon

A class to download a dataset from the DB.

Details about parameters and features of this class can be found in the documentation of DataDownloader, of which it is just a TensorFlow-specialized sub-class with dynamic=False.

init_samples(init_trials: int = -1) → None#

Initialize samples (and targets, if needed).

A new attempt to download samples will be made every ten seconds, for init_trials times. :param init_trials: maximum number of attempts to fetch data :type init_trials: int

property need_targets: bool#

Compute if targets have to be downloaded.

Returns:: Whether targets (or labels) should be downloaded
Return type:: bool

on_epoch_end() → None#

Callback called at the end of each training epoch

If self.shuffle is set to True, data is shuffled.

class DynamicDataGenerator(**kwargs: Any)[source]#

Bases: _TFDataGenerationCommon

A class to download batches from the DB.

Details about parameters and features of this class can be found in the documentation of DataDownloader, of which it is just a TensorFlow-specialized sub-class with dynamic=True.

init_samples(init_trials: int = -1) → None#

Initialize samples (and targets, if needed).

A new attempt to download samples will be made every ten seconds, for init_trials times. :param init_trials: maximum number of attempts to fetch data :type init_trials: int

property need_targets: bool#

Compute if targets have to be downloaded.

Returns:: Whether targets (or labels) should be downloaded
Return type:: bool

on_epoch_end() → None[source]#

Callback called at the end of each training epoch

Update data (the DB is queried for new batches) and if self.shuffle is set to True, data is also shuffled.

PyTorch#

SmartSim includes built-in utilities for supporting PyTorch in training and inference.

class StaticDataGenerator(**kwargs: Any)[source]#

Bases: _TorchDataGenerationCommon

A class to download a dataset from the DB.

Details about parameters and features of this class can be found in the documentation of DataDownloader, of which it is just a PyTorch-specialized sub-class with dynamic=False and init_samples=False.

When used in the DataLoader defined in this class, samples are initialized automatically before training. Other data loaders using this generator should implement the same behavior.

init_samples(init_trials: int = -1) → None#

Initialize samples (and targets, if needed).

A new attempt to download samples will be made every ten seconds, for init_trials times. :param init_trials: maximum number of attempts to fetch data :type init_trials: int

property need_targets: bool#

Compute if targets have to be downloaded.

Returns:: Whether targets (or labels) should be downloaded
Return type:: bool

class DynamicDataGenerator(**kwargs: Any)[source]#

Bases: _TorchDataGenerationCommon

A class to download batches from the DB.

Details about parameters and features of this class can be found in the documentation of DataDownloader, of which it is just a PyTorch-specialized sub-class with dynamic=True and init_samples=False.

When used in the DataLoader defined in this class, samples are initialized automatically before training. Other data loaders using this generator should implement the same behavior.

init_samples(init_trials: int = -1) → None#

Initialize samples (and targets, if needed).

A new attempt to download samples will be made every ten seconds, for init_trials times. :param init_trials: maximum number of attempts to fetch data :type init_trials: int

property need_targets: bool#

Compute if targets have to be downloaded.

Returns:: Whether targets (or labels) should be downloaded
Return type:: bool

class DataLoader(dataset: _TorchDataGenerationCommon, **kwargs: Any)[source]#

Bases: DataLoader

DataLoader to be used as a wrapper of StaticDataGenerator or DynamicDataGenerator

This is just a sub-class of torch.utils.data.DataLoader which sets up sources of a data generator correctly. DataLoader parameters such as num_workers can be passed at initialization. batch_size should always be set to None.

Slurm#

`get_allocation`([nodes, time, account, options])	Request an allocation
`release_allocation`(alloc_id)	Free an allocation's resources

SmartSim API

Contents

SmartSim API#

Experiment#

Settings#

RunSettings#

SrunSettings#

AprunSettings#

JsrunSettings#

MpirunSettings#

MpiexecSettings#

OrterunSettings#

SbatchSettings#

QsubBatchSettings#

BsubBatchSettings#

Singularity#

Orchestrator#

Orchestrator#

Model#

Ensemble#

Machine Learning#

TensorFlow#

PyTorch#

Slurm#